Microsoft’s solution to this transition from 32-bit to 64-bit has been to add legacy support for most 32-bit applications. In other words, most 32-bit applications will function in the 64-bit operating environment. Keep in mind that other operating systems operating on a 64-bit architecture cannot load or run 32-bit applications at all.

To help make the transition easier, Microsoft has designated that all 32-bit application should, by default, be loaded into the Program Files (x86) folder rather than getting mixed in with true 64-bit applications in the regular Program Files folder.

Source

"what would go wrong if I somehow avoided the redirection mechanism and forced everything to install to the real C:\Program Files?"

Nothing. The two program directories are only for organization, or to keep programs that have two version a 32-bit and 64-bit version separate, like Internet Explorer. But you can install a 32-bit program in "Program Files" and a 64-bit program in "Program Files x86" and nothing will happen the program will run the same.

[Wiki][2] says:

Some application installers reject spaces within the install path location. For 32-bit systems, the short name for the Program Files folder is Progra~1. For 64-bit systems, the short name for the 64-bit Program Files folder is Progra~1 (same as on 32-bit systems); while the short name for the 32-bit Program Files (x86) folder is now Progra~2.

Microsoft’s solution to this transition from 32-bit to 64-bit has been to add legacy support for most 32-bit applications. In other words, most 32-bit applications will function in the 64-bit operating environment. Keep in mind that other operating systems operating on a 64-bit architecture cannot load or run 32-bit applications at all.

To help make the transition easier, Microsoft has designated that all 32-bit application should, by default, be loaded into the Program Files (x86) folder rather than getting mixed in with true 64-bit applications in the regular Program Files folder.

Source

"what would go wrong if I somehow avoided the redirection mechanism and forced everything to install to the real C:\Program Files?"

Nothing. The two program directories are only for organization, or to keep programs that have two version a 32-bit and 64-bit version separate, like Internet Explorer. But you can install a 32-bit program in "Program Files" and a 64-bit program in "Program Files x86" and nothing will happen the program will run the same.

Wiki says:

Some application installers reject spaces within the install path location. For 32-bit systems, the short name for the Program Files folder is Progra~1. For 64-bit systems, the short name for the 64-bit Program Files folder is Progra~1 (same as on 32-bit systems); while the short name for the 32-bit Program Files (x86) folder is now Progra~2.