Using the example of $\ce{XeF4}$:
What is the physical explanation enforcing the symmetry of the $\ce{1b_{1g}}$ orbital on the fluorine atoms? Why isn't the symmetry of a nonbonding orbital arbitrary? If it's going to be nonbonding anyways, why can't we, for example, have a Fluorine p-orbital arrangement facing towards Xenon with three positive p-orbitals and one negative p-orbital?
To elaborate:
If I imagine a free Xenon atom in space, and the approach of four individual fluorine atoms, I would expect the bond formation to be randomized with respect to the orientation of the fluorine p-orbitals, and therefore for some arrangements to not be perfectly symmetrical, such as 3 positive p-orbitals, 1 negative, facing inward. I understand bonds can't be made without symmetry between the Xenon and Fluorine orbitals; that makes physical sense because we can argue it by looking at orbital overlap that dictates bonds can only occur with appropriate symmetry. But in a nonbonding case, such as $\ce{1b_{1g}}$ above, I don't understand why symmetry is also required.
Source for pictures: