In standard classical mechanics' textbooks (for instance Morin) one finds that while calculating the angular momentum vector of a rigid body for a 3-D general case from the definition:
$$L=r \times p$$
The requirement to define a moment of inertia tensor, which encodes the full information of the mass distribution undergoing rotations, naturally arises.
We also find that the $I_{i,j}$ terms (for $i\neq j$) remain invariant under exchange of $x$ and $y$ since their product is involved.
This means that the moment of inertia is symmetric.
Now, from the derivation it wasn't clear to me why it comes out that way.
I am guessing since products of coordinates are involved in the non-diagonal terms it has to with rotational invariance of the space. I say this in the sense that rotating $x$ into $y$ coordinates leave the tensor invariant but I am not satisfied with my guess.
In short, I am asking:
"Besides the mathematical derivation how can the symmetric property of the MOI tensor be explained on physical grounds? And if the matrix was not symmetric what would that imply for the space around us?"