In the proof of Wigner's theorem, the crucial role is played by the quantity $|\langle\phi|\psi\rangle|^2$ where $|\psi\rangle$ and $|\phi\rangle$ represent two distinct physical states (or more precisely, two distinct rays) of a system. Expanding the states in the orthonormal eigenbasis $\{|a_n\rangle\}$ of a Hermitian operator $\hat{A}$ as $$|\psi\rangle=\sum_n c_n|a_n\rangle, \quad |\phi\rangle=\sum_n d_n|a_n\rangle,$$ the object of interest can be written as, $$|\langle\psi|\phi\rangle|^2=\left|\sum_n d_n^*c_n\right|^2.$$ Wigner's theorem is based on the notion of symmetry transformation which changes the states from $|\psi\rangle\to|\psi^\prime\rangle$ and $|\phi\rangle\to|\phi^\prime\rangle$ keeping $|\langle\phi|\psi\rangle|^2$ unchanged. Well, we could define symmetry transformation in this abstract way and proceed from here.
But I would like to feel it in my bones. What does the object $|\langle\psi|\phi\rangle|^2$ mean, physically? Even if not measurable, I would at least like to understand, why we expect this quantity to remain unchanged under a rotation, translation, etc.
Please note that neither $\psi$ nor $\phi$ be eigenstates of any hermitian operator. If $|\phi\rangle=|a_m\rangle$, where $|a_m\rangle$ is an eigenstate of a hermitian operator $\hat{A}$ with eigenvalue $a_m$, then postulate of QM, the quantity $|\langle a_m|\psi\rangle|^2$ has a direct physical meaning. It is the probability that the outcome of measurement will of the observable $A$, will return a value $a_m$. But $|\langle\phi|\psi\rangle|^2$, for a general state $|\phi\rangle$, do not have any such interpretation, as far as I understand.