VEV implying symmetry breaking, but unable to pick out specific subgroup?

Question

Let's say we have a scalar theory with an $O(N)$ symmetry, for which the scalar fields $\phi_{nm}$ transform as a rank $2$ tensor. I can write down an action which spontaneously breaks the symmetry

$$S=-\int d^dx\Big(\frac{1}{2}\partial_{\mu}\phi_{nm}\partial^{\mu}\phi^{mn}+\lambda(\phi_{nm}\phi^{mn}-v^2)^2\Big)$$

At tree level, $\phi_{nm}$ acquires a VEV: $$\langle\phi_{nm}\rangle=vM_{nm}$$

Where $M_{nm}$ is any $N\times N$ matrix such that $M_{nm}M^{mn}=1$. Depending on the form of $M_{nm}$, different symmetries are preserved. For instance, if $M_{nm}=\delta_{nm}/\sqrt{N}$, then the full $O(N)$ symmetry is preserved. However it is completely conceivable for $M_{nm}$ to be some matrix which is only invariant under some subgroup of $O(N)$, say $O(K)$ for $K<N$.

What I am wondering is, will this theory inevitably have to pick out one subgroup which is left preserved (perhaps via some other VEV)? Or does this theory have a very complicated moduli space which has regions which have different symmetry breaking patterns?

Answer 1

Let me be more explicit with my suggestion, an extensive comment. Your O(N) representation is reducible, $$\phi= S+A+sI, $$ where S is the symmetric traceless piece, A the antisymmetric piece, and s the singlet, proportional to the trace of $\phi$, which poisons the symmetry breaking in your potential, as you observed already,
$$ \lambda \left (\operatorname{Tr} S^2- \operatorname{Tr} A^2 +Ns^2 -v^2 \right )^2, $$ with $\langle s\rangle= v /\sqrt{N}$.

Check how the traces decouple the irreps in the reduction. All three summands may contribute a piece of v.e.v.

You've already seen this with N=3, breaking up into a quintet, triplet, and singlet. Triplet! So, to explore its v.e.v., take S=0 and s=0, and unfold A, $$ A_{mn}= \epsilon_{mnk} \varphi ^k, $$ hence the O(3) vector linear σ-model, $$ \lambda \left ( 2\varphi^m \varphi^m -v^2 \right )^2. $$ Take $\langle \varphi ^3\rangle= v/\sqrt{2}$ as usual, which breaks O(3) down to O(2).

In the generic matrix language, $$ \langle \phi_{mn} \rangle= \langle A_{mn} \rangle=\epsilon_{mn3} v/\sqrt{2}, $$ retains 1-2 interchange (anti)symmetry.

Likewise, if we took s=0 and A=0, check that the same O(2) subgroup of the O(3) leaves the minimal vev $$\langle S \rangle = \frac{v}{\sqrt{6}} \begin{bmatrix} 1&0&0\\ 0 & 1 & 0 \\0& 0& -2 \end{bmatrix} $$ invariant. Note N=3 allows for a common result for the interstitial $\lambda_2=0$ of Ling-Fong Li 1974!!

So this O(2) is always unbroken, but vanishing v.e.v. s for the nontrivial irreps restore the other 2 symmetries. In your case, you should be able to plot the three rep components' v.e.v.s through barycentric coordinates in an equilateral triangle.

Such metastable potentials, "trough potentials" in field space, occur in supergravity theories and rely on minute extraneous for radiative correction terms to trigger a vacuum choice.

It is then not unreasonable to study the structure of all 3 representations scalar-coupled in the potential for generic N.