Basis functions in group theory and wave functions

Question

I just started to study group theory by reading the book Group theory: Applications to the physics of condensed matter by M. S. Dresselhaus. In chapter 4 it was mentioned:

Suppose that we have a group $G$ with symmetry elements $R$ and symmetry operators $\hat{P}_R$ . We denote the irreducible representations by $\Gamma_n$, where $n$ labels the representation. We can then define a set of basis vectors denoted by $\left| \Gamma_n j\right>$.

...

These basis vectors relate the symmetry operator $\hat{P}_R$ with its matrix representation $D^{(Γ_n)} (R)$ through the relation \begin{equation} \hat{P}_R \left|\Gamma_n \alpha\right> = \sum_j D^{(\Gamma_n)}(R)_{j\alpha} \left| \Gamma_n j \right>\end{equation} The basis vectors can be abstract vectors; a very important type of basis vector is a basis function which we define here as a basis vector expressed explicitly in coordinate space. Wave functions in quantum mechanics, which are basis functions for symmetry operators, are a special but important example of such basis functions.

I don't understand the definition here. Are basis functions defined through the equation above? How do I know whether a function is a appropriately chosen basis function that generate an irrep or not? Also, under what situation do wave functions become basis functions defined here (I'm guessing when the Hamiltonian possess the symmetry associated with the group), and why?

I have tried to search for answer in the book and also on the internet, but found nothing useful. It would be great if someone can provide some help. Thank you.

My attempt to the question

I noticed that we can have any set of basis and prove that the coefficients in the equation above are indeed the representation of the group. Therefore, I believe that the basis vectors can really be any set of basis in a vector space. However, I would like to know whether this statement is true. Also, there are still several problems as listed below.

Let's assume we have a set of basis vectors $\left|\Gamma_n i\right>$ in a vector space, and we have group elements $\alpha$, $\beta$, $\gamma$ with $\gamma = \beta\alpha$, and the corresponding symmetry operators $\hat{P}_\alpha$, $\hat{P}_\beta$, $\hat{P}_\gamma$. We let $\hat{P}_\alpha$ act on a basis vector, and the result should in general be able to be expanded by the same set of basis: $$ \hat{P}_\alpha \left|\Gamma_n i\right> = \sum_j C^\alpha_{ji} \left| \Gamma_n j \right> $$ Next we also let $\hat{P}_beta$ act on it: $$ \hat{P}_\beta\hat{P}_\alpha \left|\Gamma_n i\right> = \hat{P}_\beta \sum_j C^\alpha_{ji} \left| \Gamma_n j \right> = \sum_{j, k} C^\beta_{kj} C^{\alpha}_{ji} \left|\Gamma_n k\right> $$ But at the same time, $\hat{P}_\gamma = \hat{P}_\beta \hat{P}_\alpha$, so $$ \hat{P}_\gamma \left|\Gamma_n i\right> = \sum_k C^\gamma_{ki} \left| \Gamma_n k \right> $$ We see that $C^\gamma_{ki} = \sum_j C^\beta_{kj} C^\alpha_{ji}$, which shows that $C$ is a set of matrices that follows the same multiplication rules as the group, indicating that it must be a representation of the group.

Now, several problems arise here:

1. What are the conditions required to ensure that the representation here is irreducible?
2. If my proof is correct, it seems that the basis vectors do not even need to be orthogonal, as long as they are linearly independent. Is that true?
3. The vector space here can be any vector space, as long as they have well-defined innerproduct. But the basis functions listed in the character tables can be quadratic, so what is the definition of inner product here?

Answer 1

The basis functions are whatever you want them to be. There are no constraints on them other than that they actually form a basis for the representation (a linearly independent set of vectors that span all the vectors that transform under the representation). If you have two bases $|\Gamma_nj\rangle,|\Gamma_n'j\rangle$ for the same representation, then you can write each vector in one as a linear combination of the vectors in the other and arrange the coefficients into a matrix $B$ that helps transform between the two: $$|\Gamma_n'\alpha\rangle=\sum_jB_{j\alpha}|\Gamma_nj\rangle.$$

The equation given in your book defines the matrix $D^{(\Gamma_n)}(R)$ that represents the operator $\hat P_R,$ not the basis: the $\alpha$'th column of the matrix representation forms the coefficients on the $\Gamma_n$ basis vectors when you apply the symmetry operator to the $\alpha$'th element in that basis. If you have this matrix $D^{(\Gamma_n)}(R)$ for the $|\Gamma_nj\rangle$ basis, you can convert it into any other basis once you have the matrix $B$ by the similarity transformation $$D^{(\Gamma_n')}(R)=BD^{(\Gamma_n)}(R)B^{-1}.$$

Read $B^{-1}$ as converting from the new basis back to the old one, $D^{(\Gamma_n)}(R)$ as performing the operation in the old basis, and $B$ as converting back to the new basis. $D^{(\Gamma_n')}(R)$ satisfies the equation you gave, but for the new basis: $$\hat P_R|\Gamma_n'\alpha\rangle=\sum_jD^{(\Gamma_n')}(R)_{j\alpha}|\Gamma_n'j\rangle.$$

This also tells you how to find reducible representations: Consider what happens if you can find a single similarity transformation like $B$ that makes $D^{(\Gamma_n)}(R)$ (proper) block triangular (with the same block sizes) for all $R$: $$D^{(\Gamma_n')}(R)=BD^{(\Gamma_n)}(R)B^{-1}=\begin{bmatrix}D^{(\Gamma_n',11)}(R)&D^{(\Gamma_n',12)}(R)\\\mathbf{0}&D^{(\Gamma_n',22)}(R)\end{bmatrix}$$

where $D^{(\Gamma_n,11)}(R)$ is a square matrix of reduced (but nonzero) dimension compared to $D^{(\Gamma_n)}(R).$ Say it is a $m\times m$ matrix. This means that the first $m$ columns of $B$ transform only into linear combinations of each other under $D^{(\Gamma_n)}(R)$ for all symmetry elements $R$. Equivalently, the vectors formed by taking each of the first $m$ columns of $B$ and using them as coefficients for the $|\Gamma_nj\rangle$ transform only into linear combinations of each other under $\hat P_R$. The span of these vectors together with this action of the group upon them form a subrepresentation of $\Gamma_n$ (and you can use these vectors as a basis for this representation). $\Gamma_n$ is irreducible iff there is no similarity transform (i.e. change of basis) that simultaneously block-triangularizes the matrix representations of all the symmetry operators.

Here is an example where we find a representation of all of $C_\mathrm{3v}$ in the valence s orbitals of $N\!H_3$. Due to the chosen basis, the matrix representation of all the symmetry operators are immediately seen as block triangular, indicating reducibility. If a different basis were chosen, you might not get that. The mathematically important part is whether there is a basis where the matrix representations of all the symmetry operators are block-triangular.

To your problem (3): inner products? Where? Nothing at all in either your question or my answer requires inner products. You need to decompose vectors as linear combinations of basis elements. That's all.

As to your "wavefunctions" question: Wavefunctions appear as the basis vectors of a representation when wavefunctions are the thing you're studying. Again, that's really it. You start with some Hilbert space of wavefunctions $\mathcal{H}$ and a Hamiltonian $H$. You look for a group $G$ and a unitary representation $\hat U\!_{R\in G}:\mathcal{H}\to\mathcal{H}.$ "Physically relevant" representations are the ones that commute with the Hamiltonian. You may also find subrepresentations of this one. For any representation, you may want to pick out some basis vectors for it.

TL;DR: Basis vectors are not important! The physically relevant things are the representations, consisting of a subspace of some vector space and a group action on that subspace. The basis vectors just make computation on those objects possible (the subspace is the span of any basis for it, and the group action defined on a basis extends to the whole (sub)space). You choose the basis however you want to make computation/interpretation easy.