$\def\mybra#1{\langle{#1}|}$ $\def\myket#1{|{#1}\rangle}$ $\def\Tr{\mbox{Tr}}$ $\def\mybraket#1{\langle{#1}\rangle}$ $\def\B#1#2{B^{#1}_{#2}}$
$\def\C#1#2{C^{#1}_{#2}}$
$\def\r#1#2{\hat\rho^{#1}_{#2}}$
I want to expand my density operator $$ \hat \rho = \sum_{jm j'm'} \myket{jm}\mybraket{jm|\hat \rho|j'm'} \mybra{j'm'}\tag1 $$ with spherical tensor operators, $$ \hat\rho = \sum_{k}\sum_{q=-k}^k c^k_q\r kq,\tag2 $$ where $\myket{jm}$ is eigen vector of angular momentum $\hat J_z$, and $$ \hat\rho^k_q = \sum_{jmj'm'} \B{jm}{kq j'm'} \myket{jm}\mybra{j'm'}\tag3 $$ is spherical operator that satisfies $$ [\hat J_z,\r kq]=q\hbar \hat\rho^kq,\quad [\hat J_\pm ,\hat\rho^kq] = \hbar\sqrt{k(k+1)-q(q\pm 1)}\r k{q\pm1}. \tag4 $$ Using recursion relations of C-G coefficients, $\B{jm}{kqj'm'}$ can be determined to be C-G coefficient except a scale factor, $$ \B{jm}{kqj'm'}=f_{{kjj'}} \C{jm}{kqj'm'},\quad \C{jm}{kqj'm'} = \mybraket{kqj'm'|(kj')jm}.\tag5 $$ The factor should be independent of $m,m'$ because the recursion relation changes $m,m',q$.
To determine $f_{kjj'}$ and $c^k_q$, I should make use of the orthogonality and completeness. For orthogonality, $$ \Tr((\r {k_1}{q_1})^\dagger \r {k_2}{q_2}) = \sum f_{k_1j_1j_1'}^* f_{k_2j_2j_2'} \C{j_1m_1}{k_1q_1 j_1'm_1'}\C{j_2m_2}{k_2q_2 j_2'm_2'} \delta_{jj_1'}\delta_{mm_1'}\delta_{jj_2'}\delta_{j_1j_2}\delta_{m_1m_2} \delta_{mm_2'}.\tag6 $$ With the restriction $m_1=m_1'+q_1, m_2=m_2'+q_2$, the delta's in the sum indicates the L.H.S equals 0 unless $q_1=q_2$. And only three summation variables $j_1,j,m$ are left, $$ \Tr((\r {k_1}{q_1})^\dagger \r {k_2}{q_2}) =\sum_{j_1,j,m} f_{k_1j_1j}^* f_{k_2j_1j} \C{j_1(m+q_1)}{k_1q_1 jm}\C{j_1(m+q_1)}{k_2q_1 jm}.\tag7 $$ Using symmetry property of C-G coefficients $$ \C{jm}{j_1m_1 j_2m_2}=\sqrt{\frac{2j+1}{2j_2+1}} (-)^{j_1-m_1} \C{j_2(-m_2)}{j_1m_1j(-m)},\tag8 $$ I get $$ \begin{aligned} \Tr((\r {k_1}{q_1})^\dagger \r {k_2}{q_2}) &= \sum_{j_1,j,m} f_{k_1j_1j}^* f_{k_2j_1j} \C{j_1(m+q_1)}{k_1q_1 jm}\C{j_1(m+q_1)}{k_2q_1 jm} \\ &= \sum_{j_1,j,m} f_{k_1j_1j}^* f_{k_2j_1j}{\frac{2j_1+1 }{2j+1}} \C{j(-m)}{k_1q_1 j_1(-m-q_1)}\C{j(-m)}{k_2q_1 j_1(-m-q_1)}\\ &= \sum_{j_1,j,m} f_{k_1j_1j}^* f_{k_2j_1j}{\frac{2j_1+1 }{2j+1}} \C{jm}{k_1q_1 j_1(m+q_1)}\C{jm}{k_2q_1 j_1(m+q_1)}. \end{aligned}\tag9 $$ The third equaliy holds because summation over $m$ is symmetric about 0. Let $f_{kj'j}=f_k\sqrt{(2j+1)/(2j'+1)}$ where $f_k$ is another factor to be determined, so $$ \Tr((\r {k_1}{q_1})^\dagger \r {k_2}{q_2}) = f_{k_1}^* f_{k_2} \sum_{j_1jm} \C{jm}{k_1q_1 j_1(m+q_1)}\C{jm}{k_2q_1 j_1(m+q_1)}. \tag{10} $$
Now, if $k_1=k_2$ I can tell the summation is $1$ using the orthonality of C-G coefficients. But I can't tell if it's 0 when $k_1\neq k_2$. What's the next step?
