Denote $\;t=20^{\LARGE\circ},\; u=\cot t,\;v=\cot 2t,\;w=\cot 4 t,\quad $ then
$$S =\tan^2 10^{\LARGE\circ}+\tan^2 50^{\LARGE\circ}+\tan^2 70^{\LARGE\circ}=\cot^2 80^{\LARGE\circ}+\cot^2 40^{\LARGE\circ}+\cot^2 20^{\LARGE\circ}=u^2+v^2+w^2,$$
$$\cot8t=\cot 160^{\LARGE\circ}=-u.$$
From the known identity
$$\cot(2a)=\dfrac12(\cot(a)-\tan(a))=\dfrac{\cot^2(a)-1}{2\cot(a)}\tag1$$
should
$$u^2=2uv+1,\quad v^2=2vw+1,\quad w^2=-2wu+1,\tag2$$
where
$$u>v>w>0.\tag 3$$
Summation of the equalities $(2)$ allows to get
$$(u-v+w)^2=u^2+v^2+w^2-2v(u+w)+2uw=3,$$
$$u-v+w=\sqrt3.\tag4$$
Then
$$v=\dfrac{u^2-1}{2u},\tag5$$
$$w=\sqrt3+v-u=\sqrt3+\dfrac{u^2-1}{2u}-u,$$
$$w=\sqrt3-\dfrac{u^2+1}{2u}.\tag6$$
On the other hand,
$$\dfrac1{\sqrt3}=\cot 3t=\dfrac{u(u^2-3)}{3u^2-1},$$
$$u^3=\dfrac{3u^2-1}{\sqrt3}+3u\tag7.$$
Therefore,
$$\begin{align}
&u^2+v^2+w^2=u^2+\left(\dfrac{u^2-1}{2u}\right)^2+\left(\sqrt3-\dfrac{u^2+1}{2u}\right)^2\\[4pt]
&=\dfrac1{4u^2}\left(4u^4+\left(u^2-1\right)^2+\left(2u\sqrt3-u^2-1\right)^2\right)\\[4pt]
&=\dfrac1{4u^2}\left(6u^4-4u^3\sqrt3+12u^2-4u\sqrt3+2\right)\\[4pt]
&=\dfrac1{4u^2}\left(6u^4-4u^3\sqrt3+12u^2-4u\sqrt3+2
-(6u+2\sqrt3)\left(u^3-\dfrac{3u^2-1}{\sqrt3}-3u\right)\right)\\[4pt]
&=\dfrac{36u^2}{4u^2}=9.
\end{align}$$