Let $R$ be the radius of convergence of the power series
$$\sum_{n = 0}^{\infty} a_n x^n.\tag{1}$$
Then define $c_n = a_n\cdot R^n$. We want the series
$$\sum_{n = 0}^{\infty} a_n^2 x^n\tag{2}$$
to converge on the closed interval $[-R^2,R^2]$, and that is the case if and only $(2)$ converges for $x = R^2$ and $x = -R^2$. If we consider only real coefficients $a_n$, then $a_n^2 \geqslant 0$ for all $n$, and the convergence of $(2)$ for $x = R^2$ already implies the (absolute and uniform) convergence on the interval $[-R^2,R^2]$. Plugging $x = R^2$ into $(2)$, we want the convergence of
$$\sum_{n = 0}^{\infty} a_n^2 R^{2n} = \sum_{n = 0}^{\infty} c_n^2.$$
And we want the series $(1)$ to diverge for $x = R$ and for $x = -R$. Plugging those values in, we want the divergence of
$$\sum_{n = 0}^{\infty} a_n R^n = \sum_{n = 0}^{\infty} c_n\qquad\text{and}\qquad \sum_{n = 0}^{\infty} a_n (-R)^n = \sum_{n = 0}^{\infty} (-1)^n c_n.$$
So we look for a sequence $(c_n)$ such that $\sum c_n$ and $\sum (-1)^n c_n$ diverge, but $\sum c_n^2$ converges. If we don't care about $\sum (-1)^n c_n$, then a famous example of such a sequence is $\bigl(\frac{1}{n+1}\bigr)_{n\in\mathbb{N}}$. Unfortunately, the alternating harmonic series converges, so we cannot use just that sequence. But we can modify that sequence to reach our goal.
One easy way is to nullify the effect of the $(-1)^n$ by inserting zeros into the sequence, say $c_n = 0$ for odd $n$ and $c_n = \frac{1}{n+1}$ for even $n$.