You already can't conclude this for ordinary real or complex numbers; the best you could possibly ask for is that $A = \pm x$ (by which I mean $A = \pm x$ times the identity matrix).
But actually you can't even expect this. $A$ can be any diagonalizable matrix whose eigenvalues are $\pm \sqrt{x}$; even if we restrict only to diagonal matrices there are $2^n$ square roots (if $x \neq 0$), and without that restriction there are uncountably many square roots.
Worse, if $x = 0$ then $A$ need not be diagonalizable at all.