$a+bi+\sqrt{a^2+b^2} = 2 + 8i$ so
$a + \sqrt{a^2 + b^2} = 2$ and $b = 8$.
So $a + \sqrt{a^2 + 64} = 2$
So $\sqrt{a^2 + 64} = 2- a$
$a^2 + 64 = 4 -4a + a^2$
$4a = -60$
$a = -15$.
$z = -15 + 8i$.
....
To do what you were attempting
You have to realize that the $Re(z) = a + \sqrt{a^2 + b^2}$ and $Im(z) = b$. I think somehow you were thinking there were threee parts $Re(z)=a$ and $Im(z) = b$ and some $Weird(z) = \sqrt{a^2 + b^2}$ and that $z\overline z = Re^2(z) - Im^2(z) + Weird^2(z)$. That simply isn't true....
$(a+bi+\sqrt{a^2+b^2})(a - bi +\sqrt{a^2 + b^2}) = (2 + 8i)(2-8i)$
$(a + \sqrt{a^2 + b^2})^2 - b^2 = 4 - 64$
$2a^2 + b^2 + 2a\sqrt{a^2 + b^2} -b^2 = -60$
$a^2 + a \sqrt{a^2 + b^2} = -30$
which is a pain to solve but can be done.