If $b^2-b<0$, then $b-b^2>0$ and $$a = \pm i\sqrt{ b - b^2 },$$ where $i$ is the imaginary unit, which by definition is the unique complex number that satisfies $$i^2=-1\Leftrightarrow i=\pm\sqrt{-1}.$$
The complex numbers are numbers of the form $a+bi$, where $a$ and $b$ are real numbers. They appear e.g. in the solution of a quadratic equation with negative discriminant, such as this one $$x^2+x+1=0,$$ whose solutions are $$x=\dfrac{-1\pm\sqrt{1-4}}{2}=\dfrac{-1\pm\sqrt{-3}}{2}=\dfrac{-1\pm\sqrt{3}\ i}{2}.$$
Example: For $b=1/2$, we have $b^2-b=1/4-1/2=-1/4$ and
$$a = \pm i\sqrt{ \frac{1}{2} - \frac{1}{4 }}=\pm i\sqrt{ \frac{1}{4} }=\pm \frac{1}{2}i .$$
We could have computed as follows
$$a = \sqrt{ \frac{1}{4 }-\frac{1}{2} }=\sqrt{ -\frac{1}{4} }=\sqrt{ -1}\sqrt{ \frac{1}{4} }=\sqrt{ -1}\frac{1}{2}=\pm i \frac{1}{2}=\pm \frac{1}{2}i .$$