If your integers are stored as $B$ bits long, then you need r to be only $B/2$ bits or less, i.e. $r \leq 2^{B/2 - 1}$. (The minus one is because you are using signed integers.) Otherwise you will get integer overflow when you compute $r * r$ and the answer won't be correct. Using doubles (long floats) instead of integers won't really help either, because you won't have enough precision to get $r*r - i*i$ when $r$ and $i$ are both large, e.g. when you try to set $i = r-1$ and $r$ is large you will actually get $i*i = r*r$ so the expression will evaluate to zero.
The other possibilities are that when you convert rr - ii to a floating point to compute the square root, the squared value might not fit precisely into the bits of mantissa precision. So actually you might need $r$ to be a few orders of magnitude less than $2^{B/2}$. Also, when you convert to a floating point, it's possible that when you compute the square root you will get a floating point answer slightly smaller than the true integer answer when the expression under the square root is a perfect integer square. Then when you cast to integer, your answer will be 1 less than it should be. To test for this and fix it, if the integer under the square-root is $A$ and you compute the floored square-root to be $B$, then you should have $A - B^2 \leq 2B$, or equivalently $(B+1)^2 > A$ (all integer arihmetic). If not, then add one to $B$.