Apparently the flaw in your formula is that while estimating the probability of a group of $x$ people with the same birthday, you neglect to account for the fact that in order for the size of the group to be exactly $x,$ nobody else among the $25$ people can have the same birthday as these $x$ people.
The probability that nobody else has the same birthday is
$\left(\frac{364}{365}\right)^{25-x},$
so a more accurate sum is
$$E(X) = \sum_{x=2}^{25}x\frac{\binom{25}{x}364^{25-x}}{365^{24}}.$$
What was surprising to me is that according to Wolfram Alpha, this gives exactly the correct answer:
https://www.wolframalpha.com/input/?i=sum+x+%2825+choose+x%29364%5E%2825-x%29%2F365%5E24+for+x+%3D+2+to+25
The reason this is surprising is that one would think you have to account for cases where there is a group of $3$ and a group of $2,$ or three groups of $2,$ and so forth.
But I think the derivation of this summand is not literally $xP(X=x),$
but $xE(\text{number of cliques of size $x$}),$
where
\begin{multline}E(\text{number of cliques of size $x$}) =\\
(\text{number of subsets of size $x$})P(\text{a given subset of size $x$ is a clique})
\end{multline}
where a "clique" is a subset who all have the same birthday which they do not share with anyone else.
To evaluate that sum (i.e., the correct one) without a calculator, I think the easiest method is to convert it back to the original problem and then observe that that problem is solved by evaluating
$$25\left(1 - \left(\frac{364}{365}\right)^{24}\right),$$
so you now have a much simpler calculation.
As a first approximation,
$$\left(\frac{364}{365}\right)^{24}
= \left(\frac{365-1}{365}\right)^{24} \approx
\frac{365-24}{365} = 1 - \frac{24}{365}. $$
Then since $\frac{24}{365} \approx \frac{24}{360} = \frac1{15},$
we're looking for something slightly less than $\frac1{15} = 0.0666\ldots.$
The number we want is at least $1\%$ smaller but not $2\%$ smaller.
So let's say it's $0.066$ to keep the number of significant digits small.
So now we have
$$ 25(1 - (1 - 0.066)) = 25(0.066) = 0.165. $$
This is a little high. The fault is mainly in the first approximation.