Apparently the flaw in your formula is that while estimating the probability of a group of $x$ people with the same birthday, you neglect to account for the fact that in order for the size of the group to be exactly $x,$ nobody else among the $25$ people can have the same birthday as these $x$ people. The probability that nobody else has the same birthday is $\left(\frac{364}{365}\right)^{25-x},$ so a more accurate sum is
$$E(X) = \sum_{x=2}^{25}x\frac{\binom{25}{x}364^{25-x}}{365^{24}}.$$
What was surprising to me is that according to Wolfram Alpha, this gives exactly the correct answer: https://www.wolframalpha.com/input/?i=sum+x+%2825+choose+x%29364%5E%2825-x%29%2F365%5E24+for+x+%3D+2+to+25
The reason this is surprising is that one would think you have to account for cases where there is a group of $3$ and a group of $2,$ or three groups of $2,$ and so forth. But I think the derivation of this summand is not literally $xP(X=x),$ but $xE(\text{number of cliques of size $x$}),$ where \begin{multline}E(\text{number of cliques of size $x$}) =\\ (\text{number of subsets of size $x$})P(\text{a given subset of size $x$ is a clique}) \end{multline} where a "clique" is a subset who all have the same birthday which they do not share with anyone else.
To evaluate that sum (i.e., the correct one) without a calculator, I think the easiest method is to convert it back to the original problem and then observe that that problem is solved by evaluating $$25\left(1 - \left(\frac{364}{365}\right)^{24}\right),$$ so you now have a much simpler calculation. As a first approximation, $$\left(\frac{364}{365}\right)^{24} = \left(\frac{365-1}{365}\right)^{24} \approx \frac{365-24}{365} = 1 - \frac{24}{365}. $$ Then since $\frac{24}{365} \approx \frac{24}{360} = \frac1{15},$ we're looking for something slightly less than $\frac1{15} = 0.0666\ldots.$ The number we want is at least $1\%$ smaller but not $2\%$ smaller. So let's say it's $0.066$ to keep the number of significant digits small. So now we have $$ 25(1 - (1 - 0.066)) = 25(0.066) = 0.165. $$ This is a little high. The fault is mainly in the first approximation.