In Chapter 5 of Baumann's cosmology book where he discusses structure formation starting from Newtonian perturbation theory, Baumann mentions at the beginning that
Newtonian gravity is a good approximation when all velocities are small compared to the speed of light and the departures from a flat spacetime geometry are small. The latter is satisfied on scales smaller than the Hubble radius, $H^{-1}$.
I am confused by the bolded statement. It makes sense that on very large scales there should be significant departures from flat spacetime since the primordial power spectrum of the gravitational potential $P_\Phi\propto k^{-3}$ (roughly) so on large scales there should be significant deviations from flat spacetime. It also makes sense that the Newtonian approximation breaks down above the Hubble radius since Newtonian gravity assumes instantaneous propagation of gravitational interactions which can't happen if the signal originates from outside the Hubble horizon and hasn't had enough time to propagate.
But what I'm fuzzy on is how we know that there isn't some radius $r_*<H^{-1}$ where departures from a flat spacetime geometry start to become significant or even how we would define "departures from a flat spacetime geometry"