I'm generally aware that there have been attempts to describe things like magnetism and the other forces geometrically, like with gravity, and that QFTs have essentially supplanted them. But it's also my understanding that per GR, we don't simply treat spacetime curvature as a model for how gravity works but rather consider space actually to be curved by the presence of mass-energy - or at least gravity is fundamentally indistinguishable from actual curvature.
My question is why is gravity special in this regard? The only obvious reason is that it's (as far as we know) asymmetric, but imagine an extraordinarily powerful magnetically charged body attracting a smaller metallic object in space. In that setup, if we only look at the magnetic field in one direction, it seems like all of the postulates of GR (equivalence principle, etc.) would be equally applicable and would lead one to derive essentially the same equations, just with different constants.
So why do we say that gravity curves space but the magnetic field doesn't? Is it simply the case that we use the best models for each force just because they are the best models that fit observation and not becuase we are convinced the forces are fundamentally different?
Maybe a clearer way to ask this is whether we believe that the acceleration caused by gravity is of a fundamentally different character (namely the curvature of space) than the acceleration caused by quantum fields and virtual particles? Or am I just asking the $billion question underlying the search for a TOE?
(Note I'm not talking about the gravity caused by the mass-energy associated with the magnetic field).