It's always nice to point out the structural similarieties between (semi-)Riemannian geometry and gauge field theories alla Classical yang Mills theories. Nevertheless, I feel the relation between the gauge group and the gauge bosons "$A_{\mu}\times T^i_{\ j}$" are much simpler than the relation betreen the diffeomorphisms/isometries and the Christoffel symbols $\Gamma_{\mu\nu}^\rho$. At the very least, at first glance, I don't see the $SO(3)$ matrix lingering in $\Gamma$. I don't really know the technicalities of the Erlangen program and I specifically don't know how that "generating a manifold from a single point using the isometry group" (as I understand it) really works.
I'd like to know if it is really directly possible to view both, that is general relativity and the classical gauge theories, just as special cases of the general Cartan connection framework, or if there is a fundamental difficulty to (I guess) general relativty which makes this not work. Especially if the answer is no, then what is it that makes the metric theory so different? (At first I'd guess it's the fact that the GR manifold looks different in each patch due to $g$, but $A$ can also be difficult and both are related to what else is in space. So I suppose it's the interplay between the isometry group and the bigger diffeomorphism group.)