The isometry group of Minkowski spacetime is \begin{equation} \mathrm{Poincar\acute{e}}=\mathrm{ISO}(\mathbb R^{1,d-1})=\mathrm O(1,d-1)\ltimes\mathbb R^d \end{equation} whose homogeneous part is $\mathrm O(1,d-1)$, the so-called Lorentz Group1. This group is simple.
On the other hand, the isometry group of Galilean space+time is2 \begin{equation} \text{Bargmann}=\mathrm{ISO}(\mathbb R^1\times\mathbb R^{d-1})\times\mathrm U(1)=(\mathrm O(d-1)\ltimes\mathbb R^{d-1})\ltimes(\mathrm U(1)\times\mathbb R^1\times\mathbb R^{d-1}) \end{equation} whose homogeneous part is $\mathrm O(d-1)\ltimes\mathbb R^{d-1}$, the so-called (homogeneous) Galilei Group. This group is not semi-simple (it contains a non-trivial normal subgroup, that of boosts).
Another reason is that the postulate of causality is trivial in non-relativistic systems (because there is an absolute notion of time), but it imposes strong restrictions on relativistic systems (because there is no absolute notion of time). This postulate is translated into the quantum theory through the axiom of locality, $$ [\phi(x),\phi(y)]=0\quad\forall x,y\quad \text{s.t.}\quad (x-y)^2<0 $$ where $[\cdot,\cdot]$ denotes a supercommutator. In other words, any two operators whose support are casually disconnected must (super)commute. In non-relativistic systems this axiom is vacuous because all spacetime intervals are spaceliketimelike, $(x-y)^2<0$$(x-y)^2>0$, that is, all spacetime points are casually connected. In relativistic systems, this axiom is very strong.
1: More generally, the indefinite orthogonal (or pseudo-orthogonal) group $\mathrm O(p,q)$ is defined as the set of $(p+q)$-dimensional matrices, with real coefficients, that leave invariant the metric with signature $(p,q)$: $$ \mathrm O(p,q):=\{M\in \mathrm{M}_{p+q}(\mathbb R)\ \mid\ M\eta M^T\equiv \eta\},\qquad \eta:=\mathrm{diag}(\overbrace{-1,\dots,-1}^p,\overbrace{+1,\dots,+1}^q) $$
The special indefinite orthogonal group $\mathrm{SO}(p,q)$ is the subset of $\mathrm O(p,q)$ with unit determinant. If $pq\neq0$, the group $\mathrm{SO}(p,q)$ has two disconnected components. In this answer, "Lorentz group" may refer to the orthogonal group with signature $(1,d-1)$; to its $\det(M)\equiv+1$ component; or to its orthochronus subgroup $M^0{}_0\ge+1$. Only the latter is simply-connected. The topology of the group is mostly irrelevant for this answer, so we shall make no distinction between the three different possible notions of "Lorentz group".2: One can prove that the inhomogeneous Galilei algebra, and unlike the Poincaré algebra, has a non-trivial second co-homology group. In other words, it admits a non-trivial central extension. The Bargmann group is defined precisely as the centrally extended inhomogeneous Galilei group. Strictly speaking, all we know is that the central extension has the algebra $\mathbb R$; at the group level, it could lead to a factor of $\mathrm U(1)$ as above, or to a factor of $\mathbb R$. In quantum mechanics the first option is more natural, because we may identify this phase with the $\mathrm U(1)$ symmetry of the Schrödinger equation (which has a larger symmetry group, the so-called Schrödinger group). Again, the details of the topology of the group are mostly irrelevant for this answer.