In this thread Lorentz transformations for spinors, V. Moretti made a claim as follows:
"it is possible to prove that no non-trivial finite-dimensional unitary representation exists for a non-compact connected Lie group that does not include proper non-trivial closed normal subgroups".
What is the mathematical proof of this claim?
Proposition. Let $G$ be a connected non-compact Lie group that is a semisimple Lie group and $$U: G \ni g \mapsto U_g \in B(H)$$ ($B(H)$ being the set of bounded operators $A:H \to H$) a continuous unitary representation over the finite-dimensional Hilbert space $H$. The following facts hold.
(a) $U$ cannot be faithful.
(b) If $G$ is a simple group or, more generally, if $G$ does not contain non-trivial proper normal closed subgroups, then $U$ is the trivial representation $U : G \ni g \mapsto I$.
Remarks
(1) Notice that no hypothesis is made on (ir)reducibility of the representation.
(2) The theorem applies to the simple Lie group $SO(1,3)_+$ since this is non-compact, connected and it does not include non-trivial closed normal subgroups: its strongly-continuous unitary representations are infinite-dimensional or trivial.
(3) The same result is valid for $SL(2,\mathbb C)$, which is non-compact and connected but not simple. Indeed, $\{\pm I\}$ is the unique non-trivial proper normal closed subgroup of $SL(2,\mathbb C)$. A finite-dimensional continuous unitary representation $U : SL(2, \mathbb C) \to B(H)$ cannot be faithful by (a). Therefore the closed normal subgroup $U^{-1}(I)$ cannot be trivial and therefore coincides with either $SL(2,\mathbb C)$, making $U$ trivial, or with $\{\pm I\}$. Let us examine this second possibility, and prove that $U$ has to be trivial also in this case. As is well known, the Lie group $SL(2,\mathbb C)$ is the universal covering of the Lie group $SO(1,3)_+$, and $\{\pm I\}$ is just the kernel of the covering homomorphism, so $SO(1,3)_+$ is diffeomorphic to $SL(2,\mathbb C)/\{\pm I\}$. It is easy to prove that, consequently, $U : SL(2, \mathbb C) \to B(H)$ defines a finite-dimensional continuous unitary representation $$U' : SO(1,3)_+ \ni \pm A \:\mapsto U_A \:\in B(H)\:.$$ The representation $U'$ must be trivial by (b). In turn, $U$ must be trivial as well because $U'(SO(1,3)_+)=U(SL(2,\mathbb C))$ and $U'(SO(1,3)_+)= \{I\}$.
Proof of the proposition.
Let us identify $H$ with $\mathbb C^n$ by means of an orthonormal basis. In this way, the representation $U$ can be viewed as an injective continuous group homomorphism $f : G \to U(n)$.
(a) Our final goals is proving that $f(G)$ is a compact embedded submanifold of $U(n)$ and that the injective homomorphism $f: G \to f(G)$ is actually a homeomorphism. This is not possible, because $G$ is not compact by hypotheses.
By known theorems on Lie groups , $f$ is differentiable (analytic) and $df|_e$ is a Lie algebra homomorphism which is injective if $f$ is faithful (because the kernel of $f$ is the discrete subgroup $\{e\}$). Assuming that $f$ is injective (i.e., $U$ is faithful), consider the Lie subalgebra $a := df|_e T_eG \subset u(n)$ where $u(n)$ is the Lie algebra of $U(n)$. Since $df|_e$ is injective, $a$ is isomorphic to $T_eG$. There is exactly one connected Lie subgroup $K \subset U(n)$ whose Lie algebra is $a$ in view of a known theorem. By definition of Lie subgroup, $K$ is an immersed submanifold of $U(n)$. However since this subgroup has a semisimple Lie algebra, Theorem 14.5.9 of [1] implies that it is closed in U(n) and thus it is an emebedded submanifold as a consequence of Cartan theorem.
It must be clear that $f(G) \cap K$ contains all one-parameter subgroups of $U(n)$ generated by the elements of $a$ because these subgroups are simultaneously in $K$ and in $f(G)$, as the reader can prove immediately. On the other hand, every element $h\in K$ is a finite product of elements belonging to the one-parameter subgroups of $K$ and thus $h$ is also a finite product of elements of $f(G)$. Since $f$ is a group homomorphism, every element $h\in K$ satisfies $h\in f(G)$. We have so far established that $K= f(G)$. The map $f: G \to K$ is a bijective differentiable map from the manifold $G$ to the embedded submanifold $K$ of $U(n)$. Since $df|_g = dL_{g^{-1}} \circ df|_e \circ d R_{g}$ where $R_g : G \ni h \mapsto hg \in G$ and $L_k : U(n) \ni r \mapsto kr \in U(n)$ are diffeomorphisms and therefore both $dL_{g^{-1}}$ and $d R_g$ are a bijections, we conclude that $df|_g$ is everywhere injective. As a consequence, if $p=\dim G$ and $q=\dim U(n)\geq p$, then for any chart $(S_g,\phi)$ around any $g \in G$ there is some chart $(V_g,\psi)$ in $U(n)$ around $f(g)$ with $$\psi \circ f \circ \phi^{-1}(x^1,\ldots, x^p) = (x^1,\ldots, x^p, 0,\ldots, 0) $$ where $(x^1,\ldots, x^p)$ belongs to the open set $\phi(V_g) \subset \mathbb R^q$. Since $f(G)= K$ is an embedded submanifold of $U(n)$, we have that $V_g \cap f(G)= f(S_g)$ possibly restricting $V_g$ around $f(g)$. In other words $f(S_g)$ is open in the induced topology of $f(G)\subset U(n)$. Since $g\in G$ is arbitrary and the property is valid by replacing $S_g$ with any smaller open set containing $g$, the injectivity of $f$ proves that $f: G \to K= f(G)$ is open: every open set $A\subset G$ is the union of open sets $A = \cup_{g\in G} A \cap S_g$; since $f$ is bijective onto $K$ we also have that $f(A)= \cup_{g\in G}f(A \cap S_g)$, which is open because union of open sets. The inverse $f^{-1}: K \to G$ which, again, exists because $f$ is bijective onto $K$, is therefore continuous. $K$ is closed and hence compact ($U(n)$ is compact). This is not possible, because $f^{-1}(K) = G$ is not compact by hypotheses and $f^{-1}$ is a continuous function. We conclude that $f: G \to U(n)$ cannot be injective, that is, $U: G \to B(H)$ cannot be faithful.
(b) If $G$ does not include non-trivial proper closed normal subgroups, the normal closed subgroup $U^{-1}(I)$ of $G$ must equal either $G$ or $\{e\}$. In the second case $U$ would be faithful, which is not permitted by (a). Summing up, $U^{-1}(I) = G$ so that $U(G)=\{I\}$. QED
[1] Hilgert, J., Neeb, K.-H.: Structure and Geometry of Lie Groups. Springer, New York, (2013).
