quotient group scheme

Question

assume I have a group $G$ over a field of char 0 and $H$ a closed subgroup. When is it true that the group $N(H)/H$ is representable? If $G$ has nice properties, like to be reductive or unipotent is it true that $N(H)/H$ is representable and that it has the same properties?

Answer 1

This is true in general. In fact, if $G$ is any fintie type group scheme over any field $F$ and $H$ is any closed subgroup scheme, then the quotient $G/H$ sheaf (in the fppf topology) is representable. One can see, for example [Mil, Theorem B.37].

As to your second question, the quotient $N(H)/H$ should inerit most reasonable problems from $N(H)$, so properties that descend to closed subgroups and quotients will be satisfied also for $N(H)/H$.

For example, if $G$ is unipotent then so is $N(H)$ and thus so will be $N(H)/H$ (since the map $N(H)(\overline{F})\to (N(H)/H)(\overline{F})$ is surjective and thus by the functoriality of the Jordan decomposition every element of $(N(H)/H)(\overline{F})$ is unipotent).

The group $N(H)/H$ is not likely to be connected, let alone reductive, even if $G$ is. As a simple example, if $G$ is reductive and $H=T$ a maximal torus, then $N(T)/T$ is the Weyl group scheme $\Omega(G,T)$ which is a finite etale group scheme over $F$ (in particular not connected).

This last example also indicates another powerful observation about $N(H)/H$, which is that it embeds into the sheaf $\mathrm{Aut}(H)$ (which is itself representable in good situations--e.g. see [Con, Theorem 7.1.9]).

[Con] Conrad, B., 2014. Reductive group schemes. Autour des schémas en groupes, 1, pp.93-444.

[Mil] Milne, J.S., 2017. Algebraic groups: The theory of group schemes of finite type over a field (Vol. 170). Cambridge University Press.