I am confused about basics of $\mathbb{Z}_2$ (and likely other) gauge theories and plain disorder. Let $$H=H_F + h\,H_{EM}$$ $$H_F = -t\sum_l (c^\dagger_l \sigma^z_{l,l+1} c_{l+1} + h.c.)$$ be (the 'texbook') chain of length $N$ of spinless fermions $c^{(\dagger)}_l$ at sites $l$, coupled via a $\mathbb{Z}_2$ Peierls factor $\sigma^z_{l,l+1}$ to the gauge field. The pure gauge field's Hamiltonian $H_{EM}$ is not relevant and $h$ is some coupling constant. As usual, $H$ should commute with all local generators $G_l = \sigma^x_{l-1,l} (-)^{n_l} \sigma^x_{l,l+1}$, where $n_l=c^\dagger_l c_l$ is the particle number.
Now my problem(?) is, that from the literature, it is never absolutely clear to me, but often it seems implied, that for $h=0$, i.e. when matter decouples from the gauge field, then, $H_F$ represents a translationally invariant free fermion Hamiltonian, which can be solved by Fourier transformation to produce the standard $\cos(k)$ dispersion.
I understand, that the superselection sectors with $G_l=1$ or $G_l=-1$ for all $l$ are special, but for $h=0$ and instead of the $G_l$, one can also just use that all $[H_F,\sigma^z_{l,l+1}]=0$ and classify w.r.t. $2^N$ (mostly) random and static distributions of $\sigma^z_{l,l+1}=\pm 1$. Therefore I would expect, that most of the eigenstates of $H_F$ are Anderson-localized with energies that have nothing to do with a $\cos(k)$-band.
So, would a proper statement be, that only the gauge vacuum returns the free matter of the un-gauged theory at $h=0$ and moreover, that the fermions in almost all of the remaining superselection sectors are localized. ... Or is there some 'voodoo' I am lacking?