$\newcommand{\D}{\mathrm{D}}\newcommand{\ker}{\operatorname{ker}}\newcommand{\ind}{\operatorname{ind}}\newcommand{\dim}{\operatorname{dim}}$To count the zero-modes of an operator, you must, well, look at its kernel. In this case, the relevant operator is a Dirac operator, $\D_*$, in the presense of an instanton. You can show the two following cute facts:
- $\ker \overline{\D_*}=0$, in the background of an instanton.
- $\ker\D_* = \ker\overline{\D_*}\D_*$, where $\overline{\D_*}$ is the adjoint of $\D_*$ (under the inner-product that defines the action).
With the first fact, you can write
\begin{align}\#(\text{zero-modes}) &:= \dim\ker\D_* = \dim\ker\D_*-\dim\ker\overline{\D_*} =: \\
&= \dim\ker\D_*-\dim\operatorname{coker}\D_* =: \ind\D_*.
\end{align}
With the second fact you can write
$$\ind\D_* = \ind\overline{\D_*}\D_*.$$
Therefore you need to look at the index of the square of the Dirac operator. Here you can either immeadiately use the Atiyah-Singer index theorem, or you can prove a physicist version of it, i.e. you can go on to show that it is given by the difference of the traces of the heat-kernels:
$$\ind\overline{\D_*}\D_* = \operatorname{tr}\mathrm{e}^{-s \overline{\D_*}\D_*} - \operatorname{tr}\mathrm{e}^{-s \D_*\overline{\D_*}},$$
and since it is independent of the parameter $s$ can be evaluated at the limit $s\to 0$, giving precisely
$$\ind\overline{\D_*}\D_* = 2 (\operatorname{tr}T_aT_b)\left(\frac{1}{16\pi^2}\int \operatorname{tr}F\wedge F\right) = 2 N k,$$
in the $N$ representation of $U(N)$, with $k$ being the instanton number.
For more information and justification of some of my formulas, these lecture notes on instantons by Vandoren and van Nieuwenhuizen are quite useful.
