There isn't an intrinsic difference between a band insulator and a semiconductor - the difference between the two is one of degree, not of kind.
Generally, we use the term 'semiconductor' to refer to band insulators where either (i) the bandgap is small enough that at room temperature there will be enough thermal excitations to bring sufficient population from the valence to the conduction band for the conductivity to be affected, (ii) the Fermi energy is close enough to either end of the bandgap that there is a nontrivial effect on the population of the relevant band, or (iii) the material is easy enough to dope that we can control where the Fermi energy sits in the bandgap.
Note, in particular, that all of those conditions have a bunch of qualifiers (like "enough") which have no intrinsic meaning: they depend on judgement and on the situation. A material might be classed as an insulator in some conditions and as a semiconductor in others, depending on things like temperature, the tools we have available to dope it, and how small a conductivity we're willing to consider before we impose an 'insulator' status.
As far as silicon and diamond go, they are very similar materials, having closely related chemistry (top of group IV), identical crystal structures, and bandgaps far out of the range where room-temperature excitations can pull a significant population to the conduction band. (In this connection, note that $1\:\mathrm{eV}/k_B = 11,600 \:\rm K$. Going from 1.1 eV to 5.5 eV changes $e^{-k_BT_\mathrm{room}/\Delta E}$ from 2% to 0.4%; that's not trivial but it's just a matter of degree; to the extent that thermal excitations get charge to the conduction band, both materials have similar behaviour.) From what I can tell, though, the main difference between the two is that the Fermi energy of silicon sits much closer to the valence band, whereas in diamond it is closer to the middle of the gap.
More importantly, though, this is a misconception:
Diamond, which is an insulator [...].
Want to see why? Simply google for "diamond semiconductor", which will provide plenty of cases where the semiconductor side of the behaviour is being actively used or considered.