Take the example:
$$\ce{N2(g) + 3H2(g) <-> 2NH3(g)}$$
I understand that if I increase the pressure of the system, it'll shift towards the $\ce{NH3}$ side. This is because of the reaction rate increasing (increased collisions) more for the forward reaction than the reverse reaction. What I don't understand is why a decrease in pressure of the system would shift the equilibrium to $\ce{N2(g) + 3H2(g)}$. There is overall less collisions, but there's still more moles of reactants to collide and less molecules for $\ce{NH3}$ to collide and decompose.