That is a very good question!
To answer it, let's focus on a mirror. What surface of a mirror have unique compared to other objects? It is very smooth. If otherwise, it would scatter the incoming light in random direction.
And that is what other objects do! However that is just a macroscopic approximation.
To dig in deeper, we need to think what happens when a photon 'collides' with an atom.
As you know, shining light on an atom can excite it's electron and make them emit photons while they transition to the lower energy levels. This is correct, but not the only scenario that can happen. The photon can also get scattered if it's energy is higher than required, or insufficient. It can also transfer it's energy into vibration of an atom and produce effectively heat, or both can happen simultaneously.
The object will therefore absorb the wavelength that closely matches it's electron energy levels, some will get scattered and some transformed into heat. Some light interactions with materials may be so negligible that photons act as if the object was not there.
The light we see from the object is usually the light reflected from it. But it can also emit it's own light, 'contributing' to the color we perceive. (as hydrogen isotope; tritium).