There is a favorite question on graduate schools' qualifying exams in physics,

Why the sky is blue?

The answer is the Rayleigh scattering. Shorter wavelength photons are more likely to change the direction in the atmosphere which is why the bluer, shorter-wavelength light is overrepresented in the light coming from random directions of the sky.

I believe that the same reason "why the sky is blue" is also the reason why "the sky under certain circumstances is not so blue". When the Sun is shining directly on the glass, but not at you, the photons may scatter of the dirt and impurities on the glass. Unlike the air, these impurities are able to change the direction of all photons.

So if the dirt on the glass isn't directly illuminated by the Sun, the photons from generic places of the sky only arrive to your eye if they originated from the solar photons scattered by the atmosphere, and these photons tend to be blue. However, if the glass is directly illuminated, you are receiving lots of photons from the Sun that change their direction while hitting the dirt on the glass, and these are not shifted to the blue end of the spectrum because the scattering off the dirt isn't Rayleigh scattering; it is more color blind. Therefore the color seen in the generic directions becomes greyer.

There is a favorite question on graduate schools' qualifying exams in physics,

Why the sky is blue?

The answer is the Rayleigh scattering. Shorter wavelength photons are more likely to change the direction in the atmosphere which is why the bluer, shorter-wavelength light is overrepresented in the light coming from random directions of the sky.

I believe that the same reason "why the sky is blue" is also the reason why "the sky under certain circumstances is not so blue". When the Sun is shining directly on the glass, but not at you, the photons may scatter of the dirt and impurities on the glass. Unlike the air, these impurities are able to change the direction of all photons.

So if the dirt on the glass isn't directly illuminated by the Sun, the photons from generic places of the sky only arrive to your eye if they originated from the solar photons scattered by the atmosphere, and these photons tend to be blue. However, if the glass is directly illuminated, you are receiving lots of photons from the Sun that change their direction while hitting the dirt on the glass, and these are not shifted to the blue end of the spectrum because the scattering off the dirt isn't Rayleigh scattering; it is more color blind. Therefore the color seen in the generic directions becomes greyer.

Note that I am pretty much saying that the right question should have been the opposite to yours: why the sky is blue in the first situation. It's the "default state" for light coming from some directions to be color-neutral i.e. "grey". This occurs when the photons' directions are being changed pretty much independently of their color or wavelength, and this condition is obeyed when the scattering off the impurities on the glass is the dominant source of light from a direction. On the other hand, it's "extraordinary" for light not to be color-neutral i.e. to be blue, and that's why it's still the blue sky, and not the grey sky, that deserves a "special" explanation. It's the usual explanation based on the Rayleigh scattering! In this sense, the answer to your question is simply that the "special processes" that make the sky unusual i.e. blue are not dominant in the second situation.