Most Internet connections use shared bandwidth at some point in the path, for ADSL this will be at the local exchange. The degree of sharing used to be referred to as the contention ratiocontention ratio.
Basically, if your neighbours become more active at night (their sons have all discovered World of Warcraft?) your speed can be affected.
It is possible for ISPs to use "traffic shaping" to throttle back certain types of traffic but I believe this is not the norm - or at least, ISPs rarely admit it. Some ISPs have been known to restrict usage by people who they believe are making unfair and excessive use of their broadband - but I think this mostly applies to a very small percentage of people.