As a crewmember, I can explain why I think this is an "extraordinary circumstances".
Crewmembers, especially flight deck crew, while on duty (this includes the time they spent at a destination) is controlled by many rules when it comes to "rest", how long they need to sleep and when can they drink alcohol, etc. Even rules regarding diving and skydiving are controlled as it might affect the crew health.
However, if the crew did not get the minimum required rest before a flight (usually 8-12 hrs of pure rest) they are considered "NOT LEGAL" to fly, as their performance might not be optimum and might cause a catastrophe, especially if the flight is a long haul flight (EU to US for example).
Lack of sleep can lead to something known in the aviation as "unsafe level of fatigue", which is known to be the reason of many incidents and accidents. Therefore, many countries, including the US and all of the EU and many other parts of the world, have very strict rules when it comes to crew fatigue, this is called "fatigue risk management".
I know that it seems "silly" to passengers who were delayed for a couple of hours, but flight crew are usually under great stress when flying and they are responsible for the lives of hundreds of passengers, they know that so they will never take a chance to operate heavy machinery while on an unsafe level of fatigue.
Back to your case, this has happened to me personally, you are sleeping at your room, it's 03:00 after midnight and you have a pickup at 08:00 in the morning, some guest in the hotel decides to smoke, fire alarms goes off, speakers all over the hotel ask the guests to leave the hotel, fire trucks arrive, etc. etc., the whole process will take anything from 1 hour to 2 hours.
The crew goes back to the room, can't get back to sleep, they call the crew control/flight operations and report fatigue, the crew control calculates the rest required and informs the crew, the passengers usually gets informed if the delay is significant and you know the rest of the story.
This is only happens if the crew are spending a night somewhere away from their base, if this happened at a crew base then some other crew will cover the flight.
Now, is this considered an "extraordinary circumstance"? my opinion: YES it is!
Who to blame? well, the guy who smoked? a faulty smoke detector? but for sure not the airline as they were simply ensuring the safety of the passengers.