So the data centre's 'getting a little hot' – at 57°C, that's quite the understatement
You know that thing that's supposed to put out the fire?
On Call Welcome to another edition of The Register's On Call in which incompetence saves a reader's bacon from a close encounter with the frying pan.
Our story takes us back to the halcyon days of 2014 and a medical facility where our hero, Regomised as "Chris", worked.
As has been the trend over recent years, the on-premises data centre for which Chris was responsible was not large ("a couple of rows of servers," he recalled) but still needed cooling and environmental management.
On the night in question, Chris (being on call) was notified that there was a problem with the air conditioning. A co-worker, who lived closer than Chris, has already been in and said "things were getting a little hot."
An understatement, as it turned out.
Chris instructed his colleague to shut down whatever he could and embarked on the journey to the site.
Awaiting him was what remained of his colleague: a sweat monster. Things were indeed "getting a little hot."
"After a few well-placed words from myself, we started to evaluate it even closer."
The AC wasn't just broken, it was actually blowing hot air into the room, heating things up even further. "The room was close to 135° Fahrenheit."
That's over 57° Celsius. Not ideal. Forget clean shutdowns; the servers needed to be turned off now.
The engineering team was consulted on exactly how bad hitting the Big Red Button would be, although in the end the new power distribution system (installed recently) was used "to bring servers down quicker but not cleaner."
How did Chris know the exact temperature reached in the room? "Half the sprinkler heads popped," he told us.
Which raises the question of why Chris and his co-worker were not treated to a soaking by a suppression system convinced things were on fire. "The engineering people had done a test at one time and forgot to close a valve," he told us.
- See that last line in the access list? Yeah, that means you don't have an access list
- Breaking Bad or just a bad breakpoint? That feeling when your predecessor is BASIC
- Malware and Trojans, but there's only one horse the boss man wants to hear about
- Exsparko-destructus! What happens when wand waving meets extremely poor wiring
"So instead of dousing servers and storage with water, it all went out an exit pipe."
"I was very glad they had not made that work right," he confessed.
"Sadly, though, they did tell me later on they would fix it so it would work later.
It took a 23-hour stint on site to bring everything back online.
"Hopefully," said Chris, "they never 'fixed' the sprinkler after I left."
We can but hope that whatever the eventual repair was, the engineers took the opportunity to move on from just water in the DC.
Ever had your bacon saved from a frying pan encounter thanks to negligence of others? Share your story with an email to On Call. ®