Getting that syncing feeling after an Exchange restore
We've got the whole weekend. What could go wrong?
Who, Me? It's Monday, and this week's column contains another reminder to check that those backups really have worked in an unfortunately synchronized episode of Who, Me?
Our tale comes from a reader we'll call "John" (because that is not his name) and takes place in the glory days of Windows NT 4 and Microsoft Exchange Server 5.5.
John had earned his stripes in the technical support department of a mail-order parts company (remember those halcyon days prior to ecommerce?) and was promoted to into the giddy level of "the IT team."
"Which comprised of me and one other guy," he noted.
Still, it did mean John got to do all manner of exciting work, such as the data processing for the catalog mailings in SQL Server. When this colleague (who was minding the domain and dealing with the foibles of Exchange) left, John was handed the keys to the server room. He even got to go on some courses for a MCSE certification, thanks to a boss who reckoned himself a bit of a technical whizz.
Time passed and nothing much went wrong. The lights stayed on. Computers hummed. John was clearly doing a splendid job.
Right up until, as he put it, "the first domino fell."
"The Exchange server," said John, "which doubled as a Fax gateway (because email wasn't really a thing yet and almost everyone got their account statement via fax – that weekly fax job was literally critical to the company's cash flow), started blue screening."
A look at the code on the Blue Screen of Death (BSOD) hinted that the motherboard might be at fault.
"The boss asks me if I'm up for the task of the Exchange rebuild," recalled John. "I tactfully suggest we get in outside help – 'I know a guy who's great with Exchange!'"
Deal! The plan was to do the work on Friday night. All simple enough – back up the Exchange Server to tape. Back it up again, just in case. Then get cracking on the hardware.
All went well. The new motherboard went in. "We pulled out the 8-way Fax card marveling at its grotesqueness and stuck it on the new motherboard."
By midnight Windows was back. Exchange was back. Everything was service packed. Just the tape restore of the data…
…which worked fine. A few obscure ESEUTIL commands were required and… Exchange wouldn't start. Odd. The other backup was tried. Again, Exchange wouldn't start and bleated about a corrupt database.
By now it was 2am, and John decided to call it a night. After all, there was the whole weekend left to get things running.
It isn't hard to guess what happened next. It turned out that ESEUTIL had sufficient options that the potential for error was vast. "Nothing worked," said John, "nobody had a mailbox anymore."
This was bad. But not a disaster. Backups ran every day (a full one on Sunday and then daily incrementals) so restore those, right? They were restored… fingers were crossed… Exchange was restarted…
No luck. Repairs attempted using ESEUTIL took hours, but nothing worked.
At this point it was 4am on Sunday morning and tools had to be downed for a few hours in favor of sleep.
"Tomorrow arrives," said John, "and this time we try to bring back the machine with the old motherboard because we are running out of ideas. Screwdrivers out. Four hours later same result. Back to the new motherboard because we found some more ESEUTIL settings. Alas. Turns out every single one of the 36 backup tapes were corrupt."
- You need to RTFM, but feel free to use your brain too
- Know the difference between a bin and /bin unless you want a new doorstop
- Whatever you do, don't show initiative if you value your job
- Brute force and whiskey: The solution to all life's problems
A weekend to sort things had turned into a mere seven hours before staff were due to arrive for work.
It being midnight on Sunday, desperation set in.
"What's better than no email service at all? Empty mailboxes!"
And so there was one last effort. Exchange was reinstalled and configured from scratch. The directory was restored. Sure, everyone would come into work to find their mailbox empty, but at least it would be possible to send and receive messages from the outside world.
At 6am, the duo stood at the front of the building, ready to dispense the bad news to staff as they arrived.
"One by one the message is delivered 'your email works but you don't have any of your old emails'."
The responses were mixed: "The best of which was the warehouse manager: 'No worries mate, it was all useless shit anyway.'"
Finally John's boss (the one who reckoned himself a technical whizz) turned up. Half an hour of questions and explanations followed before he accepted John's apologies: "Well, OK, you did everything you could. It's OK. I've got a backup of all my contacts on this…"
He held up his fancy new Palm PDA, one that had dispensed with the cable for a wireless link.
"He smugly walked into the office into and prepared to get all his mailbox back," recalled John.
"Except the sync settings in the Palm software said 'take the most recent'."
"What's newer than Friday's full mailbox? Today's empty one"
And with that the duo beat a hasty retreat. Very much as case of "I'll get my coat…"
Ever received a slapping from the synch settings? Or seen a Friday night just-a-job explode into a weekend of catastrophe? Of course you have, and you should share your tale with an email to Who, Me? ®