Monday, January 24, 2011

Sunday, Rainy Sunday

Got a call at Noon on Sunday. Went like this:
<boss>: <myname> I just got a call from <co-worker> we've got water in the suite, start shutting things down.
<me>: Will do, I'll log in now. Are you on your way to the office?
<boss>: Yup, I'll call you.
<me>: Okay.
10 minutes later
<boss> Yeah, we've got water, shut it down.
<me>: Doing it now, almost all off.
<boss> Okay. We've got standing water in the suite. Not good.
<me> Damn. How's our office?
<boss> Wet.
<me> Alright. See you in a bit

And from then on it was a cold shower. No, really. A pipe in one of our conference rooms, blew the sprinkler head off. Water was gushing down, water water every where, not a drop to drink.

Luckily our office is no longer the server room. So the bank of batteries that got wet have been off for 2 months. I got everything shutdown, thank God for VMWare ESXi. It was awesome to just log into the vCenter server and see all of my servers right there.

This morning wasn't too much of a hassle either. All of my users were back up and running at 9AM. There was still some cleanup to do of old equipment that was on the floor. Although we may have lost one of our new projectors it could've been worse.