Imaginary Landscape
RSS FeedLast week our office T1 stopped working. For a Web company, the loss of our connectivity is a body blow.
It turns out that a new tenant is moving into our building, replacing an auto body shop. This new tenant is a microbrewery, so its arrival is quite popular with staff. I know I'll be happy to replace the too-often smell of fiberglass resin and exhaust with whatever odors result from the beer making process. I mention this because within that space is the main phone "nest" that powers the building and our T1.
As part of the tenant transition process, the landlord had to clean up 8+ years of auto body crud, which involved power washing and sand blasting.
The landlord assured us that it would not be a problem - that they would build a box around the nest of exposed phone wires to keep it safe. Which worked, up until the point where our T1 crapped out.
Upon further inspection, the "box" was not well affixed to the wall surrounding the phone nest. In fact, there was a several inch gap just above our T1 line - which I'm sure resulted in a torrent of water, sand and detritus waterfalling through the gap.
Our disaster recovery plan dictates that we send everyone home when office connectivity goes down so they can work remotely, which is what we did after it appeared the outage might last awhile. A skeleton crew remained at the office to answer phones and coordinate repairs.
I was more than a little skeptical about the repair process. Savvis provides our T1 service, over an AT&T; line. This requires a two-vendor dance. First the call into Savvis, then their coordination with AT&T.; However, it was beautifully choreographed and nothing short of miraculous.
Savvis performed their remote checks, determined it was a problem with the line, ticketed AT&T;, who performed their remote checks and determined it was a problem with the line between us and their local network center. AT&T; dispatched a truck that checked the line all the way to our boxed nest.
The technician needed a different truck, went back to his facility and returned within an hour. Spent a couple hours working on the line (it turns out that water had seeped in between the wires and the sheath of our T1 line). Replaced the line, tested it and all the while remained helpful and committed to the repair.
We were back up and running within 5 hours. Thank you Savvis and AT&T; for stepping up.
Updated 07/14/10 @ 11:20AM CDT by brian
Categories: Business Technology
