HGN-Daniel Posted June 1, 2008 Share Posted June 1, 2008 Explosion/fire in their electrical gear room at their H1 datacenter. 9,000 servers down (7500 clients). No people hurt, no servers damaged. <global warming joke> Link to comment Share on other sites More sharing options...
JasonF Posted June 1, 2008 Share Posted June 1, 2008 WOW! Looks like it was their Houston DC. From Doug Erwin: This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost. Here is their forums thread for more info - http://forums.theplanet.com/index.php?showtopic=90185&pid=592379&st=0entry592379 Link to comment Share on other sites More sharing options...
Derek Posted June 1, 2008 Share Posted June 1, 2008 They are doing a very goob and honest job of reporting what is going wrong. +rep to planet. Link to comment Share on other sites More sharing options...
adamnp Posted June 1, 2008 Share Posted June 1, 2008 HLStatsX was affected by this outage, if any of you have customers purchase hlstatsx premium that use logs07, logs08 and logs13.hlstatsx.com and might be asking why they are down, thats why! Link to comment Share on other sites More sharing options...
qwidjib0 Posted June 1, 2008 Share Posted June 1, 2008 Wow, I feel for their techs, it's a pretty unenviable position to have 7500 systems administrators calling you up every hour in fear for their companies. The Planet does seem to be doing a good job with the updates and PR. Didn't ev1's old restored hospital/data center catch fire once before? Link to comment Share on other sites More sharing options...
DougK94 Posted June 1, 2008 Share Posted June 1, 2008 If I remember right, they had a generator catch fire a few years ago. Link to comment Share on other sites More sharing options...
adamnp Posted June 1, 2008 Share Posted June 1, 2008 Yeah, think it was late 2005, Same routine...They are definately handling it better this time around, which is good to see Link to comment Share on other sites More sharing options...
ECF Posted June 1, 2008 Share Posted June 1, 2008 Ouch! 7500 clients down. I hope they are able to get things fixed soon. Link to comment Share on other sites More sharing options...
Derek Posted June 1, 2008 Share Posted June 1, 2008 HLStatsX was affected by this outage, if any of you have customers purchase hlstatsx premium that use logs07, logs08 and logs13.hlstatsx.com and might be asking why they are down, thats why! Yea I found that out too. Sucks! Link to comment Share on other sites More sharing options...
ECF Posted June 1, 2008 Share Posted June 1, 2008 Well, please keep this thread updated if you hear any news from them. Link to comment Share on other sites More sharing options...
JasonF Posted June 1, 2008 Share Posted June 1, 2008 I talk with another Planet forum moderator on IM and he has not heard any news. He has 8 out of 10 servers currently down. He had to sign up for a new server this morning, upload a backup of his company's website and change his name servers to get his site back up to communicate with clients. Link to comment Share on other sites More sharing options...
JasonF Posted June 1, 2008 Share Posted June 1, 2008 The update thread that The Planet has for the outage has this on the bottom of the page. 2493 User(s) are reading this topic Their forums are getting hammered right now and are at a crawl. I am sure they are going to lose many customers over this unfortunately. Wonder if Softlayer's sales team is working overtime. Link to comment Share on other sites More sharing options...
ECF Posted June 1, 2008 Share Posted June 1, 2008 I am sure many provider's sales teams are working ovetime to pick up the slack from this not just softlayer. I feel for the people this is effecting. There is nothing like having your website/servers go down. It brings your business to a halt. Link to comment Share on other sites More sharing options...
HGN-Daniel Posted June 1, 2008 Author Share Posted June 1, 2008 Not to mention that LayeredTech just increased their prices $20-$30.. good day for their competitors I guess Link to comment Share on other sites More sharing options...
DougK94 Posted June 1, 2008 Share Posted June 1, 2008 I never had any servers in Houston, but up until about 2 1/2 years ago had many in Dallas with TP. I am not sure that I would leave them now if this effected me, as it is one of those things that can happen and they appear to be working hard on it and are keeping people informed about it. Yes I would be upset to be down that long, but right now it does not seem like TP/EV1 can be blamed for the outage, just a freak thing that can happen once in a while. Thank goodness no one was injured, as they very well could have been. Over the years, TP/EV1 has had very good uptime on their network. Link to comment Share on other sites More sharing options...
milergenuendraft Posted June 2, 2008 Share Posted June 2, 2008 heres the 5pm update We continue to pursue our plans to provide initial power to our H1 data center this evening. It will take several hours to assure power can be safely restored to the facility. Based on how the initial work goes, we will have more information to provide you in the upcoming hours. We will post another update by 7:30 pm tonight. In the meantime, we have further rerouted both old and new IP addresses for our name servers previously housed in H1. This means that the servers can start resolving IP addresses on both their former and new addresses, and this will alleviate the issues we have been seeing with propagation delay from this address change. ive been useing the planet for many many many years and have never had a problem i agree id be upset at the down time but wouldent switch providers. i also got lucky as hell i just upgraded servers i was in the H1 DC a week ago now im in the H2 DC and didnt even know till i saw this thread and loged into my planet account lol Link to comment Share on other sites More sharing options...
milergenuendraft Posted June 2, 2008 Share Posted June 2, 2008 also just realized its 8:10pm right now so there hasnt been a 7:30update at least not in orbit i live maybe 5 hours from there H1 DC and about an hour from dallas i was woundering why my sales guy was so busy lol Link to comment Share on other sites More sharing options...
adamnp Posted June 2, 2008 Share Posted June 2, 2008 This will bring many new discounts by their sales team in the coming wake to make up for the issues at hand, for the people their services serve. Unfortunate, but definately can happen anyday anytime to any of the dc's any of us host in :{ Link to comment Share on other sites More sharing options...
milergenuendraft Posted June 2, 2008 Share Posted June 2, 2008 they finally put an update up at 8:30pm they got part of there stuff back up but no power to the H1 DC yet and they said there gonna try to have another update around 10pm Link to comment Share on other sites More sharing options...
JasonF Posted June 2, 2008 Share Posted June 2, 2008 I had to call SL about a box I had and they said their sales team are working overtime due to the issue at TP and LayeredTech's price increase. Link to comment Share on other sites More sharing options...
swish Posted June 2, 2008 Share Posted June 2, 2008 I bet SL is lovin it. Link to comment Share on other sites More sharing options...
milergenuendraft Posted June 2, 2008 Share Posted June 2, 2008 hey i have the 10PM update its good and bad well maybe not super good but its a good start lol As previously committed, I would like to provide an update on where we stand following yesterday's explosion in our H1 data center. First, I would like to extend my sincere thanks for your patience during the past 28 hours. We are acutely aware that uptime is critical to your business, and you have my personal commitment that The Planet team will continue to work around the clock to restore your service. As you have read, we have begun receiving some of the equipment required to start repairs. While no customer servers have been damaged or lost, we have new information that damage to our H1 data center is worse than initially expected. Three walls of the electrical equipment room on the first floor blew several feet from their original position, and the underground cabling that powers the first floor of H1 was destroyed. There is some good news, however. We have found a way to get power to Phase 2 (upstairs, second floor) of the data center and to restore network connectivity. We will be powering up the air conditioning system and other necessary equipment within the next few hours. Once these systems are tested, we will begin bringing the 6,000 servers online. It will take four to five hours to get them all running. We have brought in additional support from Dallas to have more hands and eyes on site to help with any servers that may experience problems. The call center has also brought in double staff to handle the increase in tickets we're expecting. Hopefully by sunrise tomorrow Phase 2 will be well on its way to full production. Let me next address Phase 1 (first floor) of the data center and the affected 3,000 servers. The news is not as good, and we were not as lucky. The damage there was far more extensive, and we have a bigger challenge that will require a two-step process. For the first step, we have designed a temporary method that we believe will bring power back to those servers sometime tomorrow evening, but the solution will be temporary. We will use a generator to supply power through next weekend when the necessary gear will be delivered to permanently restore normal utility power and our battery backup system. During the upcoming week, we will be working with those customers to resolve issues. We know this may not be a satisfactory solution for you and your business but at this time, it is the best we can do. We understand that you will be due service credits based on our Service Level Agreement. We will proactively begin providing those following the restoration of service, which is our number priority, so please bear with us until this has been completed. I recognize that this is not all good news. I can only assure you we will continue to utilize every means possible to fully restore service. I plan to have an audio update tomorrow evening. Until then, Douglas J. Erwin Chairman & Chief Executive Officer looks liek they will have 2/3rds up and on a more normal power system the other bit will be southern enginered with a generator lmao they busted some serious A$$ to get that far in 28 hours, thats crazy that it moved the walls that far i wanna know what blew up lol there insurance company didnt see this one coming lol Link to comment Share on other sites More sharing options...
Jdubz31 Posted June 2, 2008 Share Posted June 2, 2008 FYI, the Planet has posted a better update status page. http://service-update.theplanet.com/ Looks like H1 Phase 2 is up, and Phase 1 is coming up as I write this. Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.