Jump to content

The Planet Down

HGN-Daniel

By HGN-Daniel
in Off-Topic

Recommended Posts

JasonF Super Member

JasonF

Posted
Posted

WOW! Looks like it was their Houston DC.

 

From Doug Erwin:

This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost.

 

Here is their forums thread for more info - http://forums.theplanet.com/index.php?showtopic=90185&pid=592379&st=0entry592379

Link to comment
Share on other sites
qwidjib0 Conversation Superstar

qwidjib0

Posted
Posted

Wow, I feel for their techs, it's a pretty unenviable position to have 7500 systems administrators calling you up every hour in fear for their companies. The Planet does seem to be doing a good job with the updates and PR.

 

Didn't ev1's old restored hospital/data center catch fire once before?

Link to comment
Share on other sites
Derek Super Member

Derek

Posted
Posted
HLStatsX was affected by this outage, if any of you have customers purchase hlstatsx premium that use logs07, logs08 and logs13.hlstatsx.com and might be asking why they are down, thats why!

 

 

Yea I found that out too. Sucks!

Link to comment
Share on other sites
JasonF Super Member

JasonF

Posted
Posted

I talk with another Planet forum moderator on IM and he has not heard any news. He has 8 out of 10 servers currently down. He had to sign up for a new server this morning, upload a backup of his company's website and change his name servers to get his site back up to communicate with clients.

Link to comment
Share on other sites
JasonF Super Member

JasonF

Posted
Posted

The update thread that The Planet has for the outage has this on the bottom of the page.

 

2493 User(s) are reading this topic

 

Their forums are getting hammered right now and are at a crawl. I am sure they are going to lose many customers over this unfortunately.

 

Wonder if Softlayer's sales team is working overtime.

Link to comment
Share on other sites
ECF Super Member

ECF

Posted
Posted

I am sure many provider's sales teams are working ovetime to pick up the slack from this not just softlayer.

 

I feel for the people this is effecting. There is nothing like having your website/servers go down. It brings your business to a halt.

Link to comment
Share on other sites
DougK94 Super Member

DougK94

Posted
Posted

I never had any servers in Houston, but up until about 2 1/2 years ago had many in Dallas with TP. I am not sure that I would leave them now if this effected me, as it is one of those things that can happen and they appear to be working hard on it and are keeping people informed about it. Yes I would be upset to be down that long, but right now it does not seem like TP/EV1 can be blamed for the outage, just a freak thing that can happen once in a while. Thank goodness no one was injured, as they very well could have been. Over the years, TP/EV1 has had very good uptime on their network.

Link to comment
Share on other sites
milergenuendraft Conversation Starter

milergenuendraft

Posted
Posted

heres the 5pm update We continue to pursue our plans to provide initial power to our H1 data center this evening. It will take several hours to assure power can be safely restored to the facility. Based on how the initial work goes, we will have more information to provide you in the upcoming hours. We will post another update by 7:30 pm tonight.

 

In the meantime, we have further rerouted both old and new IP addresses for our name servers previously housed in H1. This means that the servers can start resolving IP addresses on both their former and new addresses, and this will alleviate the issues we have been seeing with propagation delay from this address change.

 

 

ive been useing the planet for many many many years and have never had a problem i agree id be upset at the down time but wouldent switch providers. i also got lucky as hell i just upgraded servers i was in the H1 DC a week ago now im in the H2 DC and didnt even know till i saw this thread and loged into my planet account lol

Link to comment
Share on other sites
adamnp Super Member

adamnp

Posted
Posted

This will bring many new discounts by their sales team in the coming wake to make up for the issues at hand, for the people their services serve.

 

Unfortunate, but definately can happen anyday anytime to any of the dc's any of us host in :{

Link to comment
Share on other sites
milergenuendraft Conversation Starter

milergenuendraft

Posted
Posted

hey i have the 10PM update its good and bad well maybe not super good but its a good start lol

 

As previously committed, I would like to provide an update on where we stand following yesterday's explosion in our H1 data center. First, I would like to extend my sincere thanks for your patience during the past 28 hours. We are acutely aware that uptime is critical to your business, and you have my personal commitment that The Planet team will continue to work around the clock to restore your service.

 

As you have read, we have begun receiving some of the equipment required to start repairs. While no customer servers have been damaged or lost, we have new information that damage to our H1 data center is worse than initially expected. Three walls of the electrical equipment room on the first floor blew several feet from their original position, and the underground cabling that powers the first floor of H1 was destroyed.

 

There is some good news, however. We have found a way to get power to Phase 2 (upstairs, second floor) of the data center and to restore network connectivity. We will be powering up the air conditioning system and other necessary equipment within the next few hours. Once these systems are tested, we will begin bringing the 6,000 servers online. It will take four to five hours to get them all running.

 

We have brought in additional support from Dallas to have more hands and eyes on site to help with any servers that may experience problems. The call center has also brought in double staff to handle the increase in tickets we're expecting. Hopefully by sunrise tomorrow Phase 2 will be well on its way to full production.

 

Let me next address Phase 1 (first floor) of the data center and the affected 3,000 servers. The news is not as good, and we were not as lucky. The damage there was far more extensive, and we have a bigger challenge that will require a two-step process. For the first step, we have designed a temporary method that we believe will bring power back to those servers sometime tomorrow evening, but the solution will be temporary. We will use a generator to supply power through next weekend when the necessary gear will be delivered to permanently restore normal utility power and our battery backup system. During the upcoming week, we will be working with those customers to resolve issues.

 

We know this may not be a satisfactory solution for you and your business but at this time, it is the best we can do.

 

We understand that you will be due service credits based on our Service Level Agreement. We will proactively begin providing those following the restoration of service, which is our number priority, so please bear with us until this has been completed.

 

I recognize that this is not all good news. I can only assure you we will continue to utilize every means possible to fully restore service.

 

I plan to have an audio update tomorrow evening.

 

Until then,

 

Douglas J. Erwin

Chairman & Chief Executive Officer

 

 

looks liek they will have 2/3rds up and on a more normal power system the other bit will be southern enginered with a generator lmao they busted some serious A$$ to get that far in 28 hours, thats crazy that it moved the walls that far i wanna know what blew up lol there insurance company didnt see this one coming lol

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

Go to topic listing

  • Who's Online   0 Members, 0 Anonymous, 7 Guests (See full list)

    • There are no registered users currently online
×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue. Terms of Use