DreamHost Blog » Power Outage Update

Power Outage Update

September 12, 2005 on 6:38 pm | In Foobars, Insider View, Updates by Josh Jones |

Evacuation!

Okay, this sucked. Literally 10 seconds after Nate posted that last entry, we received an announcement over our loud speakers that our building was being evacuated and we had to leave immediately! (Also, sorry about that post being temporarily taken down, Nate thought it would be better to remove it since it was so dumb, but methinks better to just add something else!)

Unfortunately, shortly thereafter we figured out (thanks to EVDO!) our sites were no longer accessible, as our three network providers Level 3, Global Crossing, and Mzima were all down. Our physical servers were okay, and probably really happy sitting in the data center (we were no longer able to access) with nothing to do.

But that didn’t last long. Shortly thereafter the entire building where our data center is located’s back-up generators (there are SUPPOSED to be four) stopped working, and all power was gone. We were able to get back into our data center then, and it was like the day after tomorrow or something. Really creepy just walking through rows of dark, quiet, dead server after dark, quiet, dead server. We’ll post some pics from that soon..

So, now, finally power is back up and servers are being restored. We lost a net app and had to replace the head unit from a hot spare. A few servers were of course wonky, and right now three bad cabinet switches are still being worked on (so many web sites ARE still down).

There will be more posts soon, and hopefully everything will be up within the next hour.

Also, just FYI for people who don’t know, we keep http://status.dreamhost.com/ in San Jose, completely off our network, just to update people in times like this when our panel (still down) and everything is caput. This is actually the first time we’ve used it!

Now to go update it…

167 Comments

Pages: [1] 2 3 4 » Show All

1

Any updates?

Comment by Shaun Kelly — September 12, 2005 #
2

You guys are doing a heck of job -

/ *Edit this out*
and please don’t make me look like a fool in front of paying customers for a 3!rd time -

Get that San Jose office a big fat fiber pipe that can co-locate **at least mail** MAIL is critical

end edit/*

Comment by ecco loco — September 12, 2005 #
3

Any updates indeed…. it is disappointing that DH is still down, all these hours after the power outage.

Comment by Kyle Cordes — September 12, 2005 #
4

Yes, this kind of down time for mail REALLY needs to be dealt with. I can easily live with losing the servers, but having my mail go down is extremely bad.

Comment by Nicholas Rubenstein — September 12, 2005 #
5

Get Gmail! :D

Comment by GF — September 12, 2005 #
6

My site appears to be back, but is returning a MySQL error. I can reach all the other pages fine, but the php pages (like the index) are poop.

Comment by Chris — September 12, 2005 #
7

Well done on getting everying (mainly) back together again. I had to laugh, as I was just about to start a meeting introducing some people to the wonderful dreamhost. We had a power failure in Adealide, South Australia for around 5 hours two weeks ago, which was the final straw in finding external hosting. So there I was about to demonstrate how much better off we would all be with this new host and … nothing :)

The status site saved my butt, and all concerned at this end are happy enough.

cheers all

Comment by Shaun Branden — September 12, 2005 #
8

updates? one of my sites i still down :(

Comment by Juan — September 12, 2005 #
9

Wow, thanks for status.dreamhost.com! I’m still waiting for my site to come back up, I hope everything is going well over there at Dreamhost HQ, it seems like you all are handling the situation very well.

Comment by C Montoya — September 12, 2005 #
10

Does anyone know if email sent to dreamhost customers during the outage is now completely lost? Did outside senders to dreamhost recipients receive a delivery error message?

Comment by Galen — September 12, 2005 #
11

We’ve all got gmail, yahoo ,hotmail, ICQ, this that and the next waste of time( well at least some of us)

When the power is off ..

we don’t get email forwarding, autoresponding, etc etc -our stupid little blackberry gets nothing our pathetic palm SYNCS nothing and I’m sure the next IPOD will be worthless also

Comment by ecco loco — September 12, 2005 #
12

Yes, status.dreamhost.com has been very informative/helpful - and so has Gmail ;)

Comment by Matthom — September 12, 2005 #
13

Uh oh.. everything is more or less up, but the building JUST told us.. the power MAY GO AGAIN.. the generators are running now, but we’re on UPS.

IF we do go down again, it’ll be much quicker coming back up, and hopefully the building guys are just scaring us, and us in turn are just scaring you.

Stay tuned.. to status.dreamhost.com! Blog.dreamhost.com would go down if power goes..

Comment by Josh — September 12, 2005 #
14

“Everything” doesn’t seem to be including my sites or mail. Maybe a little more detail on the status site would be helpful.

Comment by Chuck — September 12, 2005 #
15

Hi Guys,
Looks like the mail servers / mysql servers my sites use are still out of commission.
Any word when we’ll have service again? My customers are getting cranky.
Hope you’ve not been to badly affected.

Relevant Domains:
http://www.flickkerbug.com (email especially!)
http://www.accesshealthcare.co.nz (database especially)
http://www.pacificpassages.com (email especially!)
http://www.emporio.co.nz (email especially!)

Cheers
Cam

Comment by Cam — September 12, 2005 #
16

Every time I find a new host, my site goes down shortly. Sometimes the server is hacked because of poor security (not to hack my site, but they were shared), but most often because I chose crappy providers who just have tons of downtime.

So I finally found the best hosting company in the world, and they have a citywide power outage!

I take full responsibility for putting my cursed web site on your servers. Sorry. =)

Having helped run an ISP, I can understand that no matter how perfectly you set things up, something WILL go wrong. I can also say from experience, that it is really cool to at least have an actual news event to explain your outage, rather than having to just say, “oops.”

(It is also REALLY cool for it to be someone ELSE’s problem, rather than hosting my own site!)

Comment by Christen — September 12, 2005 #
17

Hey there.

The nice thing is that my site, or at least the architecture thereof, is up and going. However, the MySQL servers aren’t talking to my site — which is bad news for WordPress. So keep up the good work and hopefully we’ll all be happy tomorrow.

Comment by Thomas — September 12, 2005 #
18

The people are working hard to get everyting up again, ya’ll need to chill out, go out for a walk, take a breath till it’s up again. :)

Comment by GF — September 12, 2005 #
19

Please fix the NTP servers at
66.33.216.{11,130,131,23}. Only two (out
of six) do work correctly (66.33.206.5; 66.201.54.67).

Just trying to help your customer (hopefully, they
wont become ex-customer if you fix it soon).

Thanks and regards.

Comment by Sylvester — September 12, 2005 #
20

I think you guys handled the situation as best you could. You’re not the only hosting company that went down. Thank you for keeping us up to date and for getting customer sites back up before worrying about your own site (except for this one, which has been very informative).

I have 10 domains on my account and my phone and IM were a little nuts today with folks who couldn’t get to the email or sites. It helped that I was able to give them your status page URL.

I think email during the outtage was held and will come through. I had someone mention that they sent me an email 5 minutes after DH went offline around 4 pm (EST) and I did receive it at 7:50 pm.

Comment by Judi Sohn — September 12, 2005 #
21

Arg, I can understand your frustrations and that you generally have system backups and that a city-wide power outage is pretty, well, catastrophic, but I’m thinking you need to invest in more backup generators or something. And test them often so that you know they kick in when they’re supposed to. My clients depend on me, my email being backed up caused a headache on several levels (one got stuck in outbox, a deliverable web graphic expected by end of day, juuuust as things went down - curse my metal body…I wasn’t fast enough…), and I CAN’T USE MY BLOG! Must. Have. Blog.

I know you understand. :D

Is it possible to split up your site into a couple different cities so that, say, mail doesn’t go down when websites do and vice versa? I could stay sane that way. Well, I hope we all learn something from this anyway. (I still like you guys.)

Lynne

Comment by Lynne — September 12, 2005 #
22

Well my web sites (and e-mail, announcement lists, blogs, etc.) are still down. :(

Can someone please verify that our e-mail is at least bouncing back to senders now rather than going into a black hole (as mine did in 2003)? Thanks!

Good luck getting everything back up and running.

Comment by Ruby — September 12, 2005 #
23

This happens so rarly that putting servers in another city would be a waste of money and Im sure everyone wanna pay less possible for the hosting…

Comment by GF — September 12, 2005 #
24

Is it possible that a MySQL database could have been erased due to the power outage? One of my databases is empty, and the Wordpress site that uses it is broken, whereas another of my Wordpress sites which uses a different database is working fine.

Comment by C Montoya — September 12, 2005 #
25

My poor students are high and dry, and serves them right for thinking they could leave their assignment books in school because my website always has the assignments :) Hope we’re back up soon, though.

Comment by Delia — September 12, 2005 #
26

Sorry, but your comment has been flagged by the spam filter running on this blog: this might be an error, in which case all apologies. Your comment will be presented to the blog admin who will be able to restore it immediately.
You may want to contact the blog admin via e-mail to notify him.

What is this?

Comment by James — September 12, 2005 #
27

Wow. status.dreamhost.com has been a great tool, especially in finding out why my site and another popular one (halobabies.net) was down. Thanks for the updates!

Comment by PlasmaFire3000 — September 12, 2005 #
28

I’m just glad the service was down for a pretty good reason! Can’t fight major power outages I guess. My web site is now kicking and hopefully my email service will be restored soon ;-)

Comment by Jennifer — September 12, 2005 #
29

Dreamhost should hire the guy from DirectNIC. He’s moving to California, you know, and looking for a job.

He managed to keep DirectNIC running - that is, running without a single second of downtime throughout Hurricane Katrina and its aftermath. Dreamhost could take a few lessons from that.

Comment by Daniel Drucker — September 12, 2005 #
30

They must be happy, no hws for today! :D

Comment by James — September 12, 2005 #
31

Site is up, but it is pretty much useless because MySQL is still down. I hope this restart is almost finished.

Comment by Damin — September 12, 2005 #
32

I’m a long-time customer and I have a lot of sites hosted on Dreamhost, many of which are on different servers, using different IPs.

Well, -none- of my sites is up. No Web, no mail, nothing.

And guess what… my main competitor, Dreamhost’s customer from last March, has its only site up and running! :-D

Comment by Low — September 12, 2005 #
33

LOL why not get Harry Potter to help them out while we’re at it?

Comment by James — September 12, 2005 #
34

Actually, I was wrong: my web panel and e-mail are up and running now! Only my websites are still unavailable. We’re getting there…

Comment by Ruby — September 12, 2005 #
35

Or just put it on http://www.blogger.com….

Comment by James — September 12, 2005 #
36

Better yet, Mike - pick a friggin’ place that isn’t on a coast at all - something between a large city and a smaller one, that wouldn’t be a target for hurricanes, floods, landslides, attacks, or rattlesnakes. Why rattlesnakes? Because they are nasty.

Or Boston. Pick Boston. Then I can knock on your door when this stuff happens. ;P

Comment by Lynne — September 12, 2005 #
37

email pleaaaazze :)

Comment by riki — September 12, 2005 #
38

I am no NIC expert, but regarding UPS and such, if power outage is widespread I do not think UPS does any good anyway. Certainly some piece of data equipment down the line would be effected. Sure UPS is important for data integrity and to make it easier to come back on line when networking is restored, but I don’t think it would have saved Dreamhost’s bacon in a case like this.

Comment by Mike M — September 12, 2005 #
39

Canada would be better. Btw http://dreamhost.com/ is back up!

Comment by James — September 12, 2005 #
40

From what people are saying the whole damn pipeline went down, correct?

Decentralize, people, decentralize. Or the rattlesnakes will get us.

Comment by Lynne — September 12, 2005 #
41

Granted, it has only been an hour since the last update, but it is about 30mins too long.

Any idea about when anything will be up?

Email? Database? Sites?

Comment by PaulaO — September 12, 2005 #
42

By the way, can I get some of the crack that these stats are on? On the system stats page it reports that all of my servers have been up 100% of the time in the last day. What’s up with that?

Comment by Ruby — September 12, 2005 #
43

Woot! I’m back up! Thanks Dreamhost! Now to back up my database, just in case.

Comment by Damin — September 12, 2005 #
44

Not fair!

Comment by C Montoya — September 12, 2005 #
45

My webmail is still down:
# MySQL Error: Lost connection to MySQL server during query

:-(

Comment by Anti-pode — September 12, 2005 #
46

My database just came back up too! It’s on the slow side, but it’s there.

Comment by Judi Sohn — September 12, 2005 #
47

Thanks guys. :) We appreciate all of the hard work you’re doing. It means a lot.

Comment by Dede — September 12, 2005 #
48

E-mail and control panel are back, although I agree with Ruby about the crack the stats page is running on. Now if I could just get the sites back. (7:50 pm PT).

Comment by Chuck — September 12, 2005 #
49

This is one of those few instances when having the time on comments would help, so that when Josh says things might go down again we’d have an idea WHEN he said that.

Comment by jsp — September 12, 2005 #
50

[...] Update: Dreamhost has (of course) blogged the event here. [...]

Pingback by LiftPort Staff Blog » Blog Archive » Site Outage — September 12, 2005 #

Pages: [1] 2 3 4 » Show All

Sorry, the comment form is closed at this time.

Power Outage Update

167 Comments

Categories:

Archives:

Meta:

Power Outage Update

167 Comments

Categories:

Archives:

Other Sites

Meta: