Power Outage Update

September 12, 2005 on 6:38 pm | In Foobars, Insider View, Updates by Josh Jones |

Evacuation!

Okay, this sucked. Literally 10 seconds after Nate posted that last entry, we received an announcement over our loud speakers that our building was being evacuated and we had to leave immediately! (Also, sorry about that post being temporarily taken down, Nate thought it would be better to remove it since it was so dumb, but methinks better to just add something else!)

Unfortunately, shortly thereafter we figured out (thanks to EVDO!) our sites were no longer accessible, as our three network providers Level 3, Global Crossing, and Mzima were all down. Our physical servers were okay, and probably really happy sitting in the data center (we were no longer able to access) with nothing to do.

But that didn’t last long. Shortly thereafter the entire building where our data center is located’s back-up generators (there are SUPPOSED to be four) stopped working, and all power was gone. We were able to get back into our data center then, and it was like the day after tomorrow or something. Really creepy just walking through rows of dark, quiet, dead server after dark, quiet, dead server. We’ll post some pics from that soon..

So, now, finally power is back up and servers are being restored. We lost a net app and had to replace the head unit from a hot spare. A few servers were of course wonky, and right now three bad cabinet switches are still being worked on (so many web sites ARE still down).

There will be more posts soon, and hopefully everything will be up within the next hour.

Also, just FYI for people who don’t know, we keep http://status.dreamhost.com/ in San Jose, completely off our network, just to update people in times like this when our panel (still down) and everything is caput. This is actually the first time we’ve used it!

Now to go update it…

168 Responses to “Power Outage Update”

  1. Shaun Kelly Says:

    Any updates?

  2. ecco loco Says:

    You guys are doing a heck of job -

    / *Edit this out*
    and please don’t make me look like a fool in front of paying customers for a 3!rd time -

    Get that San Jose office a big fat fiber pipe that can co-locate **at least mail** MAIL is critical

    end edit/*

  3. Kyle Cordes Says:

    Any updates indeed…. it is disappointing that DH is still down, all these hours after the power outage.

  4. Nicholas Rubenstein Says:

    Yes, this kind of down time for mail REALLY needs to be dealt with. I can easily live with losing the servers, but having my mail go down is extremely bad.

  5. GF Says:

    Get Gmail! :D

  6. Chris Says:

    My site appears to be back, but is returning a MySQL error. I can reach all the other pages fine, but the php pages (like the index) are poop.

  7. Shaun Branden Says:

    Well done on getting everying (mainly) back together again. I had to laugh, as I was just about to start a meeting introducing some people to the wonderful dreamhost. We had a power failure in Adealide, South Australia for around 5 hours two weeks ago, which was the final straw in finding external hosting. So there I was about to demonstrate how much better off we would all be with this new host and … nothing :)

    The status site saved my butt, and all concerned at this end are happy enough.

    cheers all

  8. Juan Says:

    updates? one of my sites i still down :(

  9. C Montoya Says:

    Wow, thanks for status.dreamhost.com! I’m still waiting for my site to come back up, I hope everything is going well over there at Dreamhost HQ, it seems like you all are handling the situation very well.

  10. Galen Says:

    Does anyone know if email sent to dreamhost customers during the outage is now completely lost? Did outside senders to dreamhost recipients receive a delivery error message?

  11. ecco loco Says:

    We’ve all got gmail, yahoo ,hotmail, ICQ, this that and the next waste of time( well at least some of us)

    When the power is off ..

    we don’t get email forwarding, autoresponding, etc etc -our stupid little blackberry gets nothing our pathetic palm SYNCS nothing and I’m sure the next IPOD will be worthless also

  12. Matthom Says:

    Yes, status.dreamhost.com has been very informative/helpful - and so has Gmail ;)

  13. Josh Says:

    Uh oh.. everything is more or less up, but the building JUST told us.. the power MAY GO AGAIN.. the generators are running now, but we’re on UPS.

    IF we do go down again, it’ll be much quicker coming back up, and hopefully the building guys are just scaring us, and us in turn are just scaring you.

    Stay tuned.. to status.dreamhost.com! Blog.dreamhost.com would go down if power goes..

  14. Chuck Says:

    “Everything” doesn’t seem to be including my sites or mail. Maybe a little more detail on the status site would be helpful.

  15. Cam Says:

    Hi Guys,
    Looks like the mail servers / mysql servers my sites use are still out of commission.
    Any word when we’ll have service again? My customers are getting cranky.
    Hope you’ve not been to badly affected.

    Relevant Domains:
    http://www.flickkerbug.com (email especially!)
    http://www.accesshealthcare.co.nz (database especially)
    http://www.pacificpassages.com (email especially!)
    http://www.emporio.co.nz (email especially!)

    Cheers
    Cam

  16. Christen Says:

    Every time I find a new host, my site goes down shortly. Sometimes the server is hacked because of poor security (not to hack my site, but they were shared), but most often because I chose crappy providers who just have tons of downtime.

    So I finally found the best hosting company in the world, and they have a citywide power outage!

    I take full responsibility for putting my cursed web site on your servers. Sorry. =)

    Having helped run an ISP, I can understand that no matter how perfectly you set things up, something WILL go wrong. I can also say from experience, that it is really cool to at least have an actual news event to explain your outage, rather than having to just say, “oops.”

    (It is also REALLY cool for it to be someone ELSE’s problem, rather than hosting my own site!)

  17. Thomas Says:

    Hey there.

    The nice thing is that my site, or at least the architecture thereof, is up and going. However, the MySQL servers aren’t talking to my site — which is bad news for WordPress. So keep up the good work and hopefully we’ll all be happy tomorrow.

  18. GF Says:

    The people are working hard to get everyting up again, ya’ll need to chill out, go out for a walk, take a breath till it’s up again. :)

  19. Sylvester Says:

    Please fix the NTP servers at
    66.33.216.{11,130,131,23}. Only two (out
    of six) do work correctly (66.33.206.5; 66.201.54.67).

    Just trying to help your customer (hopefully, they
    wont become ex-customer if you fix it soon).

    Thanks and regards.

  20. Judi Sohn Says:

    I think you guys handled the situation as best you could. You’re not the only hosting company that went down. Thank you for keeping us up to date and for getting customer sites back up before worrying about your own site (except for this one, which has been very informative).

    I have 10 domains on my account and my phone and IM were a little nuts today with folks who couldn’t get to the email or sites. It helped that I was able to give them your status page URL.

    I think email during the outtage was held and will come through. I had someone mention that they sent me an email 5 minutes after DH went offline around 4 pm (EST) and I did receive it at 7:50 pm.

  21. Lynne Says:

    Arg, I can understand your frustrations and that you generally have system backups and that a city-wide power outage is pretty, well, catastrophic, but I’m thinking you need to invest in more backup generators or something. And test them often so that you know they kick in when they’re supposed to. My clients depend on me, my email being backed up caused a headache on several levels (one got stuck in outbox, a deliverable web graphic expected by end of day, juuuust as things went down - curse my metal body…I wasn’t fast enough…), and I CAN’T USE MY BLOG! Must. Have. Blog.

    I know you understand. :D

    Is it possible to split up your site into a couple different cities so that, say, mail doesn’t go down when websites do and vice versa? I could stay sane that way. Well, I hope we all learn something from this anyway. (I still like you guys.)

    Lynne

  22. Ruby Says:

    Well my web sites (and e-mail, announcement lists, blogs, etc.) are still down. :(

    Can someone please verify that our e-mail is at least bouncing back to senders now rather than going into a black hole (as mine did in 2003)? Thanks!

    Good luck getting everything back up and running.

  23. GF Says:

    This happens so rarly that putting servers in another city would be a waste of money and Im sure everyone wanna pay less possible for the hosting…

  24. C Montoya Says:

    Is it possible that a MySQL database could have been erased due to the power outage? One of my databases is empty, and the Wordpress site that uses it is broken, whereas another of my Wordpress sites which uses a different database is working fine.

  25. Delia Says:

    My poor students are high and dry, and serves them right for thinking they could leave their assignment books in school because my website always has the assignments :) Hope we’re back up soon, though.

  26. James Says:

    Sorry, but your comment has been flagged by the spam filter running on this blog: this might be an error, in which case all apologies. Your comment will be presented to the blog admin who will be able to restore it immediately.
    You may want to contact the blog admin via e-mail to notify him.

    What is this?

  27. PlasmaFire3000 Says:

    Wow. status.dreamhost.com has been a great tool, especially in finding out why my site and another popular one (halobabies.net) was down. Thanks for the updates!

  28. Jennifer Says:

    I’m just glad the service was down for a pretty good reason! Can’t fight major power outages I guess. My web site is now kicking and hopefully my email service will be restored soon ;-)

  29. Daniel Drucker Says:

    Dreamhost should hire the guy from DirectNIC. He’s moving to California, you know, and looking for a job.

    He managed to keep DirectNIC running - that is, running without a single second of downtime throughout Hurricane Katrina and its aftermath. Dreamhost could take a few lessons from that.

  30. James Says:

    They must be happy, no hws for today! :D

  31. Damin Says:

    Site is up, but it is pretty much useless because MySQL is still down. I hope this restart is almost finished.

  32. Low Says:

    I’m a long-time customer and I have a lot of sites hosted on Dreamhost, many of which are on different servers, using different IPs.

    Well, -none- of my sites is up. No Web, no mail, nothing.

    And guess what… my main competitor, Dreamhost’s customer from last March, has its only site up and running! :-D

  33. James Says:

    LOL why not get Harry Potter to help them out while we’re at it?

  34. Ruby Says:

    Actually, I was wrong: my web panel and e-mail are up and running now! Only my websites are still unavailable. We’re getting there…

  35. James Says:

    Or just put it on http://www.blogger.com….

  36. Lynne Says:

    Better yet, Mike - pick a friggin’ place that isn’t on a coast at all - something between a large city and a smaller one, that wouldn’t be a target for hurricanes, floods, landslides, attacks, or rattlesnakes. Why rattlesnakes? Because they are nasty.

    Or Boston. Pick Boston. Then I can knock on your door when this stuff happens. ;P

  37. riki Says:

    email pleaaaazze :)

  38. Mike M Says:

    I am no NIC expert, but regarding UPS and such, if power outage is widespread I do not think UPS does any good anyway. Certainly some piece of data equipment down the line would be effected. Sure UPS is important for data integrity and to make it easier to come back on line when networking is restored, but I don’t think it would have saved Dreamhost’s bacon in a case like this.

  39. James Says:

    Canada would be better. Btw http://dreamhost.com/ is back up!

  40. Lynne Says:

    From what people are saying the whole damn pipeline went down, correct?

    Decentralize, people, decentralize. Or the rattlesnakes will get us.

  41. PaulaO Says:

    Granted, it has only been an hour since the last update, but it is about 30mins too long.

    Any idea about when anything will be up?

    Email? Database? Sites?

  42. Ruby Says:

    By the way, can I get some of the crack that these stats are on? On the system stats page it reports that all of my servers have been up 100% of the time in the last day. What’s up with that?

  43. Damin Says:

    Woot! I’m back up! Thanks Dreamhost! Now to back up my database, just in case.

  44. C Montoya Says:

    Not fair!

  45. Anti-pode Says:

    My webmail is still down:
    # MySQL Error: Lost connection to MySQL server during query

    :-(

  46. Judi Sohn Says:

    My database just came back up too! It’s on the slow side, but it’s there.

  47. Dede Says:

    Thanks guys. :) We appreciate all of the hard work you’re doing. It means a lot.

  48. Chuck Says:

    E-mail and control panel are back, although I agree with Ruby about the crack the stats page is running on. Now if I could just get the sites back. (7:50 pm PT).

  49. jsp Says:

    This is one of those few instances when having the time on comments would help, so that when Josh says things might go down again we’d have an idea WHEN he said that.

  50. LiftPort Staff Blog » Blog Archive » Site Outage Says:

    [...] Update: Dreamhost has (of course) blogged the event here. [...]

  51. Bruce Says:

    All three of my web sites are still down hours after power has been restored…no updates on the emergency status page or blog…what gives ??

  52. BesterP12 Says:

    My sites are back up, but:

    Site Temporarily Unavailable
    We apologize for the inconvenience. Please contact the webmaster/ tech support immediately to have them rectify this.

    error id: “bad_httpd_conf”

    Glorianon.com
    graphicbuds.com
    tamlincat.com
    Besterp12.com

    It’s great to see some sign of life though.

  53. Ray Says:

    Question…

    The boxes should not only be on generator power, but also on UPS. If the UPS start to run out of battery power, all the servers should begin their shutdown before the UPS lose power.

    I assume something went wrong? No UPS software installed correctly? Obviously when machines go down there is always a problem with something not coming up, but that’s normally only if you remove the drives or work on the hardware or move the servers. Even if you lose a drive you should have multiple hot spares setup to rebuild without loss of data.

    I am very interested in knowing why you ‘lost’ some equipment and why some DB are not coming up. I do computer adminstration for a living and we have gone through similar problems but when the generator runs out, then the huge UPS battery back for our floor kicks in. We could probably go 24 hours without power, depending on temperature.

    If you could answer some of these questions I would appreciate it.

    Ray

  54. Shaun Kelly Says:

    Um. I realize y’all are probably having a grand ole time, having been in the trenches when an international company is down and my boss is outside my cube running interference for his boss, who is talking to his boss because of what I haven’t gotten back online yet….

    But. Once you have it back up, I’m gonna expect an explanation, including who owns the generators, how often they test, who owns the UPS, how often they get tested, and an explanation of why the servers either weren’t shudown gracefully, or why there wasn’t sufficient backup servers or cluster resources to bring everything up in a limping mode while the hardware that wouldn’t power up gets fixed.

    From here, it looks to me like you lost power on running servers due to lack of an automated shutdown, or inadequate UPS capacity, neither of which are things I want to keep paying you to not plan for, and I look forward to a post-mortem report that says different.

  55. Andy Says:

    Hey guys - time for a new update post - this is 50 comments… also - can you confirm if email has been bouncing or is stored…?

  56. Rome Says:

    Hey, folks, just an FYI about UPSs — their purpose isn’t to keep electrical devices constantly running during a power failure. That’s what generators are for. UPSs are big batteries that give you those precious 5 minutes to properly shut down connected devices instead of grabbing that extension cord in both hands and yanking with all your back muscles.

  57. Brian Says:

    DH - you guys have had a busy day. Get a beer, run over the lessons learned so you can do better next time.

    Everyone else who is kvetching - what do you expect? You don’t pay that much for the service. DH is for the money the best provider we’ve had.

    If you want 24×7 uptime, redundant network paths and etc you’re can get that - but you’ll pay for it.

  58. Missing Monkey Head Says:

    Not Terrah!

    The whole of neurobashing.com was off the grid for a long time today… due to power outages in L.A. ……

  59. Mike M Says:

    Def. DH needs to look deep into what went wrong with their UPS system and why things did not shut down gracefully. I hope they do.

    But IMHO if you are running a Web server for an International company at DH I think you are nuts.

  60. Mike M Says:

    And I meant to say BIG international company. I.e. some Fortune 5000 (if that exists) or the like.

  61. Jad Says:

    Oww! Talk about embarassing!

    Some snotty web guy that is taking over one of my customers sneered at DH today for lacking proper “environmental controls”.

    F@#$ him, but really guys — where’s all the cool data center power backup stuff when you needed it???

    Embarassed, but will get over it soon.. /jad

  62. LowLevel Says:

    Now e-mail works but all my web sites are still down. :( It’s an improvement, though.

  63. Shaun Kelly Says:

    It’s just my personal mail ond domain that’s down, sorry if I wasn’t terribly clear.

    Although I’m suddenly wondering how much Directnic costs….if anyone has generator expertise right now, it’s them :)

  64. james Says:

    everything working but notoriously slow!!

  65. Jason Says:

    My websites are all still down and mail forwarding is down as well. 10 pm central time US. Please update us, thanks.

  66. foo Says:

    I guess I’m a little confused. What’s the point in having big UPS systems and generators if everything goes dark anyway? *Sigh*

  67. Mark Says:

    Cut them a little slack about the server shutdown thing. I’ve had plenty of servers (Dell 2650’s, 6650’s, etc.) just decide to not start after a proper shutdown. Usually the disks decided to give up the ghost. A few times we lost the controllers in the NAS devices, resyncing disks, restoring terabytes from backup. These things take time.

    Keep up the good fight Dreamhost. (and i’d recommend renegotiating your contract with the company in charge of the generators. at the least yell and scream at them for 20 minutes)

    DR’s a pain

  68. Chris Hobbs Says:

    Regarding e-mail non-deliveries - it’s going to be a function of the total time DH is down and the settings on other providers’ servers. On the mail server I run for work we’re set to try to deliver mail for up to 3 days. So as long as a remote server gets its act together within 72 hours, mail we send will be delivered. If I had set the trigger to 6 hours, however, I’d already be bouncing mail that was destined for DH servers. But that would be a very, very unusual scenario. Most MTAs would be set for at least 2-3 days.

    Many systems *will* send back warning reports to local users informing them that mail has not been delivered *yet*, which is not the same as saying it will not be delivered. Again, this is configurable by the remote server admin.

    Hoep this helps alleviate some fears. Chances are, assuming a relatively quick return to normalcy (tick, tick, tick…) all e-mail should eventually be delivered.

  69. Jon Spreadbury Says:

    One domain up (inspiredsnl.com), one down (spreadburys.com), web panel functional. Standard webmail page can’t connect to MySQL; “aftermarket” SquirrelMail is functional. Mail hosts are also down.

  70. BesterP12 Says:

    Okay, FTp access isn’t there which makes sense if they’re working on quid. Sites are still down. Attempts to go to the main page on any sites get the above message, attempts to go anywhere else get 404′d. Hopefully it will all be fixed before tomorrow morning, or I’m really going to be getting complaints. (teamb.tamlincat.com hosts a listing of assignments for the local Jr. High and some of the parents can be very annoying. ONe of the reasons I dropped PHPWebhosting actually. This is the first time DH has given me any problems so I’ll give you the benefit of the doubt and assume you’ll be faster at recovery if this every happens again.

    Heh, also, the panel is back up, but you may want to update the stats at Status->Server Status

    Up Forever
    Up Forever
    Up Forever

  71. LowLevel Says:

    Just a quick report: now all my sites and their MySQL databases are up. Webmail is not (Can’t connect to 10.3.67.49 as squirrelmail).

    Things are definitely moving!

    Back to work, now. I have to publish my daily news! :)

  72. ecco loco Says:

    Get rid of the blog and educate these peePz as to what life really is

    most have no idea what it’s like to be down

    every webhost has downtime
    and always will (deal with it my puppy’s)

    every site will go down (and)
    every site will go back up

    EVERY webhosting service will see this
    EVERY WEBMASTER will deal with this
    alert your customers and life will return to normal

    once again *forthosethatmissedit*

    GIVE US MAIL

  73. KF Says:

    I’m back up, I’m happy to report. And given that my previous hosting provider used to go down simply because it was Tuesday, and would never respond to email queries, and would get their servers back up pretty much as soon as the trained gerbils got the wheels powering the whole operation back up to speed, I’m not going to complain about a little blackout-induced site withdrawal.

    But I do hope the DWP figures out who crossed the streams and has a good, long talk with them.

  74. The Map Room Says:

    Outage

    The Map Room was unavailable for nearly seven hours today; my hosting provider was knocked offline by the power outage that hit Los Angeles this afternoon. All my sites were offline until just a few minutes ago. More posts tomorrow….

  75. LW Says:

    My site is still down :( I actually recommended DH to a friend yesterday and created some referral code. When do you guys think things will be fixed? Thanks.

  76. Renata Says:

    All my sites are still down. At least I had the emergency status page to send to my domain mail users… How I miss my e-mail. I’d only suggest you to write down GMT hours in the emergency status page, as I had to Google for PDT to know when BRT was it updated.

  77. John Says:

    Received no mail between 3:44 and 8:06PM. Nothing’s trickled in since then, which is rare for me. What happened to the mail that was lost during this time?

  78. Mike Mu Says:

    Mail delay info …

    I believe many mail servers use a decay algorithm to attempt resends. So the longer a site is down (DH), the longer it will take for the mail server that has mail bound for you to attempt redelivery.

    I searched and qmail — a popular MTA — uses a “quadratic retry schedule”, which can mean a long time for mail to get delivered after an extended outage. See www . lifewithqmail . org/lwq.html#retry-schedule

  79. Kimmie Says:

    I can totally see that it was through no fault of your own (unless DH can cause city-wide blackouts?), and of course things happen. Sucks that the generators went down, but that’s life. Glad everything is being fixed. It’s things like this that made me come back to DH… As opposed to hosts that went down and never came up (ugh), went down daily, etc. Ya’ll rock. Keep it up!

    To some commenters who are insisting that the Server Status is off… There’s a reason for:

    System Status is not guaranteed accurate! It can not measure network failure downtime, system misconfiguration downtime, or some short downtimes. It is only a rough reporting of the historical uptime for your webserver’s hardware.

    :)

  80. David Says:

    It’s 10:30 pm central and my email is still down. Website is up, for which I am grateful. But with local primary elections tomorrow, it is really unhelpful to not have email.

  81. BesterP12 Says:

    Oh yeah still much better than other servers I’ve been on. Although all of them started out as having great service. I was on Prohosting for a long time. Time was we could call them up if we had a problem and always get the same tech guy and we’d always be right back up. We even worked together to get the (then) new Mod_rewrite installed on our server when leechers started realy hitting our bandwidth.

    Later on as they expanded, they got new people, one in particular knew less about their systems than I did and things started taking longer to get fixed. Finally their billing dept started billing me on random dates and screwing up my balances(two bank withdrawls withing one month for someone who lived month to month at the time is very bad.)

    So I looked around to see what was widely recomended. That was PHPWebhosting. They too were great for a while. Then around last Jan they turned useless. So I decided to try DH. I’m really hoping this not shutting down safely was a one time thing because they were caught off gaurd and not the start of a downward spiral, but I’ve just been burned in the past.

    It will take much more than one outage and a few complaints from visitors for me to switch though. Especially at this price.

  82. Cliff Says:

    One would think the data centers and network providers would have been a bit better prepared with back-up to the back-up and/or power disaster recovery in place.

    Many DCs were taken off line today…wake up call to all providers, data centers and ISPs.

  83. BesterP12 Says:

    To some commenters who are insisting that the Server Status is off… There’s a reason for:

    I figured something like that, just nice to find one bit to laugh at in this.

  84. james Says:

    All of my clients sites are still down, all 8 of them,,
    email is still down for all of them..

  85. Chris Hobbs Says:

    I’m certainly not ready to throw in the towel with DH - hell, I’ve been with them since January 1999 and this is only the second downtime of any real duration I’ve ever experienced with them (and this is much worse than the last I recall).

    Keep up the hard work guys, but please do get my site, clanhobbs.org, back as soon as you can - it’s not much but it’s home :)

  86. David A Says:

    So, do we get some credit? :)

    I left Hosting Matters two weeks ago because they were becoming totally inept at what they were supposed to be doing. I still have 41 sites there, and the site I have here at DH is my “test” before I decide to move everything.

    Up until today, I had really no complaints — except for a couple of minor items that I am learning to live with. How DH handles this issue will go a long ways in helping me make my decision to move everything over here.

    It is now 8:45 PDT and my site is “up” but minus the SQL server. Being a WordPress site, not having the SQL means that it STILL won’t work though.

    I suppose I will have to put an index page up to explain to my readers that Sh*t happens and the pooper scoopers are on their way…

  87. josh Says:

    Just an update (8:45pm PDT) .. we’re down to about one half rack of servers (a dozen or so) still down, due to a wonky switch. Hopefully everything will be ironed out within the next hour.

    Here’s a quick run-down of how our power works.. we’re in a building downtown that is sort of specialized as a data center. There are lots of data centers in the building, and the building itself (which used to be a large bank’s internal data center) provides the power to the tenants.

    This is (supposed to be) “good” power. IE, the building itself has huge UPSes and four large backup generators that _are supposed_ to be enough for all the data centers.

    When the power went out for the city, the UPSes came on, and things were fine (that’s when Nate made that last post about us being okay). However, as they started to run out the generators had to kick in. It turned out, and this is the boo-boo, the generators could NOT generate enough power for all the data centers after all! And so, rather than just doing some sort of lower-level of power output, they are apparently configured to all just shut themselves off when things are overloaded.

    So, basically a mini version of the LA power outtage happened right inside our building.

    The grid is definitely back on now, and the reason the generators were on earlier was the building engineers are testing things and messing with stuff.

    Ironically, the building had been telling us they were in the process of getting an extra generator installed because they were afraid they were starting to get to their power limits.

    Yup.

  88. Ray Says:

    QUOTE: “Hey, folks, just an FYI about UPSs — their purpose isn’t to keep electrical devices constantly running during a power failure. That’s what generators are for. UPSs are big batteries that give you those precious 5 minutes to properly shut down connected devices instead of grabbing that extension cord in both hands and yanking with all your back muscles.

    Comment by Rome — September 12, 2005 #

    HAHA 5 MINTUES? If you are running a small one out of your house. At our work our UPS can keep up our whole floor for 8 hours. Even my home UPS, which I have two 1500VA UPS and some smaller 1000VA UPS can keep me up for over an hour with my equipment.

    5 MINS LOL, this is supposed to be a DATACENTER!

  89. Joe Agpawa Says:

    Oh thanks, not fair at all.. Dreamhost gets their page back up but I still have mulitple client sites down at 8:50 PM pst..

  90. ashot Says:

    Whoa, if I try to access dreamhost.com I get redirected to:
    http://www.dreamhost.com/dedicated/dedicated/dedicated/dedicated/dedicated/dedicated/dedicated/dedicated/dedicated/

    and its not found. This is as of 11:55 est.

  91. Gary Says:

    12:00am Eastern and traceroutes to DH nameservers are still choking on: fe1-mz.dreamhost.com (216.193.192.50).

    AFAICT, personal domains are equally fucked.

    Still that wonky switch or is there another grid-related outage? Why not update http://status.dreamhost.com/ instead of taking the time to *not* blog about what’s still not up?

  92. Shaun Kelly Says:

    You don’t own your own UPS, then?

  93. David Says:

    9pm PST- 8 hours of total downtime, all services and sites.

    I chose Dreamhost. I like Dreamhost. I want to stay with Dreamhost. Tell me why Dreamhost is located in a city that is widely acknowledged to be a leading candidate for a major earthquake, one of the choicest terrorist targets, and in the middle of a miserably fragile electrical grid (I lived there through the rolling blackouts a few years ago), yet is not adequately prepared for the electrical failure that could be caused by any of those three events? I include your choice of buildings, network providers, whatever in your blame- you chose them as part of your job, just as I chose Dreamhost as part of mine. What is important is what we choose to do now.

    I recommend in the strongest possible way you investigate moving the location, decentralizing, or convincing us this won’t happen again through your extensive new equipment, planning, and new network providers. It doesn’t matter how good the deal is or how cool the ISP may be if my site isn’t there when it needs to be. No jokes about rattlesnakes here, just a dead serious note that you can show to your network providers- to the building landlord who maintains the generators- to anyone else responsible for this outage, so they realize this is very serious for some of us.

    If in the end, after all the reports on what happened are filed, and the general electrical infrastructure is the sole culprit and makes it not possible to provides secure services from Los Angeles, then one of us will be moving on- your choice.

    Thank you for your efforts, PLEASE get my site back up soon, PLEASE insure this won’t happen again.
    David

  94. E. Marie Says:

    Well, congratulations to the people who have sites back up … for awhile one of my four was available, but now they’re all down and email is punk as well. I wish Dreamhost was into a little more customer communication–the “fully operational very soon” posted on the status page at 6:46 pm has proven not to be true. (When that was posted, I at least had one site that was still up!)

    Could ya at least update the status page? Jeese.

  95. John Says:

    No offense guys, but I’m a bit disappointed at the lack of back up power… you should take some notes from directnic… they were able to keep their customers online throughout Hurricane Katrina. Read their blog: http://www.livejournal.com/users/interdictor/

    It’s a tough situaton, I’m sure, but I’ve been down over 12 hours…

  96. A Day In The Life… » Power Glitch Says:

    [...] This site and related sites, including inhisname.net, caffeinated.org, and newtomorow.org sites were down for a few hours due to a severe power outage in the Los Angeles area today. This brought down our hosting provider, DreamHost, when issues with backup generators led to a shutdown of systems and indeed entire major routing nodes. Human error strikes again. [...]

  97. Derik Says:

    Take this quote from the post…

    “Unfortunately, shortly thereafter we figured out (thanks to EVDO!) our sites were no longer accessible, as our three network providers Level 3, Global Crossing, and Mzima were all down. Our physical servers were okay, and probably really happy sitting in the data center (we were no longer able to access) with nothing to do.”

    Narrow it down to this…

    “our sites were no longer accessible, as our three network providers Level 3, Global Crossing, and Mzima were all down.”

    It doesn’t matter if their servers themselves were online - their network providers were all down - they couldn’t have had (much) access if they wanted to…

    Besides - being in one of the largest cities in the world - in one of the best weather areas (Southern California - aka “no seasons - year round calm”) should be plenty safe enough for a datacenter… Then some idiot from the utility company screws up the city - and that’s proved wrong…

    Basically - I may be wrong, but Dreamhost theoretically couldn’t have had our sites up anyways… as the network providers were down too… At least, how I’m reading it…

    -Derik
    http://www.derikolsson.com

  98. Vince Says:

    “Just an update (8:45pm PDT) .. we’re down to about one half rack of servers (a dozen or so) still down, due to a wonky switch. Hopefully everything will be ironed out within the next hour.”

    It would be useful to have a bit more detail. A dozen or so out of how many servers? And what about email? That still doesn’t appear to be working.

  99. Gary Says:

    Derik: Agreed, yet how can Dreamhost claim to have any semblance of “control” or “accountability” by way of this blog or the status-dot… both of which are apparently still working… yet no updates to either… and no service to customers.

    If it’s not their fault… that’s fine. Tell us what’s going on. If it’s not their fault, then they have the time to tell us what’s happening.

  100. David Says:

    “…fragile electrical grid (I lived there through the rolling blackouts a few years ago)”

    You do know those were fake, If I remember correctly that was part of the whole Enron mess.

    As for this situation, I know that all the people in the datacenter are going to be working extra long and probably throughout the night. Murphy’s law anyone? Give them a tiny little break. Maybe they’ll throw in a gift and x10 our space and bw… ;p

  101. Stewart Vardaman Says:

    Been with DH for the past two years, and I’ve been very happy. Today has been a mess, but overall I’ll still recommend DH.

  102. Joe Says:

    Please, I need some help. I have a dedicated server that lost mysql & mail service for a few dozen sites (for whatever reason) 48 BEFORE the power outage. I don’t know why it was taking so long to get a support response, but now with extra time added on for the outage, my customers are getting irate (especially over the email).

    Are the old support requests still being dealt with, or are they completely on the back burner for now? (I know, I know, bigger fish to fry…)

    I’d love to at least have an acknowledgement that somebody has read my support request.

    Thanks.

  103. Derik Says:

    Re: David

    Earthquakes aren’t going to take down Los Angeles - any Californian knows that - you’d need one of the largest earthquakes on record to damage the city… A rarity that is about as likely as another Katrina hitting.

    “Fragile Elecricity”
    The rolling blackouts are a thing of the past - I too lived there, and have gone back many times since then. There are ~11 more nuclear power plants in California - use up as much juice as you’d like :).

    Re: Gary - they’ve been explaining what’s been going on throughout the time since they’ve gone back online… Now apparently they’re fixing some servers that are having problems from the power surge - and are almost back to 100% again. Ctrl-F, “josh”, you’ll find his updates.

    -Derik
    http://www.derikolsson.com

  104. Nate Says:

    My power went out and came back on around 2pm. It is now 9:30 and 7.5 hours later I still have no email. Are you rebuiding these servers from the ground up? We were hit by a power outage, not a cat 5 huricane. This amount of downtime is unexpected.

  105. Renata Says:

    Hooray, all my sites are back up. Now I can go to bed, sleep and have sweet dreams. :)

  106. mia Says:

    i’ve been watching the downtime today and have seen several of my peer’s sites go live one-by-one over the past few hours. so, it is obvious that the DH guys are working hard. give them a break. granted, mine is still down, but hey, what can ya do?

    thanks for your hard work, guys. as for the technical problems, we know you’ll reassure us after all this mess that it has been taken care of.

    thanks again for years of great service.

  107. K Tran Says:

    My three sites are up including email and ftp access. I second the x10 space!

  108. C Montoya Says:

    My sites seem to be back up.

    I think Dreamhost did allright, I mean, the data didn’t get erased, which is key, but there’s definitely room for improvement. I’m sticking with Dreamhost on this one, hopefully it’s not a mistake. Just promise me you guys will get those generators.

  109. Gary Says:

    Derik: I’ve read all of Josh’s comment posts. So, I grant that the comments section has been updated… but not the original blog post (quick-and-dirty, up-front update on the post will save a lot of customers a lot of time with the scrolling and the apple-F “josh” bullshit).

    The status-dot is still hours old.

    I’m not at all displeased with the service that Dreamhost has provided us over at (currently down neurobashing.com).

    I’m not at all happy about the abject lack of current information/communication being conveyed by obviously-unaffected websites such as this blog and the status-dot website.

  110. JMK Says:

    I have a dedicated server and it’s still down. I also can’t get to the web panel. Presumably, you guys know which customers are still affected–an email would be nice. When I sent a note to the sos address, I got an auto-response.

  111. JMK Says:

    And shazam! I complain, and I am heard! It’s up now. Thanks!

  112. sam Says:

    Did any sites get hacked? A client reported seeing something like ’spy kids own you’ or something along those lines….

    Thanks, Sam

  113. Bob Says:

    Ever hear of backup power?

  114. Ron Says:

    I’d like the JMK treatment please.

  115. Derik Says:

    “Derik: I’ve read all of Josh’s comment posts. So, I grant that the comments section has been updated… but not the original blog post (quick-and-dirty, up-front update on the post will save a lot of customers a lot of time with the scrolling and the apple-F “josh” bullshit).” - Gary

    I completely agree - it would be wiser - and easier for customers… these updates should also go on the emergency page… I hope your site(s) get(s) up soon, Gary. This could have been executed better in some areas, but I’m sure DH is working hard to get everything back.

  116. Emily Says:

    My email is working now, but not too well. Everytime I try to do something, it takes me to the login page again. Is this happening to anyone else?

  117. Scott Says:

    My sites are back up and webmail appears to be working as well. Everything’s a little slow, but it beats error messages. I don’t picture DH as having a ton of people working there, so cut these guys a little slack, if they were blogging all the time with updates they wouldn’t be getting things fixed.

  118. Cary Says:

    10pm California time, and both of my sites are still down.

    http://www.cancer-news-watch.com and http://www.missionremission.com

    Any news coming out of DreamHost HQ? Even a couple of words would be nice.

  119. Lucky Says:

    10:00 PM PST and my sites are STILL down. Am I REALLY that lucky? At least my email is up so people can bitch me out for the sites being down.

  120. Gary Says:

    Derik et al (esp. DH staff): I have no doubt that DH is working as hard as they can to get things back to normal. It’s really the only thing they can do to keep their customers from calling for their heads on spikes.

    Not their fault? Keep us updated during idle time waiting for upstream providers to fix *their* shit on the live sites they have available (here and status-dot).

    Maybe their fault? Be straight with the customers.

    Websites being down isn’t the end of the world; but the service provider is beholden to its customers. I’m not saying they need to bend over backward to make us happy (the provider’s performance is subject to customer scrutiny and judgment); but it is their responsibility to keep us informed… if they wish to keep our business.

    Working hard to fix a problem is one thing… but nobody will ever realize the effort put forth if there’s no communication.

  121. Mohamed Mansour Says:

    aha :)
    Everything is up :)
    DH is fast nice …. From a major power outtage everything got restored quickly and nothing lost I guess !

    Im wondering if any servers got wrecked and why didn’t the 4 power ups didn’t power up :s I thought it should power up in a case like this :)

  122. mb Says:

    yeah, my site is still down as well (http://www.semidigital.com). is there any news that all of this will be back up by midnight? if not, when? you’ve mentioned “in the next hour,” several times… now when?

    i think we’re all losing a bit of patience.

    thanks.

  123. beejereeno Says:

    It’s too funny that the people with attitude posting here are mostly those running blogs :) I have a business site with 11 client web sites under mine and I just now got in here to read. I’ve been posting emails w/status updates to my clients all evening instead of bitching about it. We don’t know what happened, nor can any of us do much about it. Patience is a virtue.

  124. Ampersand Says:

    My sql is still down - which means my blog, by far the most popular part of my website, is dead. Oy.

    I know y’all are doing the best you can, but let me add my voice to all the other folks suggesting that if something like this ever happens again, you should make sure to be providing regular updates. Even someone posting once an hour to say “nothing new to report, we’re still working on getting stuff back up, check for a new update an hour from now” - just that and not one word more - would be very reassuring, compared to what you’re doing now.

    Good luck getting our sites back up! :-)

  125. Scott Says:

    MB, http://www.semidigital.com/ works for me.

  126. Christopher Mills Says:

    Not complaining, exactly, but curious: my site was down, then came back, and now appears to be down again (although this time my e-mail still works). Just wondering why.

  127. Gary Says:

    beejereeno: You’ve been posting emails w/ status updates to your clients all evening. You’re doing it right and being–IMHO–a good service provider. That’s the point.

  128. mb Says:

    scott,
    yep, it just went live. :)

    hooray! thanks DH.

  129. Jack Says:

    I have a few personal sites up and a few professional ones as well. As much as it’s frustrating that they all went down, I know the deal and you plan for the worst, hope for the best and when all is said and done sometimes things go awry.

    I have no doubt DH staff is doing all they can. And I want to echo another posters comments. If you’re SNAFUed professionally by this, call and e-mail clients. In my case simply saying “The Los Anegles blackout knocked us out…” was more than enough to handle it.

    As for blogs. Damn. This proves what most of us know. I read blogs as well, but good lord are bloggers collectively a whiny bunch. I’m personally sorry the world will not be able to read about your cats latest antics and such, but enough already. In the great scheme of things events like this are par for the course.

    Bloggers, please. Learn to cope and deal. Not to bitch and moan.

  130. MissMeliss Says:

    Have had email since 5-ish CDT and have been able to see my files for over an hour, but FINALLY apache has been restarted. Thank you, DH.

    (My husband works for…another hosting company…and at least explained everything that I didn’t quite understand about what COULD be wrong. That helped, too.)

  131. josh Says:

    Just an “update” folks…

    We’re still working on machines, and if you’re still having site problems, we know! We’re doing our best to get everything sorted out as fast as possible.

    I’ll make one more (final) post when we think we’ve really fixed absolutely everything. If you’re still having problems then, please contact support through our panel!

    (Or if that’s not working, post here! :))

  132. Chris Hobbs Says:

    Hmmm, methinks maybe some sites have been moved to different servers? I just tried connecting via ssh to my site and have been warned that my cached hostkey info does not match what the server is presenting me. Anyone else seeing the same?

  133. Discourse.net Says:

    LA Power Outage Dims Discourse

    The LA blackout seems to have taken the site down with it. Supposedly Dreamhost has an emergency backup system, but it seems to have failed completely. Probably FEMA-approved hardware… Update: Details if you care……

  134. Scott Says:

    Thanks! We are back on the air!

  135. Tomas Says:

    I’ve been on both sides of this sort of failure - as an end-user and as the project manager/engineer responsible for premises equipment serving the fiber connection to the real world for multiple major communications carriers.

    After the first failure of the building provided power backup at one of the buildings I’d engineered, most of the tenants, including us, got together to protect ourselves.

    With the cooperation of the building management company we built UPS sets for each customer (8 hour minimum), engineered and installed multiple redundant generator sets, AND put in a massive external power connector so we could roll up a trailer mounted 1MW generator to handle ‘essential power’ if all else failed.

    The entire building was divided into ‘essential power’ and ‘everything else’ so we could load shed if things got sticky.

    We DID survive a four and one half day power outage that took 31 square blocks of the city out.

    We had dry ice in some areas to cool overheated electronics, we had hardly any lighting, we had one elevator for emergency use, but everyone kept running.

    Had we not had a previous failure we would not have made ourselves ready for ‘the big one.’

    This is your opportunity to learn, guys. Look around at what did and did not work after you get everything up again. Fix what didn’t work and get guarantees from the folks backing YOU up.

    If you contract with the building for backup power, make sure they can provide it - and include penalties in the contract if they don’t.

    Overall? Yeah, you had a failure. It took too long to recover. It won’t happen again, right? ;o)

    Learn from this and you’ll be ahead of the game.

    Oh, yeah, GOOD JOB! Really! Until the first one like this you really don’t have that gut feeling for what’s needed. You performed well getting it all back together when outside folks let you down. Thanks.

  136. A Slightly Insane World Says:

    Today’s Downtime

    Err.. sorry bout that… It appears that Dreamhost uses servers and internet connections that require electricity.. so, after the (probably now unemployed) Power Company worker shut off half of Los Angeles, Dreamhost no longer had a connection to the…

  137. Jad Says:

    Ok, I’ll let it go with this one final rant:

    “Note: System Status is not guaranteed accurate!…”

    I’ve said this before to DH management — the status tab should be deleted, unless it’s going to be *honest*.

    To give the disclaimer that, essentially, not all downtime events are going to be counted or reflected in the stats is a total luse, worthy of the current administration in Washington.

    Otherwise where I come from it’s called a lie. Please either be honest about the numbers or take them away, ’cause right now they aren’t worth the diskspace they occupy.

    /jad

  138. Who turned off the lights? : Losing it[1] Says:

    [...] DreamHost Blog Power Outage Update [...]

  139. josh Says:

    Just a note, as of right now everything is believed to be good to go! If you’re still having any site or email problems, please contact support (after you’ve verified it’s still broken for sure!)

    Thanks for the patience everybody.. more tomorrow!

  140. Fin Says:

    Good lord some of you guys are whiny.

    that is all.

  141. Sanity Software News » Blog Archive » LA Power Outage Takes Site Offline Says:

    [...] Check out the CNN article and the DreamHost blog for more information on the power outage. [...]

  142. Simon Says:

    its nice to know i was right and that it wasnt dreamhosts fault but their network providers… :D

  143. html life » ¿Una coincidencia? Says:

    [...] Lo que le paso a Dreamhost: Okay, this sucked. Literally 10 seconds after Nate posted that last entry, we received an announcement over our loud speakers that our building was being evacuated and we had to leave immediately! [...]

  144. Bieber Labs » DreamHost Blog » Power Outage Update Says:

    [...] Apparently there was a power outage in L.A. that took the site down yesterday. You can get details from it in the DreamHost blog at DreamHost Blog » Power Outage Update. [...]

  145. John Says:

    So much for all those 100%’s on the server up time screen…. they’d BETTER reflect this outage.

  146. shez Says:

    shouldnt there have been a backup servers preferably somewhere in another location to deal with this type of “emergency”?

  147. GF Says:

    Any ideas what this is?
    Site Temporarily Unavailable
    We apologize for the inconvenience. Please contact the webmaster/ tech support immediately to have them rectify this.
    error id: “bad_httpd_conf”

  148. Dolores G Says:

    Where is the mail that was lost in the interim?

    Can you guys get it together to post a single, comprehensive update? I’m getting tired of these “Update coming soon!!!!” messages.

    I just want to know WHERE my mail is that was lost in the downtime.

  149. asuh Says:

    blackout

    Much of Los Angeles “went dark” yesterday during the middle of the afternoon. I have to quote this phrase because, well, the sun was out. But for much of the afternoon, power was off for nearly a million people. Luckily,…

  150. Sarah Says:

    All the RB fans will be delighted to see it’s back up no doubt.

  151. hwrd » Blog Archive » Where did I go? Says:

    [...] DreamHost Blog » Power Outage Update Technorati Tags: Dreamhost [...]

  152. soul Says:

    It’s kaputt, not caput.

  153. Dave Says:

    Nice that your back :D

  154. blog.slystuff.com Says:

    [...] http://ask.metafilter.com/mefi/23949 http://blog.dreamhost.com/2005/09/12/power-outage-update/ http://status.dreamhost.com [...]

  155. Heidi Says:

    Panicked a bit at home and at work - we probably lost business because email was down too. Some of the other hosts boasted redundancy systems. Think it’s time for that? Not blaming - after all it was out of your control - but making a suggestion for the future.

  156. DreamHost Blog » EVDO Cards Rock! Says:

    [...] This is awesome news, because as you may know, I LOVE EVDO! [...]

  157. MotoringFile » Blog Archive » No! This one goes there! That one goes there! Says:

    [...] We’re sorry about being offline for part of the day. Our host was affected by today’s power outage in Los Angeles. [...]

  158. Relacore Lipovarin Estrin-D Cortislim Suvaril Trimspa Zantrex-3 Avlimil Enzyte Magna-Rx Ogoplex Altovis Says:

    Relacore Lipovarin Estrin-D Cortislim Suvaril Trimspa Zantrex-3 Avlimil Enzyte Magna-Rx Ogoplex Altovis for the lowest price on all of these name brand product just visit link above today!

  159. lipo 6 Says:

    New Liquid Capsule Design Assists In Rapid Weight Loss!

    Lipo-6 by Nutrex is the first and only fat-burner that utilizes maximum strength liquid capsules for superior absorption and unprecedented results. Never before has a fat-loss formula offered such an advanced method of delivering powerful fat-burning compounds, quickly, effectively and safely. We guarantee, with Lipo-6 Liquid Capsules, you will experience far greater fat-burning results than with any other fat-burning product on the market today! http://www.a1anutrition.com/cgi-bin/item/LIPO6

  160. eharmony Says:

    If you looking for the right ONE you should get your free profile from eharmony today.
    http://free-blog-site.com/eharmony/

  161. ehormony Says:

    If your looking for the right person to date you have to go to ehormony for your free profile today!
    http://free-blog-site.com/ehormony/

  162. Unofficial DreamHost Blog » Blog Archive » DreamHost Year in Review Says:

    [...] Down Time Two major incidents happened in 2005. The Power Outage in entire Los Angeles in September and a massive DDoS attack in December. Major crisis like this is impossible to avoid, but when it does happen, it’s nice to have a web host that is quite informative about what’s going on. [...]

  163. DreamHost Blog » It’s all about power. Says:

    [...] Our power costs even more than yours because it’s “good” power. That is, it’s backed up by huge diesel generators and UPS batteries in the supremely unlike event of say, a Los Angeles power outage. People estimate that Google must pay about $50,000,000 a month for power for their 100,000 servers. That sounds about right to us! [...]

  164. Silent Guardian Says:

    You know guys, you’re all complaining about the downtime, a lack of suitable redundant power etc, but you’re forgetting that Dreamhost is one of, if not the most reliable of all the cheap web hosts out there. If you want redundant UPSes, more bandwidth, co-located racks etc, then why not pay for a more premium service?

  165. Unofficial DreamHost Blog » Blog Archive » DreamHost Power Outage Says:

    [...] DreamHost was affected by a similar incident last year, when the generators didn’t work. [...]

  166. DreamHost Blog » Anatomy of a(n ongoing) Disaster.. Says:

    [...] Ironically, all the recent disasters stem somewhat from us attempting to take some proactive steps to head off any sort of future power outages like the kind we experienced last year. [...]

  167. weswillis·com · Why I love Dreamhost Says:

    [...] Transparency. When a server crashes or a catastrophic power outage disrupts service, they’re quick to post to their status blog. The only time I heard from any of my previous hosting providers was when I owed them money. Dreamhost isn’t afraid to flaunt it’s faults, it embraces them. [...]

  168. Jodi Abbott Says:

    hi
    st4hrwqv5m8xvnmf
    good luck

Powered by WordPress. Pool theme by Borja Fernandez, modified by DreamHost.
Like WordPress? Consider attending WordCamp LA.
Entries and comments feeds. ^Top^