The Building Speaks

September 13, 2005 on 4:14 pm | In Foobars, Hardware, Insider View, Updates by Josh Jones | 25 Comments

Okay, we just got a fax from the building finally:

On Monday, September 12, 2005, at approximately 12:35 p.m., the building experienced a total loss of electrical power from the DWP on their primary grid. At this time, the building generators started and began supplying adequate power to the tenants.

At approximately 12:55 p.m., the building power was partially restored by the DWP. At approximately 1:05 p.m., the building experienced another total power failure from the DWP.

During this period of 12:55 p.m. to 1:05 p.m., two of the five generators failed. The remaining three generators were unable to sustain the power requirements of the building causing the emergency electrical systems to transfer into a “load shedding mode” and the building’s UPS system to turn itself off, thus preventing permanent UPS and related equipment damage.

The Office of the Building is conducting an investigation into both generator failures and will keep you apprised of the results.

At this time, the building is functioning at full capacity with four generators to support any future outages.

On behalf of the Office of the Building, we would like to thank all of you for your cooperation during the building evactuation and your patience with building personnel during this emergency.

So.. hmm, we will in turn keep you apprised.

DreamHost Department of Web Problems

25 Responses to “The Building Speaks”

  1. Will Says:
    September 13th, 2005 at 4:24 pm

    So those were the the DWP workers who cut the power. :-)

  2. nate Says:
    September 13th, 2005 at 4:40 pm

    No, Josh was awesome last night and made us all customized hardhats commemorating our enjoyable night.

    The front says “DH DWP” (in reflective letters) and the back (of mine) says “NATE SEP 12 2005″.

  3. Will Says:
    September 13th, 2005 at 5:10 pm

    You mean like this? > http://blog.dreamhost.com/images/dhdwpnate.jpg

  4. Zach Says:
    September 13th, 2005 at 5:20 pm

    Please don’t break my website or my email ever again.

  5. nate Says:
    September 13th, 2005 at 5:36 pm

    Yeah, that’s my helmet!

  6. Christopher Says:
    September 13th, 2005 at 5:49 pm

    Hey Nate, can you tell us who from the dreamy DreamHost staff is in that lovely picture?

  7. Robert Daeley Says:
    September 13th, 2005 at 5:59 pm

    Hey, y’all — is there any word on the router situation? My website has been inaccessible for some hours now.

  8. GD Says:
    September 13th, 2005 at 6:41 pm

    Maybe if you make some more helmets, I’ll quit getting packet loss which is corrupting all my files as I download them at 22KB/s.

    Or not.

  9. Robert Says:
    September 13th, 2005 at 6:45 pm

    Keep up the good work restoring us. I’ve been stuck with 8K downloads all day. I blame Global Crossing!

    Robert

  10. Dan Says:
    September 13th, 2005 at 7:14 pm

    In the back (from the left) is Nate, Terri, Jeremy and Micah. In the front (sitting) is Tavis and Jason.

  11. Mike Says:
    September 13th, 2005 at 7:48 pm

    I’m still having trouble with mysql on Scratchy.

  12. Nes Says:
    September 13th, 2005 at 8:09 pm

    Warp Pipe is still down :(

    I missed my tourney, I hope Dreamhost can come to the rescue.

  13. Jad Says:
    September 13th, 2005 at 10:45 pm

    Well there’s only one real way to test a power system to see what’ll happen..

    Now at least you know that you can’t count on your building… /jad

  14. khaled Says:
    September 13th, 2005 at 11:06 pm

    I’m about to say a couple of things about that statement by your maintenance people, because I think something is fishy, would you mind clarifying one minor point, do you guys have diesel rotary UPS systems or static ups? Just ask the building management people, because unless they’re rotary UPS connected to the generators, then that statement up there is utter bull, and I’ve got a nice long post about it. Just wanted to check my facts first. Thanks for the efforts though.

  15. Keaka Says:
    September 14th, 2005 at 2:51 am

    At this time, the building is functioning at full capacity with four generators to support any future outages.

    Great, the four generators make me feel warm and fuzzy and safe. After all, the building did so well with 5 generators :)

    Thanks DreamHost for working so hard to get everything back up. Although I think DH could have done better, I keep reminding myself that my last web host would frequently drop FTP access for days at a time with no response from support. If my last host went through a problem like this, I probably wouldn’t be able to see my server for a month!

  16. lurker Says:
    September 14th, 2005 at 1:37 pm

    “tavis” travis maybe?
    anyways, he’s hot!!!! :)
    woot

  17. Bill Ekhardt Says:
    September 14th, 2005 at 2:46 pm

    Thanks for your work through all of this, Dreamhost. Please convey to your building owners that their explanation is woefully inadequate, and that at least this customer of yours finds their quality of backup insufficient to be in the business of supporting a datacenter.

    My desire is that they make hard expensive and drastic changes, or you find another location for your data center.

  18. Gary Anderson Says:
    September 15th, 2005 at 2:27 am

    Nice photo of the crew.

    You know, things will always break. It is not so much of a question of preventing them from breaking as it is being able to fix them when they break. Good job.

    I guess that is it part of the ingenuity that Harvey Mudders have. (Like the famous, giant slingslot that lobbed water ballons from a HMC dormitory to CMC.)

  19. Sage Says:
    September 15th, 2005 at 7:43 am

    So the UPS shuts off when the emergency electrical system isn’t providing enough power to the building? Isn’t this exactly the opposite of what a UPS is supposed to do?

    What can be done to prevent that from happening without damaging the UPS or connected equipment, and how does DreamHost plan to address this if they don’t maintain any part of that system themselves?

    I am not trying to be accusatory, I just would like to know what the plan of action is.

  20. Carl Youngblood Says:
    September 15th, 2005 at 7:58 am

    What really needs to be done is test the backup power system once a month or so to make sure that it’s still working. A backup system that has fallen into disrepair is not going to work well.

    I want more assurance from the building that they are going to stay more on top of things in the future.

  21. Broken Kode | We're All Broken Kode Says:
    September 22nd, 2005 at 12:52 pm

    Dreamhost Outage

    As some might have been aware, Broken Kode was down for a few hours on Sunday, along with most of the Internet (seems everyone cool hosts their sites either with Dreamhost or Mediatemple, noooch). I don’t mind that since I’ve had it happe…

  22. Broken Kode | We’re All Broken Kode » Blog Archive » Dreamhost Outage Says:
    September 22nd, 2005 at 12:55 pm

    [...] As some might have been aware, Broken Kode was down for a few hours on Sunday, along with most of the Internet (seems everyone cool hosts their sites either with Dreamhost or Mediatemple, noooch). I don’t mind that since I’ve had it happen to me before, countless times with my previous host. Dreamhost have been blogging about it. The also eventually sent an email to everyone trying to explain everything. [...]

  23. Dmitriy Says:
    December 19th, 2005 at 4:20 am

    Sounds good!

  24. Anonymous Says:
    October 6th, 2006 at 1:11 pm

    ko bd db d dl; d dm:dk nndhyt tdgghj jdkiuy ndh dndndmpdo ndjdhyu ndbghrbd dydndy dynd dndkdkthfg

  25. Rape. Says:
    October 20th, 2008 at 3:55 am

    Rape….

    Rape….

Powered by WordPress. Pool theme by Borja Fernandez, modified by DreamHost.
Like WordPress? Consider attending WordCamp LA.
Entries and comments feeds. ^Top^