38 Comments

1. 1

   Haha first reply post! ;) Great job. Glad to see that issue is resolved.

   Comment by dmacgavin — September 14, 2005 #

2. 2

   Woah! Now thats great news!
   I was just browsing my slo-mo gallery when suddenly it all went back to holyspeeds :p
   thanks!

   Comment by pint — September 14, 2005 #

3. 3

   really great to hear this is finally resolved. Hope you guys sleep well finally - not that things are back to “normal”

   Thanks for the great service you provide.

   -Matttail

   Comment by Matttail — September 14, 2005 #

4. 4

   Yay, verything seems to be working at normal speeds now, thanks!

   Comment by Bob — September 14, 2005 #

5. 5

   thank you dreamhost!!!

   Comment by Denis — September 14, 2005 #

6. 6

   Dreamhost beeing down only happens once in while. No host here in the Netherlands can match with Dreamhost, they are also much more expensive compared to Dreamhost so don’t worry. I like and really appreciate your blog, it keeps us informed and it all sounds so human. Thanks.

   Comment by Alfred — September 14, 2005 #

7. 7

   Hey, don’t keep us in suspense! Tell us the culprit option. ;-)

   Comment by Sylvester — September 14, 2005 #

8. 8

   Thanks so much for your help.

   Hey, don’t keep us in suspense! Tell us the culprit option.

   You must tell us. :)

   Comment by Patrick — September 14, 2005 #

9. 9

   Excellent, I just checked and my site’s back up to its usual great speed. Thanks Dreamhost!

   Comment by Scott — September 14, 2005 #

10. 10

    thanks DH
    it is so damn fast rite now

    Comment by ryan — September 14, 2005 #

11. 11

    Good job guys… and ignore all the stupid criticism in the forums. Looks to me like most of this was way beyond your control.

    Comment by Neil — September 14, 2005 #

12. 12

    I just want to say thanks. There’s no way you could have avoided these issues, or all the cranky “what the!?” responses from customers; the way you’ve handled the initial problem, the restoration and resolution, and the aftermath has been excellent.

    Rather than shaking my confidence, this incident has proven that I made the right decision moving to Dreamhost–the extra money is well worth it!* Keep it up!

    * The extra money is mostly because I upgraded my plan, DH represent excellent value for money.

    Comment by Calrion — September 14, 2005 #

13. 13

    Woohoo! Everythings back to normal! Great work guys, having been through a large scale powerfailure once or twice in my life with regards to hosting and such I know it sucks.

    Thanks for all your hard work in getting everything back to running the way it’s supposed to. You guys are great and are worth every cent I spend to host with you.

    Comment by Deamos — September 14, 2005 #

14. 14

    back to normal. thanks for the work guys!

    Comment by ivan raszl — September 14, 2005 #

15. 15

    I knew you guys would get it all ironed out ASAP. I’m in the crowd that could see that this was nothing more than one of those times when all your potential points of failure failed all at once. It’s rare, but it does happen, and when it does all you can do is fix it as fast as you can. No amount of grumbling or whining will help, and no one can really fault you much for being caught off guard by a cascade of failures. You didn’t black out LA, you didn’t oversell the building’s backup power capacity, and you didn’t blow the power switches or router. Hardware that runs 24/7 HATES to lose power unexpectedly, and even with scheduled maintenence a lot of times that kind of hardware is a bitch to get running again.

    So thanks for keeping up updated as much as you did. For the hours you didn’t update us, you were busy working and just had no new info, so why waste time telling us that? You did just fine. I’m sure next time there won’t be another alignment like this. And I’m sure you’re going to take a bite out of your building’s hide for the failed generators. :)

    Keep it up folks.

    Comment by Grey Hodge — September 15, 2005 #

16. 16

    [...] DreamHost Blog » Router Issues Fixed! [...]

    Pingback by rbw blog » Router Issues Fixed! — September 15, 2005 #

17. 17

    Hey guys! Keep up the good work, you’ve done a tremendous effort in bringing things back to normal. What can I say? I simply LOVE dreamhost.

    Comment by clea — September 15, 2005 #

18. 18

    Time to get those new Supervisor 720’s. Question though, from the sounds of the design, it appears like there is a single Catalyst 6500 with dual supervisor engines. Wouldn’t it be more ideal to have a dual chassis configuration so that if you have supervisor failure you failover to a completely different set of hardware? Never mind the fact that the chassis itself is a SPOF? I know I know, it’s costly, but again invaluable for such events that have just taken place.

    BTW: Invest in some Global Site Selectors and set up at the very least a skelton datecenter elsewhere, preferably on the east coast. With events like what has occured on the Gulf Coast and your recent power outages, you need geographic diversity.

    Perry
    (he who builds datacenters for a living)

    Comment by Perry — September 15, 2005 #

19. 19

    I just got your email about this. Thanks for keeping us informed. Sometimes, things like this can’t be avoided and it’s all in how you resolve it. You guys are doing a great job, as usual. You’re definitly the host with the most.

    Comment by Melisa — September 15, 2005 #

20. 20

    Thanks everybody for bringing everything back up to normal. That’s great. Thanks also for sending that explanation email that brought me here and to the statuts page. I truly appreciate the fact that you guys were so upfront about it. Very profesional way of handling things, despite the problems at hand. It just confirms (again) that I made the right choice by switching to DreamHost a couple years back.

    Comment by Denis Boudreau — September 15, 2005 #

21. 21

    Having worked during 5 years for a web hosting and network consulting provider, I was surprised the level of stress I went through with this outage: mostly none! I had an unrelated problem during the next night and it was answered within minutes! I can only dream I would have reached such levels of customer service at my previous business.

    Although I have many sites hosted with DH - some of them 24/7 businesses - the incredible level of detail and information provided (Google news “Los Angeles Power”!), not to mention the occasional joke, let me explain easily to my customers what was happeninng.

    One of them asked “do you think it’s going to be back soon ?” - well, I guess bringing power back to >500 000 people becomes a priority quite quickly but it’s also important we know what happened and what is going to happen to address all that went wrong. Well, even that has been adressed, so what can I ask more - ah, wait, there´s pictures… and a blog… :)

    Bravo DH :D

    Comment by Fabian Rodriguez — September 15, 2005 #

22. 22

    I asked a small question during the melee and was promptly and courteously answered; I had no idea bigger problems were going on. Bravo DH.

    I just tried a Google search for “dreamhost emergency status page”; I had to go to the second page of results to find the first reference (a blog entry). Consider seeing what you can do to juice up the metadata for that page so it is discoverable by those who have not bookmarked it (or can’t remember what browser they bookmarked it in, etc.). Or ask all DH users to blog about it at least once. ;-)

    Comment by K.G. Schneider — September 15, 2005 #

23. 23

    Hey - you guys did a great job responding in a situation you did not create. Thanks for all the hard work!

    Comment by Dana — September 15, 2005 #

24. 24

    Anyone not finding the same level of humor in this that the Dreamhost team is? My boss certainly didn’t when I told him we were down for 12 hours and then ran slow for the next two business days. But I’ll email him this blog… maybe that will help.

    Comment by Ted — September 15, 2005 #

25. 25

    Great job guys.

    Comment by Pete — September 15, 2005 #

26. 26

    Bloody well done

    Comment by James — September 15, 2005 #

27. 27

    thanks for all the hard work. but in all these explanations, i haven’t seen any reference to email. what do you think happened during the downtime? did it bounce? did it get stuck somewhere? i know i didn’t get the normal amount on a workday afternoon. and were all our emails that we sent during the down time truly sent?

    also, this morning i haven’t gotten any email. i’m at a hotel on their system, but it worked fine last night. wondering if this is a problem on your end…?

    Comment by mike — September 15, 2005 #

28. 28

    Kudos to you folks again — I thought your communication during this incident was pretty good. The bldg.’s power excuse is less than sufficient though… I trust you’ll handle it.

    And I’ve got to second the suggestion below about looking into geographic diversity. I’m sure that not doing that sort of thing has alllowed DH to be the lean, mean fighting machine that is and to focus on the excellent services that you have. BUT, you might want to put that option on the table for discussion anyway…

    Best.

    Comment by JT — September 15, 2005 #

29. 29

    We appreciate all the comments! To address a few of them…

    Someone asked about the level of humor we’re keeping about this event. We don’t find this situation funny like we find ‘The Office’, but we do what we can to keep our own moods high to help deal with the stress. In our business you have to keep things in perspective a little bit or you’ll burn out.

    And to the people talking about geographic diversity… We are always discussing system improvements we could make and weighing their cost (human time as well as dollars) versus the potential cost of not making them. The same goes for equipment and software purchasing decisions. We also always have a purchasing time-line in mind that allows us to accommodate our growth in the way we have determined to best meet the needs of our customers. After this particular outage we will definitely have to revisit our existing plans and make some adjustments. Sorry for the ‘not really saying much but using a lot of words’ response… We will definitely be considering all of our options!

    Comment by Dallas — September 15, 2005 #

30. 30

    Makes me realize how vulnerable we all are. I’m considering moving my website to a St. Louis or Des Moines hosting company. I mean, how often do those places suffer catastrophic disasters? I think CA is due for a big one soon, and we are all fucked when that happens. Better yet Dreamhost ought to move its shop to a podunk Midwestern city!

    Comment by DSM — September 15, 2005 #

31. 31

    Thanks DH for putting so much effort in getting all the servers back up considering the effects of the outage. I didn’t receive any complaints from my visitors, so the downtime must have been short for me. Restarting everything after a complete shutdown and solving such a weird router issue is quite a feat. :) I’m glad I have all of my domains hosted through you guys!

    -A Happy DreamHost Customer

    Comment by Borak — September 15, 2005 #

32. 32

    Even the big guys with millions invested in power generation, emergency generators and battery backups can go. Just even ask the manager at energy.rochester.edu what happened during the blackout a couple years ago that hit the north east. The entire hospital (with the exception of the OR) lost power and battery backup. Now keep in mind the hospital produces it’s own power in addition to using power off the grid. Once that grid went down it caused the generators (not the emergency generators) to go offline. The emergency generators clicked on but some failed leaving much of the building dark for a little while.

    These systems get tested every month and yet there was failure. &#$% happens!

    With that aside for me all i needed to know that there might be a problem with my site was just cnn.com reporting a major power outage in los angelas.

    Comment by Nick — September 15, 2005 #

33. 33

    Just want to thank you guys for all the hard work! I know you probably don’t hear it nearly enough.

    Comment by blue_halo — September 15, 2005 #

34. 34

    “Anyone not finding the same level of humor in this that the Dreamhost team is? My boss certainly didn’t when I told him we were down for 12 hours and then ran slow for the next two business days. But I’ll email him this blog… maybe that will help.”

    I’m not. It cost us thousands in sales and hundreds in profit. But from the response I got from the support staff when inquiring for an update during the outage, that really wasn’t important. Of course, this post will most likely be deleted soon as my others were earlier.

    Comment by Brian Roach — September 15, 2005 #

35. 35

    “Anyone not finding the same level of humor in this that the Dreamhost team is? My boss certainly didn’t when I told him we were down for 12 hours and then ran slow for the next two business days. But I’ll email him this blog… maybe that will help.”

    I’m certainly not. It cost us thousands in sales and hundreds in profits. The response from dreamhost support during the 2 days our website was unusable was pretty much “We have no idea when this will be fixed, sorry for your luck”.

    Given that in the last year, this outage is not a unique occurance, we will be looking for a more professional hosting company.

    Of course, I’m banned from posting such negative comments to this blog currently and am having to use an anonymous proxy in mexico to do so (PS. I have many in a list … deleting this post like my others will simply make me replace it).

    Brian Roach
    SpeedWerks.com

    Comment by Unhappy Dreamhost Customer — September 15, 2005 #

36. 36

    Brian, sorry you feel that way, but as Nick mentioned ‘&#$% happens!’. If you don’t account for the occasional outage in your business model for hosting a WEB BASED STORE, then it’s nobody’s fault but your own. If you’re hosting a money making site and you depend on that income, you need to plan for a worst case scenario (which this was far from). Am I happy that my own BUSINESS sites were down for hours? No. But I realize that Dreamhost did all they could with the situation they were given. If you want to keep bitching about something that was out of their control, go to another host and see how well THEY handle an outage like this!

    Comment by Mark — September 15, 2005 #

37. 37

    To Brian Roach:

    I dub you, Asshat of the Year.

    Congradulations. Go somewhere else if you dont like the hosting. For you to run a “professional” site and then to trash the host that is trying to do all they can to get you up and running isn’t very professional. I hope your customers know how you are treating others.

    Comment by dmacgavin — September 16, 2005 #

38. 38

    But I have some problems. Some times my computer become work slower

    Comment by Dmitriy — December 19, 2005 #

Sorry, the comment form is closed at this time.