38 Responses to “Router Issues Fixed!”

1. dmacgavin Says:
   September 14th, 2005 at 6:58 pm

   Haha first reply post! ;) Great job. Glad to see that issue is resolved.

2. pint Says:
   September 14th, 2005 at 7:01 pm

   Woah! Now thats great news!
   I was just browsing my slo-mo gallery when suddenly it all went back to holyspeeds :p
   thanks!

3. Matttail Says:
   September 14th, 2005 at 7:08 pm

   really great to hear this is finally resolved. Hope you guys sleep well finally – not that things are back to “normal”

   Thanks for the great service you provide.

   -Matttail

4. Bob Says:
   September 14th, 2005 at 7:42 pm

   Yay, verything seems to be working at normal speeds now, thanks!

5. Denis Says:
   September 14th, 2005 at 7:53 pm

   thank you dreamhost!!!

6. Alfred Says:
   September 14th, 2005 at 8:19 pm

   Dreamhost beeing down only happens once in while. No host here in the Netherlands can match with Dreamhost, they are also much more expensive compared to Dreamhost so don’t worry. I like and really appreciate your blog, it keeps us informed and it all sounds so human. Thanks.

7. Sylvester Says:
   September 14th, 2005 at 8:31 pm

   Hey, don’t keep us in suspense! Tell us the culprit option. ;-)

8. Patrick Says:
   September 14th, 2005 at 8:36 pm

   Thanks so much for your help.

   Hey, don’t keep us in suspense! Tell us the culprit option.

   You must tell us. :)

9. Scott Says:
   September 14th, 2005 at 9:27 pm

   Excellent, I just checked and my site’s back up to its usual great speed. Thanks Dreamhost!

10. ryan Says:
    September 14th, 2005 at 9:39 pm

    thanks DH
    it is so damn fast rite now

11. Neil Says:
    September 14th, 2005 at 10:04 pm

    Good job guys… and ignore all the stupid criticism in the forums. Looks to me like most of this was way beyond your control.

12. Calrion Says:
    September 14th, 2005 at 10:26 pm

    I just want to say thanks. There’s no way you could have avoided these issues, or all the cranky “what the!?” responses from customers; the way you’ve handled the initial problem, the restoration and resolution, and the aftermath has been excellent.

    Rather than shaking my confidence, this incident has proven that I made the right decision moving to Dreamhost–the extra money is well worth it!* Keep it up!

    * The extra money is mostly because I upgraded my plan, DH represent excellent value for money.

13. Deamos Says:
    September 14th, 2005 at 11:34 pm

    Woohoo! Everythings back to normal! Great work guys, having been through a large scale powerfailure once or twice in my life with regards to hosting and such I know it sucks.

    Thanks for all your hard work in getting everything back to running the way it’s supposed to. You guys are great and are worth every cent I spend to host with you.

14. ivan raszl Says:
    September 14th, 2005 at 11:46 pm

    back to normal. thanks for the work guys!

15. Grey Hodge Says:
    September 15th, 2005 at 12:18 am

    I knew you guys would get it all ironed out ASAP. I’m in the crowd that could see that this was nothing more than one of those times when all your potential points of failure failed all at once. It’s rare, but it does happen, and when it does all you can do is fix it as fast as you can. No amount of grumbling or whining will help, and no one can really fault you much for being caught off guard by a cascade of failures. You didn’t black out LA, you didn’t oversell the building’s backup power capacity, and you didn’t blow the power switches or router. Hardware that runs 24/7 HATES to lose power unexpectedly, and even with scheduled maintenence a lot of times that kind of hardware is a bitch to get running again.

    So thanks for keeping up updated as much as you did. For the hours you didn’t update us, you were busy working and just had no new info, so why waste time telling us that? You did just fine. I’m sure next time there won’t be another alignment like this. And I’m sure you’re going to take a bite out of your building’s hide for the failed generators. :)

    Keep it up folks.

16. rbw blog » Router Issues Fixed! Says:
    September 15th, 2005 at 12:29 am

    [...] DreamHost Blog » Router Issues Fixed! [...]

17. clea Says:
    September 15th, 2005 at 2:00 am

    Hey guys! Keep up the good work, you’ve done a tremendous effort in bringing things back to normal. What can I say? I simply LOVE dreamhost.

18. Perry Says:
    September 15th, 2005 at 3:48 am

    Time to get those new Supervisor 720’s. Question though, from the sounds of the design, it appears like there is a single Catalyst 6500 with dual supervisor engines. Wouldn’t it be more ideal to have a dual chassis configuration so that if you have supervisor failure you failover to a completely different set of hardware? Never mind the fact that the chassis itself is a SPOF? I know I know, it’s costly, but again invaluable for such events that have just taken place.

    BTW: Invest in some Global Site Selectors and set up at the very least a skelton datecenter elsewhere, preferably on the east coast. With events like what has occured on the Gulf Coast and your recent power outages, you need geographic diversity.

    Perry
    (he who builds datacenters for a living)

19. Melisa Says:
    September 15th, 2005 at 5:35 am

    I just got your email about this. Thanks for keeping us informed. Sometimes, things like this can’t be avoided and it’s all in how you resolve it. You guys are doing a great job, as usual. You’re definitly the host with the most.

20. Denis Boudreau Says:
    September 15th, 2005 at 5:48 am

    Thanks everybody for bringing everything back up to normal. That’s great. Thanks also for sending that explanation email that brought me here and to the statuts page. I truly appreciate the fact that you guys were so upfront about it. Very profesional way of handling things, despite the problems at hand. It just confirms (again) that I made the right choice by switching to DreamHost a couple years back.

21. Fabian Rodriguez Says:
    September 15th, 2005 at 6:15 am

    Having worked during 5 years for a web hosting and network consulting provider, I was surprised the level of stress I went through with this outage: mostly none! I had an unrelated problem during the next night and it was answered within minutes! I can only dream I would have reached such levels of customer service at my previous business.

    Although I have many sites hosted with DH – some of them 24/7 businesses – the incredible level of detail and information provided (Google news “Los Angeles Power”!), not to mention the occasional joke, let me explain easily to my customers what was happeninng.

    One of them asked “do you think it’s going to be back soon ?” – well, I guess bringing power back to >500 000 people becomes a priority quite quickly but it’s also important we know what happened and what is going to happen to address all that went wrong. Well, even that has been adressed, so what can I ask more – ah, wait, there´s pictures… and a blog… :)

    Bravo DH :D

22. K.G. Schneider Says:
    September 15th, 2005 at 6:33 am

    I asked a small question during the melee and was promptly and courteously answered; I had no idea bigger problems were going on. Bravo DH.

    I just tried a Google search for “dreamhost emergency status page”; I had to go to the second page of results to find the first reference (a blog entry). Consider seeing what you can do to juice up the metadata for that page so it is discoverable by those who have not bookmarked it (or can’t remember what browser they bookmarked it in, etc.). Or ask all DH users to blog about it at least once. ;-)

23. Dana Says:
    September 15th, 2005 at 6:44 am

    Hey – you guys did a great job responding in a situation you did not create. Thanks for all the hard work!

24. Ted Says:
    September 15th, 2005 at 7:05 am

    Anyone not finding the same level of humor in this that the Dreamhost team is? My boss certainly didn’t when I told him we were down for 12 hours and then ran slow for the next two business days. But I’ll email him this blog… maybe that will help.

25. Pete Says:
    September 15th, 2005 at 8:49 am

    Great job guys.

26. James Says:
    September 15th, 2005 at 9:12 am

    Bloody well done

27. mike Says:
    September 15th, 2005 at 9:21 am

    thanks for all the hard work. but in all these explanations, i haven’t seen any reference to email. what do you think happened during the downtime? did it bounce? did it get stuck somewhere? i know i didn’t get the normal amount on a workday afternoon. and were all our emails that we sent during the down time truly sent?

    also, this morning i haven’t gotten any email. i’m at a hotel on their system, but it worked fine last night. wondering if this is a problem on your end…?

28. JT Says:
    September 15th, 2005 at 10:24 am

    Kudos to you folks again — I thought your communication during this incident was pretty good. The bldg.’s power excuse is less than sufficient though… I trust you’ll handle it.

    And I’ve got to second the suggestion below about looking into geographic diversity. I’m sure that not doing that sort of thing has alllowed DH to be the lean, mean fighting machine that is and to focus on the excellent services that you have. BUT, you might want to put that option on the table for discussion anyway…

    Best.

29. Dallas Says:
    September 15th, 2005 at 11:16 am

    We appreciate all the comments! To address a few of them…

    Someone asked about the level of humor we’re keeping about this event. We don’t find this situation funny like we find ‘The Office’, but we do what we can to keep our own moods high to help deal with the stress. In our business you have to keep things in perspective a little bit or you’ll burn out.

    And to the people talking about geographic diversity… We are always discussing system improvements we could make and weighing their cost (human time as well as dollars) versus the potential cost of not making them. The same goes for equipment and software purchasing decisions. We also always have a purchasing time-line in mind that allows us to accommodate our growth in the way we have determined to best meet the needs of our customers. After this particular outage we will definitely have to revisit our existing plans and make some adjustments. Sorry for the ‘not really saying much but using a lot of words’ response… We will definitely be considering all of our options!

30. DSM Says:
    September 15th, 2005 at 12:17 pm

    Makes me realize how vulnerable we all are. I’m considering moving my website to a St. Louis or Des Moines hosting company. I mean, how often do those places suffer catastrophic disasters? I think CA is due for a big one soon, and we are all fucked when that happens. Better yet Dreamhost ought to move its shop to a podunk Midwestern city!

31. Borak Says:
    September 15th, 2005 at 1:00 pm

    Thanks DH for putting so much effort in getting all the servers back up considering the effects of the outage. I didn’t receive any complaints from my visitors, so the downtime must have been short for me. Restarting everything after a complete shutdown and solving such a weird router issue is quite a feat. :) I’m glad I have all of my domains hosted through you guys!

    -A Happy DreamHost Customer

32. Nick Says:
    September 15th, 2005 at 4:20 pm

    Even the big guys with millions invested in power generation, emergency generators and battery backups can go. Just even ask the manager at energy.rochester.edu what happened during the blackout a couple years ago that hit the north east. The entire hospital (with the exception of the OR) lost power and battery backup. Now keep in mind the hospital produces it’s own power in addition to using power off the grid. Once that grid went down it caused the generators (not the emergency generators) to go offline. The emergency generators clicked on but some failed leaving much of the building dark for a little while.

    These systems get tested every month and yet there was failure. &#$% happens!

    With that aside for me all i needed to know that there might be a problem with my site was just cnn.com reporting a major power outage in los angelas.

33. blue_halo Says:
    September 15th, 2005 at 7:53 pm

    Just want to thank you guys for all the hard work! I know you probably don’t hear it nearly enough.

34. Brian Roach Says:
    September 15th, 2005 at 9:37 pm

    “Anyone not finding the same level of humor in this that the Dreamhost team is? My boss certainly didn’t when I told him we were down for 12 hours and then ran slow for the next two business days. But I’ll email him this blog… maybe that will help.”

    I’m not. It cost us thousands in sales and hundreds in profit. But from the response I got from the support staff when inquiring for an update during the outage, that really wasn’t important. Of course, this post will most likely be deleted soon as my others were earlier.

35. Unhappy Dreamhost Customer Says:
    September 15th, 2005 at 9:46 pm

    “Anyone not finding the same level of humor in this that the Dreamhost team is? My boss certainly didn’t when I told him we were down for 12 hours and then ran slow for the next two business days. But I’ll email him this blog… maybe that will help.”

    I’m certainly not. It cost us thousands in sales and hundreds in profits. The response from dreamhost support during the 2 days our website was unusable was pretty much “We have no idea when this will be fixed, sorry for your luck”.

    Given that in the last year, this outage is not a unique occurance, we will be looking for a more professional hosting company.

    Of course, I’m banned from posting such negative comments to this blog currently and am having to use an anonymous proxy in mexico to do so (PS. I have many in a list … deleting this post like my others will simply make me replace it).

    Brian Roach
    SpeedWerks.com

36. Mark Says:
    September 15th, 2005 at 11:45 pm

    Brian, sorry you feel that way, but as Nick mentioned ‘&#$% happens!’. If you don’t account for the occasional outage in your business model for hosting a WEB BASED STORE, then it’s nobody’s fault but your own. If you’re hosting a money making site and you depend on that income, you need to plan for a worst case scenario (which this was far from). Am I happy that my own BUSINESS sites were down for hours? No. But I realize that Dreamhost did all they could with the situation they were given. If you want to keep bitching about something that was out of their control, go to another host and see how well THEY handle an outage like this!

37. dmacgavin Says:
    September 16th, 2005 at 12:13 pm

    To Brian Roach:

    I dub you, Asshat of the Year.

    Congradulations. Go somewhere else if you dont like the hosting. For you to run a “professional” site and then to trash the host that is trying to do all they can to get you up and running isn’t very professional. I hope your customers know how you are treating others.

38. Dmitriy Says:
    December 19th, 2005 at 12:34 am

    But I have some problems. Some times my computer become work slower