What a week!

July 22, 2006 on 10:23 pm | In Foobars, Updates by Dallas Kashuba |

Our string of bad luck is not over yet. After the file server crash last weekend and the subsequent internal network issues that we have been cleaning up all week, this evening brought us a Los Angeles area power outage that took down our data center (again, we know…). This time the generators worked fortunately, but the UPS system did not. That means power was out for the data center for the few minutes it takes the generators to come online. Long story short… most of our equipment is all back online now and we’ll be working through the night making sure all of your websites are back up, too.

Right now we’re still down one fileserver and that’s being worked on. There’s also still a lot of websites down and those are being brought up as fast as possible.

Keep an eye on the off network DreamHost Status page for updates as we know more.

103 Comments

Pages: [1] 2 3 » Show All

  1. 1

    The thing that Dallas didn’t say, is how smooth DreamHost is taking it, I had my doubts about the company before I signed up, but this week has proven that I made no mistake, and that the admins know what they are doing.

    Good job DreamHost, lets hope next week brings better luck.

    (p.s. the power outage reminds me of the one that Auckland (NZ) had the other month, that took a few Datacentres out for similar reasons. Unavoidable)

    Comment by Nigel Jones — July 22, 2006 #

  2. 2

    World War III, Israel, Lebanon and DreamHost!…

    DreamHost’s recent technical difficulties has made DreamHost one of the most searched keywords in blog search engine Technorati.

    Maybe not too surprising, but what is making this a bit disturbing is the other contenders for the Top Searches: Wor…

    Trackback by Unofficial DreamHost Blog — July 22, 2006 #

  3. 3

    So much for the promises after the last data center power failure.

    Comment by Vince — July 22, 2006 #

  4. 4

    I’m a new user, and something I already appreciate is how open you are about technical issues. Any business that can admit that things are screwed up and have to be re-screwed gets a few extra points in my book. My old host didn’t even have a status page…

    One suggestion for your status page, though: could you add a relative time stamp (xx.xx hours ago)? You now, so that we lazy-minded overseas users can know how long ago something happened without having to remember how many hours off PST is. :)

    Comment by Andreas — July 22, 2006 #

  5. 5

    This second incident of poor backup power from our building does not bode well. We are already discussing getting at least some of our own backup power to keep our most critical hardware functioning in the event of another power outage. That’ll be a blast… moving equipment around and re-routing all the power cords.

    Anyway, most everything’s back up now. We’re still chugging along tying up loose ends.

    Comment by Dallas — July 23, 2006 #

  6. 6

    Hope that this bad run will end soon. I just moved over to DH and all of this started happening… Clients are not to impressed at the moment..

    Comment by drnibbles — July 23, 2006 #

  7. 7

    Would this be the cause of the e-mail outage currently affecting my domain?

    Comment by Volt — July 23, 2006 #

  8. 8

    One (small?) thing that should bother dreamhosters is that people don’t find the status infomation on the dreamhost homepage.

    When I go to the homepage of dreamhost, I don’t see anything about “status” or “downtime”. This is why, I guess, people go to T’rati anfterwards.

    Should be taken care of.

    Something like “last 5 headlines from our official blog”.

    Comment by Hanan Cohen — July 23, 2006 #

  9. 9

    Nice post, but it doesn’t explain why e-mail’s been inaccessible for about 12 hours now.

    Comment by Chuck — July 23, 2006 #

  10. 10

    I understand that things have been very busy for the DreamHost folks lately, but please, can someone find the time to fix the DreamHost forums? The constant 500 server errors are driving me insane!

    Mark

    Comment by Raz2133 — July 23, 2006 #

  11. 11

    Lads… keep up the good work… We realise you are putting in the most horrific hours during a heat wave… We really appreciate it and (i hope) most of us understand that somethings are can be out of your control.. All the best for getting it all 100% and with your discussions with the DataCenter crew about the system failures.

    Comment by Skip — July 23, 2006 #

  12. 12

    One question - do you pool ALL your servers in one area, or do you spread it (eg, half on west coast, half on east coast) ?

    Reason for asking is that the power outage was blamed on the weather.
    If you have your servers spread on different places, than at least half of it would be still online - which means less problem for you too.

    Just an idea.

    Comment by harry — July 23, 2006 #

  13. 13

    My clients are frying about the recent reliability issues. I am loath to move them (over 100), but clearly DH needs to spend more of its profit margin on hardware!

    Can we all say, together, “R E D U N D A N C Y” ! !

    And, if your landlord can’t seem to get its power systems stable maybe they need to be sued for non-performance.

    We’re using DH for business, not for play. It’s got to be more reliable.

    /jad

    Comment by Jad — July 23, 2006 #

  14. 14

    Hmm, wiki.dreamhost.com doesn’t seem to work, discussion.dreamhost.com doesn’t seem to work, I can’t ssh into my server.

    But Dreamhost says everything is back runing.

    Comment by Vince — July 23, 2006 #

  15. 15

    The truth, they don’t have local UPS so they can’t check them. The building is supposed to have UPS along with generators.

    DreamHost should probably have local UPS (on each rack) that allow for 5 or 6 minutes without power. That would also allow them to have the UPS devices shut down the hardware if after 5 mins the building doesn’t have power so the hardware doesn’t fail.

    IMO, they should’ve already had this (at least on their core hardware, routers, switches, filers, and panel).

    Comment by Will — July 23, 2006 #

  16. 16

    I still can’t access my website http://www.bookdiva.net, and keep getting this dialog.

    Site Temporarily Unavailable
    We apologize for the inconvenience. Please contact the webmaster/ tech support immediately to have them rectify this.

    error id: “bad_httpd_conf”

    The dreamhost status update states that as of July 23rd, at 3:31 a.m., “All websites, email and all other services should be up and running. If you are still experiencing problems with one of your services, contact our support team right away.”

    As if July 23rd, 4:01 pm, CST, I can’t access my website.

    Comment by BookDiva — July 23, 2006 #

  17. 17

    “One suggestion for your status page, though: could you add a relative time stamp (xx.xx hours ago)? You now, so that we lazy-minded overseas users can know how long ago something happened without having to remember how many hours off PST is. :)”

    +1 to that, I have been thinking the same thing since the new status page went up

    Comment by Michael — July 23, 2006 #

  18. 18

    Are other people still having email trouble, too? I set up a new email account at a new site yesterday just before everything went all ka-blooey and I still can’t receive anything at that new account.

    I understand the problem with the power outage, but it’s the middle of the day on Sunday now and the Dreamhost status site says everything is a-ok. However, my support page in the Panel says that the “datacenter has experienced some power fluxuations” for over 24 hours and I’ve had an open support ticket for several hours with no response. Is everything really ok or what’s the problem?

    Comment by badMike — July 23, 2006 #

  19. 19

    I am also having email problems: the server drops the connection when I try to authenticate.

    Comment by Jay Goodman Tamboli — July 23, 2006 #

  20. 20

    Yup, it’s been happening to me intermittently since the file server problem started.

    Comment by Vince — July 23, 2006 #

  21. 21

    They probably didn’t push the updates out yet after getting things squared away. Can you send to that email account from another, or does it bounce?

    Comment by James Homuth — July 23, 2006 #

  22. 22

    Why buy your own UPS if UPS service is part of the rent you’re paying on the datacenter? Are your landlords prepared to cut your rent since they’ve consistently provided inconsistent power?

    Comment by Gabe — July 23, 2006 #

  23. 23

    Yeah, my email is still down as well. I love you guys, but I really could use my email before morning, eh?

    Comment by Christopher Gillen — July 23, 2006 #

  24. 24

    Haven’t had any email all day. Screw the webservers. Get the IMAP servers up.

    Call2biz is f%^$*ing stupid. Thanks for the ad, now piss off.

    Learn2Type.

    Comment by Ryan — July 23, 2006 #

  25. 25

    I hope they fix all this soon. My website keeps going offline every other hour. =(

    Comment by Seraphic — July 24, 2006 #

  26. 26

    Hang on Dreamhost! :)

    Comment by Sara — July 24, 2006 #

  27. 27

    3 days past and things are still not operating normally.

    Comment by * — July 24, 2006 #

  28. 28

    And nothing on dreamhoststatus.com about any lingering issues.

    Comment by mlee — July 25, 2006 #

  29. 29

    The new dreamhost web 2.0 logo features here:

    http://enthree.com/files/random/web2logos/
    (third row)

    (related to this post)

    Comment by Nathan — July 25, 2006 #

  30. 30

    WTF?! I just checked my site this morning and it’s down yet AGAIN! There is nothing on the Dreamhost status page to indicate that there SHOULD be a problem. So again I ask, WTF?!

    Comment by NotImpressed — July 25, 2006 #

  31. 31

    My site’s up and working fine… so I haven’t the faintest. Probably an individual server issue.

    Comment by James Homuth — July 25, 2006 #

  32. 32

    My sites are all down as well. It comes up early in the morning and late at night, but during the day, EVERY DAY SINCE LAST MONDAY, it is unresponsive. This is ridiculous.

    Comment by JR — July 25, 2006 #

  33. 33

    My sites are all down as well. It comes up early in the morning and late at night, but during the day, EVERY DAY SINCE LAST MONDAY, it is unresponsive. This is ridiculous!

    Comment by JR — July 25, 2006 #

  34. 34

    I am sincerely appreciative of the status page and how open DH is with customers about the technical issues they are facing, but, I must admit, the past week had me frustrated more than once. I am just glad that status page is still working despite the huge hits it has probably been seeing. I’m at the point where I keep the xml feed on my google start page.

    Comment by Ian Clifton — July 25, 2006 #

  35. 35

    Maybe Dreamhost will be nice enough to cover all of the ad sales I’m losing off my website. Now it’s JUST my ad server database that’s down… they’re really sticking it to me…

    Comment by TM — July 26, 2006 #

  36. 36

    The server my sites on seems to be having a load problem, I’ve let it go the last day or 2, but it’s beginning to get annoying… any chance you could take a look? http://www.rawkstar.net/ http://blog.rawkstar.net/ http://forum.rawkstar.net/ - they’re all affected. load was about 80 yesterday, went down to 4 or 5 then back up to 50 odd today.

    Cheers!

    Comment by Bob — July 26, 2006 #

  37. 37

    Wondering is the panel down tooo..

    Leon

    Comment by Leon — July 27, 2006 #

  38. 38

    You call it bad luck ? I call it inefficiency. Nothing in technology depends on luck.

    Comment by Alex Perez — July 27, 2006 #

  39. 39

    For the last 2 weeks my site has also been down about 10 times a day, I’m beginning to get very frustrated with it all. The status page may be a good idea, but doesn’t help with why my site is down - never correlates to the times mine is offline.

    Another week and I’ll be switching hosting companies! I love dreamhost for last 2 years, but enough is enough.

    Comment by Emma — July 27, 2006 #

  40. 40

    I’ve been a loyal Dreamhost customer for over three years, but these recent outages are absolutely unacceptable. I’d gladly pay twice the current rate if it means I’d get some actual service. It would be worth the money.

    Sorry, Dreamhost. I am also going to switch companies if this keeps up for another week.

    Comment by Kurt — July 27, 2006 #

  41. 41

    They NEVER say anything about all the email problems…emails not checkable, emails returned to sender, emails not going through to hotmail and yahoo. Why dont’ they say anything about ALL the MANY MANY email issues that have been ongoing for over the past week!!!

    Comment by Eric Matthew — July 27, 2006 #

  42. 42

    The dreamhoststatus.com page is sadly lacking. With so many issues all going on at once it’s impossible to keep up with what’s resolved and what’s not.

    “Intermittent email problems” got posted 1 day 23 hours ago and it says “… get everything back to normal soon”. Well, I’m STILL having the same problems. 2 further (unrelated??) postings have been made over the top, but no update of the email problem.

    We need a REAL TIME server status page so at any time we can see what’s killing us. This instead of being continually being told by the “SYSTEM-WIDE status” that NOTHING is wrong.

    I too have been here for 3 years and it’s been great up until 2 weeks ago but … my clients are asking to be let out of their contracts and I’m thinking of going too.

    PLEASE PLEASE PLEASE provide better server/system status reports ….. better still, sort out the issues.

    Comment by Kenton — July 27, 2006 #

  43. 43

    [...] This DreamHost Blog post is a pretty good example of what can happen when you expose yourself to your customers. You get the good: The thing that Dallas didn’t say, is how smooth DreamHost is taking it, I had my doubts about the company before I signed up, but this week has proven that I made no mistake, and that the admins know what they are doing. [...]

    Pingback by That Not So Fresh Feeling » The Pain of Shared Web Hosting — July 27, 2006 #

  44. 44

    [...] 这两天的问题主要是因为DreamHost的主机那边出现停电故障。一般他们都会争取在两天之内尽快恢复,但是这次拖了两天问题还在,于是给他们发了email。在1个钟头内收到他们的回复,原来是域名的IP已经被他们更换成新的了,只要改一下就可以。 [...]

    Pingback by 牛犊回来了! - 初生牛犊 — July 27, 2006 #

  45. 45

    Sorry for the off-topic comment here: I looked in vain for a contact form or an email to contact you directly.

    Could you please email me ASAP regarding a pretty urgent and important problem with your site (and no, I am not trying to sell you any sort of pharmaceuticals or poker games)…

    Comment by dr Dave — July 28, 2006 #

  46. 46

    I’ve had enough. I will start looking for another web host today for all of my client sites. The number of outages is well beyond any sane person’s limit.

    Comment by D — July 28, 2006 #

  47. 47

    I was really happy when I switched to Dreamhost 3 yrs ago and have recommended them over & over to my clients & colleagues. But those days are at an end.

    DH — I love you, but it’s getting to be like dating an alcoholic. You’re in denial about the extent of your problem, which has gotten progressively worse. All I hear from clients these days is “What is wrong with DH? Why is my website / email / etc. down?!! I thought you said they were the best!!!”

    You were, once upon a time. We had a lot of good times, but I can’t make excuses for your behavior anymore. I’ve tried asking nicely, I’ve tried begging, I’ve tried ultimatums. None of that has worked.

    I am now looking to get out of this relationship. I don’t expect my next ISP to be as friendly as you, or have as sexy a web panel, but hopefully it’ll be stable and reliable. And that’s something I value in a long-term relationship.

    Comment by Matthew B — July 28, 2006 #

  48. 48

    I’m trying to be patient, but if I need to set my homepage to Dreamhost Status so I know whether or not to bother with visiting my sites before I sit down, let me know.

    Comment by tony — July 28, 2006 #

  49. 49

    Maybe it’s time to move to a new datacenter?

    Comment by Ted — July 28, 2006 #

  50. 50

    I agree. Let’s start coming up with some recommendations for hosts to switch to.

    Notice how no matter what goes down Dreamhost.com stays live. It’s about time that Dreamhost sees how not being prepared and not treating customers as they treat themselves can hurt their bottom line, as it does ours.

    If the other hosts were smart they would create a custom promo code offering a discount from former Dreamhost customers r

    Comment by brian — July 28, 2006 #

Pages: [1] 2 3 » Show All

Sorry, the comment form is closed at this time.

Powered by WordPress. Pool theme by Borja Fernandez, modified by DreamHost.
Entries and comments feeds. ^Top^