Anatomy of a(n ongoing) Disaster..
August 1, 2006 on 12:29 pm | In Foobars, Insider View, Updates by Josh Jones |
What a three weeks…
As I’m sure most of you already know, we’ve had nothing but troubles, large troubles, for pretty much the last three weeks. A lot of these troubles were our fault, a couple of them were at least ostensibly beyond our control, and they all compounded each other.
Here I’ll try and go into as much detail as possible about what happened, why, and the steps we’re taking to stop this sort of thing from ever happening again. I can’t excuse what happened, just apologize and hopefully elucidate.
Ironically, all the recent disasters stem somewhat from us attempting to take some proactive steps to head off any sort of future power outages like the kind we experienced last year.
The Back Story
As some of you may know, we are co-located with Switch and Data in The Garland Building in downtown L.A. To say we’re co-located is a bit misleading though, since we’re now basically 95% of their data center.
Why don’t we have our own data center?
Because, believe it or not, we’re still not big enough for it to make sense. Even now, we only use about 1000 sq ft of data center space.. for it to really start to make sense to get our own space, we’d have to be using around 2500 sq ft. Mainly because when you buy a data center, you want to get one big enough to handle a lot of growth.. and although it’s cheaper per square foot than co-locating, you have to pay for all the space you’re not using yet.
And really, The Garland Building is supposed to be an excellent place for data centers. There are more than a dozen in the building. Companies like iPowerWeb, Media Temple, BroadSpire, and even MySpace (now the most popular website in the whole US!) are in there. It’s got FIVE huge generators, UPS for the whole building, on two separate power grids, and a dedicated engineering staff to make it all work flawlessly. Or so we were all assured.
Around last June though, the building informed all its data center tenants that they had essentially run out of power! Not power altogether, but the “good” power that data centers need.. i.e. ups and generator-backed power. Because Wells Fargo, who holds the master lease on the building, wasn’t sure if they were going to renew the lease when it is up in three years, they didn’t want to invest the millions of dollars to add more generators and ups to increase capacity. This is in fact the primary reason we’re still not selling any more dedicated servers .. they use too much power per dollar!
Of course, none of that was supposed to have any affect on their ability to keep the current power going in the case of an outage. September 12th, 2005 we discovered they actually couldn’t… when two of the five generators failed!
However, since then, the building has repaired and replaced the faulty generators, and given all their tenants numerous assurances that what happened before would never ever happen again.
Why didn’t we move data centers right then?
That would have been a fairly massive undertaking, resulted in even more down time, been very expensive, and actually we did look around and there weren’t any really good options for moving… data center space is becoming pretty tight (in the LA area at least) and the Garland Building is still one of the best options, believe it or not. Also, this was the first time something like this had ever happened, and it seemed pretty reasonable that it wouldn’t happen again. We even asked around and none of those other tenants mentioned above were moving, so I guess it seemed like people were generally pretty confident it was a one-time freak occurrence.
Nevertheless, we started making contingency plans, searching around for another data center that had some power and would make sense for us. Eventually, we found Alchemy, just down the hall from S+D actually, and began making arrangements for getting some space from them. They had a little bit of power available because they were moving some of their clients out to El Segundo, and because they had gotten permission from the building to install their own generator. With that generator and some UPSes they were able to convert a “dirty” power feed into “clean” (i.e. good for data center use) power.
How the troubles began.
All this took a very, very, very long time. After months of searching and negotiating with Alchemy, we still had to get Switch and Data to allow us to put a cross-connect in from their data center over to their competitors down the hall. After even more months and teeth-pulling, we finally got that up and running. In fact, we finally got the first live server up in Alchemy a little less than a month ago.
All this in an attempt to head off future power problems.
Unfortunately, shortly after setting up the new footprint, we noticed something wasn’t right. Getting to Alchemy from Switch and Data we would lose huge buckets of packets. Just as we were trying to figure out the problem, we started to have problems with one of our file servers.
This resulted in a lot of problems across the board. The web servers that mounted that filer all had problems. The mail servers that mounted that filer all had problems. In fact, one of the mail servers was mis-configured and was logging thousands of errors a second to a remote logging machine… so many in fact that it was saturating its switch and clogging up a whole chunk of our network. Which in turn caused other machines to get slow and crashy because they couldn’t get to their filers, and so on and so on.
It turned out the filer problem seemed to stem from the fact that we had one shelf of 300GB disks and one shelf of 150GB disks on it. Apparently they’re not supposed to be able to support this, or at least it’s a bad idea. So, this was entirely our fault. However, we did have a number of other filers we did this on, and we’d never had problems before. Nonetheless, we will never mix disk shelf types on a file server again.
We eventually cleared all this up.
However, the Alchemy connection problems were still ongoing.
After trying all sorts of things, we eventually decided to replace one of our distribution switches that was acting strangely with a new one. This didn’t really seem to fix the problem either. This was on Friday, July 21st.
On Saturday, July 22nd, the building lost power.
This time, the generators actually worked, but the UPS failed! Honestly, it was much better than last year’s.. but unfortunately, even a brief power outage wreaks havoc on a data center. And this one wasn’t so brief.. here’s the building’s explanation:
At around 5:21pm, on Saturday July 22nd, a brown out occurred due to record high temperatures in downtown Los Angeles. Voltage dropped due to the high demand of electrical current along with equipment failure operated by the Department of Water and Power, City of Los Angeles. This condition caused failure of “ATS-B” switch and to UPS Module #3. Engineering crews were dispatched and began repair of this damaged equipment. A power interruption was required to replace contacts in “ATS-B”.
Repair of “ATS-B” failed contacts was completed on 7-24-06. Power was restored between 4:00am and 4:30am by the Engineering department.
Thank you,
Office of the Building
So, after all the emergency filer stuff going on the previous weekend, just about the entire admin team was back last weekend, working on getting everything back up when power came back on. Even when we had power, it was in a degraded state and so the cooling wasn’t working. As temperatures rose, file servers automatically shut themselves down rather than risk being damaged by the hostile environment. Apparently, MySpace made the decision to just keep all their servers off until cooling was restored.
More network troubles..
After the power outage, we decided to just yank everything back out of Alchemy (they lost power too!) until we could figure out what was going on with the network to there. Unfortunately, this didn’t seem to fix things, and our internal (”red”) network was still really fubar. When our red network isn’t working, the panel isn’t working, webmail isn’t working, and our server configuration system starts having problems (basically, anything that connects to our internal databases).
It took us just about all of Monday to figure out (and then fix) that a lot of the file servers had bad routes after being powercycled.. and so were sending ALL their traffic through the red network, saturating it. These things are generally pretty stable and a lot hadn’t been rebooted since September 12th, 2005.. and some had apparently had their networking set up by hand instead of correctly configured via our database. We’re making sure that doesn’t happen anymore either.
More network troubles..
Once that was fixed, things generally got better. Except there was STILL strange stuff going on (causing slowness and high loads around the system, but not an actual system-wide outage), even without NFS traffic going through red, and even without anything at Alchemy. It started to look like there was a problem with one of our core routers. We called our Cisco consultant and opened a trouble ticket with Cisco themselves..
More power problems..
On Friday, July 28th, we lost power again. The building wrote:
The Garland Building experienced a dead short which resulted in a brief power outage today, July 28, 2006. The air conditioning, elevators, and the electrical utility have all been restored.
While on generator power, a dead short occurred from one of our internal telecom users. We are investigating where the dead short occurred. A follow-up memo will be sent by the end of the business day reconfirming our transfer at 11:30pm tonight. We are currently on DWP power until further notice.
And then:
The Garland Building UPS System is back on-line supplied by DWP. Diesel generators have returned to an on-call status.
The 11:30pm transfer has been cancelled due to the dead short prematurely returning us to utility power. At 4:30pm the engineers engaged the UPS System to protect all tenants at the Garland Building.
Thank you,
Office of the Building
This time, we were able to get our entire system back up much quicker and with close to no problems. Of course, it had been less than a week since our last power outage.
Alchemy was the only data center in the building who did not lose power this time.
More network troubles..
Over the weekend (this last weekend), we kept having the same ongoing weird network problems I mentioned above, and Cisco hasn’t made much progress. Yesterday, we realized the new distribution switch (an extreme) was causing spanning tree problems with the older Ciscos. Jeremy got it all figured out, but in the process it erroneously blocked our “green” (public!) network for a few brief periods, taking down everything again.
Unfortunately, that fix STILL doesn’t seem to have fixed the ongoing core network problems. We were finally able to get our tickets escalated with Cisco yesterday. It is starting to look like something may have been damaged during the first power failure, although we’re not sure. The replacement/repair cost might be around $80,000 it looks like.
And that’s where things stand today.
Our number one priority right now is getting this nagging network problem understood and fixed. Once that’s the case, we should be able to put things back in Alchemy, who didn’t lose power on Friday at least. Once things are going good there, we’ll be able to add new servers and transition old ones slowly with little to no downtime.
We’re also going to be buying our own UPSes, since we have learned we can’t trust our data center OR our building to do it. We’ll start by putting the core routers on them, then our internal databases and servers, then our file servers, and finally the hundreds of customer mail, web, and database servers.
Finally…
We’re very sorry for what happened. We definitely don’t want it to happen again, and we’re trying to take all the practical steps we can to prevent it. We never want to have another July 2006 again.
Ironically, some of the network problems seem to have stemmed from us trying to better protect ourselves from power failures. I also want to say for the record that none of these problems in my opinion stemmed from “overselling”. Rather, I’d say it’s the result of bad luck. And incompetence on our (and the building’s) part.
I don’t know if we’ll be able to change our luck, but hopefully we’ve at least learned something and will be able to become a tiny bit less incompetent in the future.
I hope you’ll all stay with us to find out.
479 Responses to “Anatomy of a(n ongoing) Disaster..”
Categories:
Archives:
- June 2009 (3)
- May 2009 (3)
- April 2009 (4)
- March 2009 (2)
- February 2009 (3)
- January 2009 (3)
- December 2008 (3)
- November 2008 (4)
- October 2008 (5)
- September 2008 (2)
- August 2008 (5)
- July 2008 (2)
- June 2008 (3)
- May 2008 (6)
- April 2008 (2)
- March 2008 (3)
- February 2008 (5)
- January 2008 (7)
- December 2007 (3)
- November 2007 (3)
- October 2007 (5)
- September 2007 (3)
- August 2007 (4)
- July 2007 (4)
- June 2007 (7)
- May 2007 (3)
- April 2007 (3)
- March 2007 (8)
- February 2007 (4)
- January 2007 (6)
- December 2006 (5)
- November 2006 (2)
- October 2006 (4)
- September 2006 (4)
- August 2006 (5)
- July 2006 (3)
- June 2006 (3)
- May 2006 (5)
- April 2006 (6)
- March 2006 (7)
- February 2006 (6)
- January 2006 (8)
- December 2005 (7)
- November 2005 (5)
- October 2005 (8)
- September 2005 (19)
- August 2005 (11)
- July 2005 (4)
Other Sites
- DreamHost Status - An off-network page for all DreamHost Announcements! (Not as silly as this blog.)
- DreamHost! - The host behind the blog.
- Forum - Discuss DreamHost and web hosting!
- Web Control Panel - Manage your account.
- Wiki - On DreamHost and web hosting.
Powered by WordPress. Pool theme by Borja Fernandez, modified by DreamHost.
Like WordPress? Consider attending WordCamp LA.
Entries and comments feeds.
^Top^
August 1st, 2006 at 12:41 pm
DreamHost Power Outage…
Downtown Los Angeles was hit by another power outage last Friday. This affected DreamHost’s services and other tenants at the Garland Building (here among MySpace), because the UPS system failed.
DreamHost was affected by a similar incident last …
August 1st, 2006 at 1:00 pm
thank you for explaining what happened over the past few days. these things happen, and you seem to be dealing with it.
some peole might say these things shouldn’t happen, and in a perfect world, they wouldn’t. but this world is far from perfect, and as long as you guys (and gals) are trying, that’s fine by me.
thanks again.
August 1st, 2006 at 1:05 pm
It’s refreshing to see honesty and a company hold their hand up and say they made a mistake. Thank you.
Lets hope your building has the same honesty in admitting their faults.
August 1st, 2006 at 1:09 pm
ditto. thanks for the good post!
August 1st, 2006 at 1:13 pm
Good to see the open and clear explanations. Thanks.
Glad it’s you running all this. ;-)
August 1st, 2006 at 1:14 pm
Josh, will the space with Alchemy allow you to start offering dedicated server again?
August 1st, 2006 at 1:15 pm
Dedicated servers at dreamhost, ya!!!!
From,
Adam
August 1st, 2006 at 1:19 pm
I’m not going anywhere, and this blog post is a big reason why. Thanks for explaining what’s going on and being up front about everything. May your cluetrain never crash.
August 1st, 2006 at 1:33 pm
I’m still here.. :)
August 1st, 2006 at 1:34 pm
I am a sysadmin/networkadmin/etc.. these things happen. Usually they happen all at once making it seem like we are entirely incompetent, but in reality, it’s chaos taking over - strange attractors and all. Point being, for a year I experienced relatively little downtime, and all at once a bunch. In no way does that cancel out the year. I am staying here for now. - Yossie
P.s. And THANKS for the explanation - knowledge is power, and I feel a bit more powerful today :)
August 1st, 2006 at 1:40 pm
I was about to make the switch to DreamHost last September when you had a major power outage. I monitored your status page and blog and decided that I would give you a try anyway. I’m glad I did and so are my clients. Your honesty about what is going on is definitely appreciated by me.
August 1st, 2006 at 1:41 pm
thanks for keeping your customers in the loop. even with all the problems that you all have had, you’re still the best webhost that i’ve worked with.
thanks!!
August 1st, 2006 at 1:41 pm
Use one vendor for all your switches. Any other way lies pain.
August 1st, 2006 at 1:56 pm
Posts like this are the reason that I love dreamhost and continue to pimp you guys out to all my friends.
August 1st, 2006 at 2:02 pm
Great post :D… thanks for the explanation!
We only joined under a week ago, but since then have been astounded by the level of honesty and customer care you guys provide. They ought to learn a few things from you over here in the UK!!
Keep trying to fix the problems, and try not to beat yourselves up too much over this period of bad luck and trial and error. We’re not moving anywhere for the foreseeable future.
Tom :)
August 1st, 2006 at 2:18 pm
Thank you so much for putting all this out in the open! I feel hugely better now.
August 1st, 2006 at 2:28 pm
The right way to do business…
I have been a Dreamhost customer since 1999, and I only recall one significant outage over that entire period. Until the last couple of weeks. Things have definately gotten pretty ugly. DreamHost Blog ｻ Anatomy of a(n ongoing) Disaster.. One……
August 1st, 2006 at 2:34 pm
We use DH for non-critical stuff, like our internal company wiki and email routing. It sounds like a number of problems were brought about by the building’s inability to provide good power.
My company has looked at many colo facilities, and I was very impressed by Equinix. When I took a tour of one of their facilties, I could see many big names (Google, Amazon) located there, and it seems like they know how to run a facility. They have 3 locations in the LA area — I’d recommend looking at moving there.
The fact that MySpace is in your building doesn’t mean much. They probably never expected to grow as much as they did, and are too busy dealing with that to focus on their datacenter situation. They started small and haven’t had time to evaluate if their current facility is appropriate for the level of uptime they require.
August 1st, 2006 at 2:55 pm
Hey, if the worst thing that can happen is Dreamhost owning up to their mistakes and promising to do better, I see no reason at all to even consider switching. The federal government should be so honest!
August 1st, 2006 at 3:03 pm
I’m not a DreamHost customer, but I just wanted to come here and leave a comment to say I admire the way you’re owning up to the problems that have been occurring recently. Laying everything out in the open for everyone to see is the way business should be done!
August 1st, 2006 at 3:47 pm
I just wanted to say how refreshing it is to get such honest information these days.
Thanks and all the best for the future.
August 1st, 2006 at 3:50 pm
I still love you guys.
August 1st, 2006 at 3:51 pm
Ah, good… it sounds like you are almost ready to have a local navvy put a shovel through the fibre outside the building…
August 1st, 2006 at 3:56 pm
I have to say that without this post, I’d be outta here. Thankfully, you guys sent out the newsletter with a pointer to this. Well done!
I subscribe to the emergency status feed and I was beginning to wonder if you had just started reporting things that had not been reported before (but happening), or if you had a new person that got chatty about emergency stuff.
Glad to hear this was an anomaly. I’ll stick with you.
Again, very well done on being so upfront about the issues; but do please try your best to prevent it (I’m sure you will).
August 1st, 2006 at 4:12 pm
Thanks for the updates and thet honest work. It’s appreciated and understood. Thanks again.
August 1st, 2006 at 4:21 pm
Thanks for being so up front with all of us. It’s rare to find a company who is willing to own up to their mistakes and be honest with customers. I know I appreciate it, and I’m sure most of your other customers do, too.
I’ve been recommending Dreamhost to all of my friends for years, and I’m going to continue to do so.
Keep up the great work.
August 1st, 2006 at 4:26 pm
I too appreciate the update.
I’ve been with DH for ~6 years now and have no intentions of leaving, but I would expect in this context to see something of a bone thrown to your customers. We recently experienced some downtime on Puzzle Pirates and ended up crediting subscribers with four days of free play. The downtime was pretty minimal, measured in minutes rather than hours, but intermittant and annoying, so we ‘paid up’ to our players. I would respectfully suggest doing something similar.
August 1st, 2006 at 4:27 pm
Well, despite it all, I still love you guys. And I try to direct as many people as I can to you, too. That way, maybe you can get that fancy data center (or not).
Keep up the good work….
Nancy
August 1st, 2006 at 4:35 pm
Thanks for being up front throughout this ordeal. I’m hoping that August goes better for you guys. Thanks!
August 1st, 2006 at 4:36 pm
Garland Building en Los Angeles
http://flickr.com/photos/alvy/42973478/
August 1st, 2006 at 4:45 pm
Thanks for the explanation. After four years I’m not planning on going anywhere, but you could include some more information on the customer service issues that have come to light in this. All the problems felt were not technical, and the inaccurate and untimely information being put out was at least as frustrating as the outages themselves to me.
August 1st, 2006 at 4:48 pm
You guys rock! I love the catostrophe pictures strewn about. Really conveys the mood :)
I used to work for PowWeb (before it was sold), who was also hosted at the Garland Building, so I hear where you guys are coming from.
This past three weeks has been rough on me with the outages, just like Im sure it has been for many customers, but knowing that you guys definitely dont WANT this to happen, and are constantly trying to make it better goes a long way.
Anyways, brighter days are ahead, and I know I am rooting for you.
August 1st, 2006 at 4:51 pm
DreamHost…
I admire honesty where I see it. Too much of the world is built on lies or at the very least deception. Recently, DreamHost (the company that hosts DigiFiend and a couple of other sites I manage) has been having……
August 1st, 2006 at 5:03 pm
go to hell, dreamhost. ive been with you 7 years and the service has been downhill since i first got on. the cutesy “family and friends” tone isn’t cutting it anymore (unless you’re completely retarded), we’re all pretty aware of the dough being raked in. i can’t believe i’m getting lame copy/paste apology responses instead of credits to my account in some form or another. i pay out my nose for you (in comparison to an equally apt host like 1and1), have referred at least 10 people to you through the years, and keep operating under this illusion that i’m getting the “best” of customer service and features. prove me right, and cut out the cuteness. focus on service and re-imbursing your customers for gigantic boners like this. i don’t care what happened to your facilities, guess what, i have problems in my life, too. do your job as a supplier and amend the situation, and credit us ALL SOMETHING for the bullshit we had to go through for this pathetic display.
August 1st, 2006 at 5:04 pm
Jeesh. Comp time for everyone!
August 1st, 2006 at 5:10 pm
It’s great to see such a detailed account of what happened. That’s why I’m still a customer.
Kind of off-topic, but: Why do internet companies constantly choose to locate in LA, which has chronic power problems in the summer? Why not Dallas, Atlanta, or Richmond? There are tons of other cities with great infrastructure, cheap land, adequate power/no brownouts, and a skilled labor force. But for some reason, LA is chosen despite its lack of adequate power during the summer. I don’t really understand that.
August 1st, 2006 at 5:15 pm
Thank you for the status report, I’m once again pleased I switched to DH earlier this year, excellent customer service and an entertaining newsletter! What more could we want?
August 1st, 2006 at 5:25 pm
Thanks so much for taking the time to give us a status report. I won’t be moving hosting companies anytime soon. You guys really rock in terms of customer support.
August 1st, 2006 at 5:25 pm
I appreciate that a company owns up to its mistakes. Also I have to note that customer support was top-notch during these incidents. While I was a bit pissed off during the occurences, this explanation brings back the warm and fuzzies.
August 1st, 2006 at 5:26 pm
I have been with Dreamhost for about 4 years now and have recomended Dreamhost to around 20 others and I am happy that I have. Has all of this been hard on us? Yes. But LIFE DOES HAPPEN! Over all I have been VERY happy with Dreamhost and have watched them gorw (and learn) and feel like this will only help them come out a little stronger! Thanks for telling us all of this so that we really can understand what happened! That is all that I ask for!
August 1st, 2006 at 5:30 pm
Thank you guys. Your customer service is amazing.
4 years and counting.
August 1st, 2006 at 5:33 pm
I’d say it’s time to hire a dedicated network engineer. You mention “consultant” in the article.
August 1st, 2006 at 5:35 pm
The hosting solution i used before you guys recently had a bunch of problems, some of which stemmed from the terrible flooding in Louisiana. So I could forgive them for the circumstances, but i could not forgive them for the terrible level of communication during and after the initial problems.
Your forthrightness and honesty are absolutely golden, and you can count on me sticking with you guys through the trying times.
keep up the good work!
August 1st, 2006 at 5:37 pm
Thanks for this write-up; we will definitely be staying with DreamHost. This is the kind of status report that keeps customers happy — well done!
August 1st, 2006 at 5:38 pm
[...] How do you treat your customers when you let them down? DreamHost did about as good a job as I can imagine after they were down during a power outage that they couldn’t control. Read the comments where customers are coming back — corporate blogging done right. Thanks to Dylan Bennett for emailing me this. [...]
August 1st, 2006 at 5:39 pm
Without this post I would seriously consider moving. In fact I even hosted one site at Godaddy.com in anticipation of moving, however, I will stay for now. Personally, I think you should at least offer your customers some sort of compensation due to the lack of planning. speaking of which:
I live in Florida, I hosted with DH in CALIFORNIA. Why did I do this? HURRICANES. I know that every year hurricanes are going to take out a large part of Floridas infrastructure. I always laugh my ass off at the idiots driving hummers and mercedes to water and ice lines in the heat while I drive my beat up Saturn around and go home to hot water and a cold refrigerator sitting on my Air-conditioned ass Watching DVD’s on the projector. I bought a generator rather than a hummer. It just made mroe sense. Same thing with a gun. my mom always bugs me about owning a worthless gun I never shoot. But if someone ever attacks my family at home, I can protect my loved ones. And yes I keep it LOADED and COCKED and yes the kids know what it is and where it is and how to handle it. That way when they see one they will know how to handle it rather than blowing thier damn friends head off.
If you prepare for the worst you will be prepared for the best. Perhaps DH should consider hiring some BOY SCOUTS to help prevent problems rather than putting the flames out when it catches fire.
All in all, If i had been at DH when all the catastrophy happened I would be absolutely beside myself. Im sure you gues have been sitting around after a hard day at work and the lights go out all at once and someone said “OH YOU HAVE GOT TO BE FUCKING KIDDING ME!!” I worked for a company once that fired me after I told them that the worst can and would happen. They fired me right after I set up the generator plans and the day was set for it to be installed. The funny thing was that the generator was partially installed and all the parts were not available the day they lost power for a week.
They called me and being the nice guy I suggested they send the guy who replaced me to the Home Depot to buy every generator they had. Idiot forgot the extension cords. Anyways, good luck, and godspeed and please heed some advice. IF IT IS GOING TO HAPPEN TO ANYONE IT WILL HAPPEN TO ME. Say that everyday when you wake up and you will never get surprised.
August 1st, 2006 at 5:42 pm
Great job being so honest guys. I’m really happy that I switched to your service after a horrible experience with C I Host… who still owes me $40.
August 1st, 2006 at 5:43 pm
You guys are honest, transparent, and rock our collective hosting world. I have been in your shoes (in NY, not LA) so I understand exactly what you mean — I found myself nodding through most of your post and I even had a flashback or two at various points. Like most of the customers here, I will use this as only further reason to sing your praises loud and long and to recruit more customers for you.
I’m glad you’re putting in your own UPSes. I second the earlier recommendation for a single vendor for your network fabric (I know, the last thing you want is a backseat sysadmin, but think of this as a friendly comment, not a whine), and on top of that, a consistent OS release on those network devices. (Yes, that’s the scar of an IOS version burn showing through.)
I’ll admit if I was spending $1000/mo on hosting with you I’d be expecting some kind of rebate… but then, it’d be in the SLA I’d signed with you. So if you felt like giving us all a free month I wouldn’t argue, but it sounds like you may have six figures of unexpected hardware expenses about to hit, so I understand if that precludes giving each small customer some kind of token of apology. Hopefully others will understand, too.
Thank you, Josh. Tell the rest of the team thanks, too. Best of luck in the coming months.
August 1st, 2006 at 5:45 pm
I really appreciate you guys! The downtime has not been much of an impact, but what HAS been an impact has been being able to go to “dreamhoststatus.com” and find out IMMEDIATELY what the problem was. My last hosting company would basically lie thru their teeth, and I appreciate that you don’t. The pictures made me die laughing, by the way… :)
August 1st, 2006 at 5:47 pm
I switched my company’s hosting over to DreamHost in March after our previous host had continued downtimes with no explanation or support. While I’ve experienced problems on DreamHost, they’ve always been taking care of quickly, friendly, and with an explanation.
Power outages happen. You don’t own the building, so you can’t ensure the generators and UPSes function correctly. Things happen. I understand.
Thanks for the explanation. Even without it, DH support and the features I am given keeps me here. I’m proud to work with DreamHost, and wholeheartedly recommend them to anyone who needs a host.
/dan mattia
August 1st, 2006 at 5:51 pm
Thank you for your honesty.
This has hurt our website - as has many others no doubt - just as we were finally starting to see some traction.
Your ‘tell it like it is’ approach is why I can tell my visitors we’re sticking with Dreamhost despite 3 weeks of torment.
Please get this fixed quickly. I want to be able to get enough traffic to be able to buy a dedicated server from you guys! :-)
August 1st, 2006 at 6:00 pm
You’d have to go down way more often to get me to go somewhere else. Don’t get me wrong, I have experienced plenty of outages with you, but you always get it fixed asap. Brown outs? As long as you’re looking into the backup equipment failures I’m happy.
August 1st, 2006 at 6:03 pm
Yes, I do appreciate the honesty, but really, being honest can’t cover everything and make all well. I don’t pay my bill to dreamhost every month for honesty, I pay for service. There is a point where you have to look at the history and decide if there are other companies with less down time history (whether or not its under their control).
Aaron
August 1st, 2006 at 6:08 pm
Rad post, Josh. And don’t worry about the power stuff. Your honesty and humor make up for it.
Oh, and regarding that .1% transfer joke in the newsletter… well, I feel like a stud because I used about 26% of my transfer. Some day, I hope to reach 50%. Ha.
August 1st, 2006 at 6:22 pm
Do your contract with your datacenter and your datacenter’s contract with the building specify compensation for power outages? That seems to me to be a good way to make sure the building people have the right incentives to keep your power going rather than making excuses.
August 1st, 2006 at 6:22 pm
While I am sure most of us appreciate the honesty and being kept in the loop, the fact remains that many of the sites being hosted here are suffering from the problems.
I switched several of the sites I maintain over to dreamhost in March and April. The problems have been sporadic. Some of my sites have been affected more than others, it just depends on use.
After this month (yesterday in fact) I have made arrangements to move at least two of my sites to another hosting service. The rest will likely follow, but aren’t as critical for up-time at the moment.
I really liked the setup at dreamhost. I really liked the demeanor of the staff, the newsletters, the status blog, the wiki, etc… but in the end I need hosting to actually be available and not have these kinds of problems.
I get phone calls when sites are down or the database doesn’t connect, or sites are slow. The best I can tell them is I will see what is wrong, but most likely we will have to wait for it to get sorted out. That makes me look bad, my clients look bad, and you look bad.
I can better support clients somewhere that has better up-time with less features so I can spend my time developing and not answering questions about why things don’t work.
August 1st, 2006 at 6:22 pm
Not a big deal. You might want to consider looking at datacenters a bit further away from California’s power grid, though, or this will probably happen again next year at about the same time :)
(The domain transfer I approved today had nothing to do with the downtime; I agreed to transfer it away several months ago)
August 1st, 2006 at 6:27 pm
you folks at DreamHost got on the Cluetrain in such a cool way. Thanks for the honest/open communication with us (and everyone else). It’s this type of thing that will keep me at DreamHost for a long, long time.
August 1st, 2006 at 6:28 pm
To the poster who wants to switch to 1and1… be my guest! More space for me. I started with 1and1 and it was *slow*. Their service was friendly and professional, but never helpful. IMAP servers were like swimming in molasses. DreamHost is for techies.
August 1st, 2006 at 6:37 pm
Thank you for the informative, honest post Josh! It is truly refreshing to have a hosting company that actually cares about it’s customers, and keeps them in the loop.
Posts like this are what set you all apart from other hosts. Many other hosts would have just pretended like the problem didn’t happen, and give vague boiler-plate responses. You take the time and energy to keep customers in the loop, which says a lot about the quality of support that Dreamhost provides.
I am constantly amazed at the knowledge and dedication of the support staff, and every one else who makes Dreamhost tick behind the scenes. Regardless of your shortcomings, which will inevitably be ironed out in due time, there are two things which will keep you guys as the leader of the pack (at least in my book): honesty, and excellent customer support. Just keep these two things paramount and Dreamhost will be the only webhost for me. :)
Thanks guys, and good luck!
Zachery
August 1st, 2006 at 6:40 pm
With the exception of hp, it seems the DH’s customers are real computer users that understand that it isn’t if, but rather when problems will occur. The difference between a good company and a great company is how they handle things when the sh@! hits the fan. DH has proven once again that they are an excellent company through their use of great communication tools to keep customers informed of the situation. The fact that they acknowledge some fault in the latest issue is almost unheard-of in the industry. DH is now the gold standard in my book.
Now if they could just get a clue and get the hell out of Hell (L.A.)! This summer is just the start of things to come with power issues. I hope DH will differentiate themselves in the marketplace and move to middle America where space and power are in abundance. I have been a customer for ~6 years and the only reason I would ever consider leaving is if DH fails to diversify or move operations out of California.
Keep up the good communication and take a trip East!
August 1st, 2006 at 6:41 pm
Thanks for the explanation.
August 1st, 2006 at 6:41 pm
thank you for being honest. This is what we need from a company not a whole it will be back on monday and come next monday it still being down.
August 1st, 2006 at 6:54 pm
Like jz I manage a lot of customers via DH, and it has not been easy getting calls from them these past few weeks. More than the hosting, it’s the email problems (this last time was a Friday and a Monday - ouch!) that have been doing most of the damage.
It doesn’t change the past, but it makes a big difference to get the full story. Thanks for posting.
August 1st, 2006 at 6:55 pm
Thanks for the info. I appreciate knowing the details! I’ve been very happy with Dreamhost.
August 1st, 2006 at 6:57 pm
I am a systems administrator by trade and I understand how things like this can happen and compound on each other. Throughout it all I have never doubted that Dreamhost would get the issues resolved and that you would tell us the truth about what was going on.
You guys rock.
August 1st, 2006 at 6:59 pm
[...] Read about it here. [...]
August 1st, 2006 at 7:00 pm
From very (very) far away I’m still here :)
Good luck (from Brazil),
Cassiano
August 1st, 2006 at 7:08 pm
Problems happen everywhere, with everyone. Unlike many/most others, DH is honest and upfront, and you don’t have to wait weeks for an explanation. Maybe I’m more forgiving than most because I just have a couple vanity websites with no sort of commerce coming for any of them, or because I’m a techie in the daylight hours also, but the good karma earned from the way you guys handle these episodes far outweighs the negative karma caused by the episodes themselves. Plus, when I think things are going crappy at work, I can remind myself of your last 3 weeks and realize they could be a hell of a lot worse!
August 1st, 2006 at 7:13 pm
Thanks for letting us know. I’m still a happy DreamHost customer. I really like a lot of things about DreamHost customer, and I’m sticking around. I enjoy your newsletters, too. :)
August 1st, 2006 at 7:16 pm
Sorry to be contrarian, but while this openness is a good thing, it’s not enough.
When someone tells me their techs are working day and night, it tells me that they didn’t have enough redundancy, pre-configured replacements, and a good emergency plan in place. Their techs are working day and night to compensate for a lack of preparation.
I just host personal sites on Dreamhost. “Mission critical” sites get a dedicated solution in a badass datacenter. But until Dreamhost’s problems, I didn’t realize how much I was relying on my personal e-mail addresses for business-related correspondence.
So, save for some simple filehosting that doesn’t matter much, I’ll be migrating all but a few of my sites off of Dreamhost this month and when my subscription comes up for renewal in 6 months, they’ll all have been migrated.
As in anything, you get what you pay for. If you want reliability, redundancy, and an SLA… it will cost. But when it affects your business, it’s worth it. IMO, all the “we’re really trying” and honesty is nice, but it’s not enough if any of your income depends on any of the services you’re hosting through DH.
August 1st, 2006 at 7:16 pm
Hi Dreamhost, thanks for the transparency. I agree on your business approach with regards to overselling, and the point about not being perfect. I still think that there is a market out there which prefers “Premier” shared hosting, something less than a Virtual Private Server, but yet better than hosting on a server shared by thousands without any redundance. Everyone is following the trend, but there are some businesses which would rather prefer a 99.5% or higher with redundancy. At the very least, the sites would not “disappear” but would just be slow when the traffic is high - a signal to upgrade to VPS or dedicated server. Completely down and fluctuations over 3 weeks or more isn’t acceptable in this business. I would think that it is ok to host low traffic critical sites, but not for business critical sites. Ask anyone running a business and that lost would mean lost of cost as well. I will still (probably) continue to host the much less critical sites here, but more impt ones would have to move out. Slow is ok, but totally down and not accessible isn’t. Apart from the events mentioned in the Status page, there are also other times when we cannot access.
August 1st, 2006 at 7:18 pm
I appreciate your hard work, honesty, and sense of humor. Definitely will stick with you!
thanks
August 1st, 2006 at 7:20 pm
I agree with most people here that your honesty is much appreciated and we’re glad you’re doing the best you can regarding the situation.
However, we have noticed a lot of problems occuring since that September Blackout. We’ve been a dreamhoster for almost five years now and all our clients are hosted here, and sad to say all our major complaints regarding your service/uptime have been for the past year or so.
I think most of us would consider one month refund for compensation for this downtime more than acceptable, and this is something we can pass on to our clients who trust us to recommend the best possible hosting for their website.
August 1st, 2006 at 7:24 pm
I agree 100% with Jonathan Feldman. I currently host a site for Monsanto with you guys and when the outage happened you better believe I got some angry calls and emails. It was so nice to be able to point my customer to your site dreamhoststatus.com that listed the issue and what was being done to solve it. Made my job a lot less easier. I too had considered leaving do to this issue and a previous hardware problem a few months ago, but this post has kept me as a customer (just don’t let it happen again *wink*).
August 1st, 2006 at 7:27 pm
IIRC, UPS manufacturers such as APC recommends not putting a UPS in series with another UPS. Also, my question is why are you in a data center like this? If you are truly commited to providing top quality service, why not colo in a meet-me facility and enter into a peering relationship with some major carriers? It just seems like you have a real business with a bullshit co-lo facility. Have you concidered moving to http://www.laiix.net/?
August 1st, 2006 at 7:34 pm
The honesty is apprecaited, it really is.
August 1st, 2006 at 7:38 pm
As I’m thankful for this note to the DH community, I must add I’m conflicted about the whole serious of events. Having been a DH user since Feb. 1998, I haven’t really observed something this wide-spread, long and exhaustive. Granted, I’m not running sort of business operations on my domains, but, I still like them up, and not slow or completely offline. The last few weeks have been really taxing… having webmail issues, ftp issues, anon ftp access by customers, etc… none of which I really ever had to worry about with DH. And, it sucks. We are paying for a service, and it wasn’t provided, in full, for several weeks. And, then, it’s hard to negate the service they did provide for so long… but, again, I was paying for it. If I didn’t pay for it, I wouldn’t have received the service… why should we pay for a service we didn’t fully receive? Hell, even Comcast and Time Warner Cable will credit you for downtimes you’ve had to deal with… should this be any different?
Yea, this is a little rant-ish, but, it’s more food for thought, poising questions, thinking outloud… I probably will still rave about DH, but, will mention this recent event.
August 1st, 2006 at 7:39 pm
start a campfire and everybody start making out. are you people serious? i’m not sure what sort of work you do, but not being able to send e-mails for about 4 hours caused me all sorts of hell. let’s break this down. 1. i pay for a service. 2. i don’t get the service. 3. i’m told, in so many words, to fuck off and understand that this is a super complicated technical thing that my layman self wouldn’t understand, so to just be content with watching dreamhost shuffle their feet in the dirt and do the “Gosh darn it! i’m so stupid!” routine. 4. watch them get the Presidential Medal of Valor for being so fucking charming. 5. risk this happening again when big bad mother earth causes a server fry, then muddle through dreamhoststatus.com and cute e-mails apologizing to me with NO REIMBURSEMENT FOR BEING A LOYAL CUSTOMER. count me out, shills. one more fuck up like this and i’m out.
August 1st, 2006 at 7:40 pm
Sorry, but we agree with ‘hp’.
While it’s great to see “Honesty”, that doesn’t make up for the thousands of dollars my small business has lost due to DH’s incompetence. This is not an isolated incident. It’s become a pattern. 4 major, multi-day outages in 2 years, with constant small outages (email, web, DB) at any given time during any given month. Completely ridiculous.
Where is the offer of compensation for services not rendered? I’ve yet to see that in the “happy-happy-blog”.
If you’re a teenager with a personal website … by all means, DH is a great place, with reasonable rates. The fact that their service is completely unreliable is probably not an issue.
If you’re a professional, or a business, and expect professional hosting. This isn’t it.
We’ll be moving our business to a professional hosting company ASAP. It is not something we look forward to.
Three years ago, DH was great. It’s not anymore. It’s a sub-par hosting company with apparently incompetant network and systems administrators, with no disaster plans or redundancy. You can get shared hosting from any number of companies at similar rates with redundant data centers, nevermind reliable dedicated hosts which DH no longer can even provide (Not that it would help when their whole infastructure is a disaster).
Our cable ISP (Comcast) we use at our business is more reliable, and that’s saying something.
BR
August 1st, 2006 at 7:56 pm
Thanks for the explanation. I probably sent you guys nasty e-mails due to the downtime… I apologize for that. Kudos to you guys for handling these crises so well!
August 1st, 2006 at 8:01 pm
I launched a new website for a new client on the 2nd July. Can you imagine the crap I have gone through for past month trying to explain why they can’t send email or why their website is down consistantly after launch? They dont care who the host is or what temperature it is in california, their email and website is down and its MY fault, TOTALLY UNACCEPTABLE.
August 1st, 2006 at 8:06 pm
I’ve been a customer for about 10 months. I’ve been involved in hosting my web sites for a VERY long time. Outages and problems happen. I’m staying. The Customer Service and the hosting costs are great.
No company is perfect and there is not a hosting company out there that can guarantee 100% uptime all the time.
Thanks Dreamhost for doing your best and thanks for letting us know what the issues are and that you are working to resolve them.
August 1st, 2006 at 8:11 pm
It sounds there are those that are fine with the apology and openness that you have in your explanation, and those that are not. Honesty is great, and it seems to weigh heavy in certain situations, but there is a service being paid for here that was not delivered.
I am not ready to jump ship, yet.
Reasons, explanations or excuses, I’ll be waiting to see what happens in the next month or two.
August 1st, 2006 at 8:12 pm
Thank you for the explanation, and for all your hard work to fix it. It’s very appreciated :) I have been here over a year, and just renewed. I am glad to see I didn’t make a bad decision.
August 1st, 2006 at 8:14 pm
While I appreciate the explanation, I dont think it clears you guys from responsibility for this major outage. With so many issues compounding one another, I would hope that your staff would be able to handle at least some of the issues quickly. The fact that you had to bring in a cisco consultant says something. And routes being configured by hand, and presumably not being saved, thus having problems after a power outage? That is the most pathetic thing I have ever heard of. I hope that this serves as an indicator for you and leads to some big changes. I am a relatively new customer and am definitely considering other options now. Some big changes, explained in detail and some sort of credit might change my mind, but this is incredibly lame.
August 1st, 2006 at 8:24 pm
Wow… After reading everything that happened, I’m quite impressed. I didn’t even notice there was something wrong during July. I signed up for dreamhost in the middle of the month and started building a webpage + forum. There were down times but those lasted less than 1 minute usually and at the most 10 minutes (that I could tell, but I wasn’t online 24/7)
Maybe it was because the place I was with before sucked a LOT. Because even with your power outages, you had better uptime. And since I don’t run a business and no one in my forum will die if they have to wait a couple minutes to access a page about their favorite celebrity, it’s all gravy =D
Thanks for the heads up and explanation =D Hope all goes well from now!
-Fuji
August 1st, 2006 at 8:24 pm
The honesty is appreciated, and hopefully similar problems can be avoided in the future. Overall I have been pretty satisfied with your response time and communication, and hopefully things will get better.
August 1st, 2006 at 8:25 pm
[...] For the last few weeks, you have no doubt noticed the site problems - mostly really slow loading or, for a while there, no loading at all. I believe the issues are all fixed now, but at any rate if you really want to know what happened (warning: lots of technical speak) with my webhost (and their heroic efforts to get the problems all fixed) read this. [...]
August 1st, 2006 at 8:25 pm
Obviously I picked the best time to join DreamHost, becoming a new member on 7/11/06. I guess I could live with the severe outages and problems I experienced the last three weeks. The little blog this here is okay and helps. But what really steams me about DreamHost are two things:
1) Can’t call a phone number. I’m sorry but this is a *HUGE* issue with me. When a system-wide problem like this occurs, it’s a no-brainer to put a broadcast message up on the menu system. Can’t use the panel when it’s down, can I? I had thought to myself — “oh, there won’t be a need to call in, I only had to call my previous hosting provider a couple times in 2 years” — but not so with DreamHost. Not only that but my responses to support requests placed through tickets is s-l-o-w!!!! Oh yeah, and the first time I requested an urgent callback some yoyo told me I wasn’t entitled to any, despite my plan clearly allowing for three per month! (And if you ask me, ANY new customer should be able to talk with someone on the phone their first month of service! Setup can be a bear!) Overall, I am sorry to say I have to give customer service a failing grade.
2) Your policy of no SpamAssassin when having a catch-all email address. Sorry, this is unacceptable. Your web pages say “catch alls ‘generate’ too much spam” but this is not true… they just all the spam to be delivered. Unless you have SpamAssassin running. My prior hosting provider has no issue with this… why does DreamHost? Someone promised to have someone set up my own copy of SpamAssassin, but they never came through. Following the instructions on my own only resulted in me hosing my own email for hours until I figured out how to fix it. Sorry, but I am not a penguin-guru.
[Why is this so important? Anytime I register with a site, I provide an address such as "dreamhost@mydomain.com" or "microsoft@mydomain.com" -- This is a very efficient way to track where mail is coming from, to "kill" an address that is being used for spam, and to sort my incoming emails into folders. I have over 200 existing mail rules and that is not even covering the ones I haven't bothered to sort.]
There is a third factor, actually, and that is I really miss cPanel. There are a couple things that the proprietary DH panel offers that are better, but these could be integrated with cPanel. Sorry but the DH panel doesn’t cut it for me. It’s slow (changes sometimes take hours to propogate) and visually/user-interface-wise it’s a total mess.
For this reason I am trying to switch hosting providers. But guess what? My domain is locked as “TRANSFERPROHIBITED” and it’s been murder trying to unlock it to complete the transfer so I can close my DH account. Three emails to support so far. And waiting, waiting, waiting for responses. (I did get one that said it was unlocked, but surprise, surprise It’s still not unlocked).
Get your act together, DH!
August 1st, 2006 at 8:25 pm
Hey, keep it up DH, the sun will come out tomorrow. As for the guy that mentioned “quit it with the friends and family tone” a big stfu. Way too many companies take themselves so seriously that I’m starting to think that insertion of a barded dildo into the rectom has become a hiring requirement. Everyone complaining, take a real deep breath. It’s August, all your customers are at the beach, so quit getting red faced and have a Choco-Taco. P.S. DH continually rocks my socks.
August 1st, 2006 at 8:31 pm
There’s lots of good, clean hydro power down here on the South Island of New Zealand… cool too… not much bandwidth though.
Thanks for the explanation and update.
I suggest you put a graph showing the system throughput over time on your status page. Our apps still seem to be grinding to a halt now and then, and it would be good to be able to check whether this was at your end.
August 1st, 2006 at 8:46 pm
How about throwing us a bone and offering a limited ‘777′ deal if we renew for a year?
I of course am appreciative that you took the time to write an explanation of what happened, but I need more than that to keep my service here. If this is even going to be a once a year occurrance, that’s too often. I would expect nothing less than for you to provide some sort of SLA to those who require it. Sure, the mom and pops are going to love your response, but those who are trying to impress clients and build a business are really left holding the bag on this one.
I can’t imagine being in your shoes from a marketing perspective. This fubar is off the charts. I look at the status page and just shake my head and wonder what the hell am I doing hosting my sites with DH. Never thought I’d say that, but this was just too much.
August 1st, 2006 at 8:47 pm
It’s difficult to read the lack of forgiveness in some of these posts. Mistakes happen. Everyone makes them. How can people live with that kind of lack of tolerance? What kind of life is that?
DH gives top notch service at amazingly discounted rates. I can’t find other hosts that would allow me the freedom to manage my server resources like I get here. That kind of flexibility is just unheard of. Any other company would have BSed the causes and wrapped the explanations in red tape. I see that kind of stuff every day and its maddening.
DH, as long as you stay this transparent, I’m not going anywhere. I really appreciate being able to tell my clients exactly what is up. I can’t get this anywhere else…
We’re all in this together and I feel like I get something special from DH. If a few power outages come with that, then so be it.
10000 thanks for all you do everyday!
August 1st, 2006 at 8:55 pm
Your post was not very comforting. I think your company has alot of things assbackwards:
1. anytime it takes that long to explain something: the person doing the explaining is the one at fault;
2. your support people kept telling me that things were fixed when in fact they were not;
3. I got tired of being asked to be more specific, when your own explanations were not any clearer;
4. even when I was told to directly e-mail someone at support, the e-mail bounced back or went directly to general mail: I can get that service from a bank;
5. if you really feel guilty about how you have messed up, then I suggest you start giving us credit on our accounts. Otherwise, your “confession” is worthless to me. I’ve lost so much business over this, that I am leaving!!!
August 1st, 2006 at 8:57 pm
You still tie a hair lip on GoDaddy.com hosting!
August 1st, 2006 at 8:58 pm
Most of the sites I host here are personal, and as big and heavily-trafficked as some may be, they’re still hobby sites, so when they go down, I’m unhappy, but I’ll live.
Clients are another kettle of fish. I’ve had clients hosted here since 2000, and I can firmly say that I’ve had no problems with DH until about a year ago. That being said, I’ve had many problems since then. Usually friendly customer service has given way to curt replies, replies two weeks after the fact, or copy/paste brushoffs. While I’d love to actually talk at length to a tech about the specific requirements of one of my sites, I never get that far and other hosts are starting to look mighty friendly, given all the downtime.
I used to work for an ISP, I know crap happens, it happened all the time to us, nobody has 100% uptime, etc, but that doesn’t change the fact that the service at DH, so excellent in the past, has gone steadily downwards in recent months. The plans are nice, but the repeated hiccups in service aren’t doing us any favors.
I’ve loved my time at Dreamhost, up until the past 12 months or so. I’ve recommended you highly to everyone, and now I’m starting to regret having hosted so many clients here.
Since we lost three weeks of stable up-time, ANY company or utility wouldn’t bill for that time. A free month won’t fix some of the underlying issues I’m experiencing, but it might stop me from marching over to another host tonight and moving some of my more key domains elsewhere.
Your post was honest and I appreciate the explanation, but I’d also appreciate you owning up further by compensating your customers for lost uptime. Repairs or no repairs, please reinforce your wonderful attitude toward your customers and give where it’s due?
August 1st, 2006 at 8:59 pm
“It’s difficult to read the lack of forgiveness in some of these posts. Mistakes happen. Everyone makes them. How can people live with that kind of lack of tolerance? What kind of life is that?”
No offense, but you obviously don’t expect much from a company that you are paying to provide a service.
If this was a one-time event over the history of DH, your point would be valid. It’s not. It’s a recurring theme.
DH has constant multi-hour outages of basic services like email (I think we have something like 20 tickets from the last 6 months about email alone - either the IMAP servers were hosed, or the SMTP servers were pushed bad configurations, nevermind all the times they get black-listed), to major outages of web servers that last all day.
Not *once* have they offered to compensate us for their lack of service. Over 3 years. It wears thin, and in our case, we gave them lots of rope to hang themselves with.
Again, like ‘hp’, I really have no idea what planet you folks are from. When your sites go down on a regular basis, and you can’t send email for hours at a time … you think this is “normal”? And that “Mistakes happen”? It’s incompetence. I don’t know if they lost all the folks that actually knew what they were doing or what, but they have a serious problem.
A high school kid with an SDSL line and a linux box could provide more reliable service.
- BR
August 1st, 2006 at 9:04 pm
[...] Things have improved over the last week, but I was left wondering if my experience was typical of how things are normally with Dreamhost. However yesterday this post showed up in my feeds, and in an email newsletter from Dreamhost. [...]
August 1st, 2006 at 9:04 pm
If the cable is out for 4 days, I get a 3 day credit on my bill.
If the phone is out for 4 days, I get a 3 day credit on my bill.
Hmmm.
August 1st, 2006 at 9:08 pm
This mess has really made me question Dreamhost, and I’ve been here for a very long time. I’d always interpreted the regular mail screw-ups, the bizarre little problems, as the result of rapid growth and maybe not making the conversion from ‘happy fun hobby business’ to ‘real business’ as smoothly as possible. But I assumed that the big picture stuff, like power, and routing tables, and all that was being taken care of, and my little problems with weird quirks of the email control panel were just growing pains, or the founders/bosses playing games with things they didn’t understand anymore (which one time an overly honest, and probably now unemployed, tech told me was the actual reason for an email outage).
Now I’m convinced that 80% of the staff spends all day just jacking around with minor crap like the control panel, and 10% of the their time praying nothing goes wrong in the big bad data center they don’t really understand, and 10% of the time looking for pretty pictures of disasters on the web instead of using those extra minutes to look up redundancy in the dictionary.
August 1st, 2006 at 9:16 pm
you guys better get your shit together!
everyone i know who uses dreamhost is very close to dumping your “service” and moving on
August 1st, 2006 at 9:18 pm
For those of you losing thousands of dollars over 24 hours, may I remind you that we pay something like 8 dollar per month. I’ve already shared with my clients where they’re hosted and what other hosting options are. None of us are really sweating this one. Sure it’s a bummer to be down but if I was gonna lose thousands per day I’d switch to redundant co-lo.
August 1st, 2006 at 9:18 pm
Outages can be a real disaster if you’re in the middle of a book launch, and there was one a couple of months ago which really hit us in the gut. However, I agree that transparency is the best policy, and DreamHost gets points for that. I’m really glad you linked all the issues together, because I think DreamHost would have looked like chaos if you kept pushing “isolated events”.
Plus the pictures made me laugh! Thanks for those. And double thanks for not taking out the gross food pictures again. ;-)
August 1st, 2006 at 9:20 pm
what happened to the DREAM in dreamhost?
its been a nightmare hosting with you guys!
i’m not going to renew my plan
August 1st, 2006 at 9:23 pm
I can’t say that I’m happy with the downtime, but I do appreciate the transparency. I use quite a few hosting companies and I can honestly say that none of them would have provided this sort of detailed account of what happened over the past month.
One other point I wanted to add - while I generally don’t believe in astrology, I’ve learned to lay low when Mercury is retrograde - which it was from 7/4 - 7/28. July 2006 at Dreamhost was classic Mercury retrograde.
August 1st, 2006 at 9:26 pm
[...] DreamHost, the hosting provider for JaredWSmith.com and about 95% of the other Web things I operate, has had a rough July thanks to the brownouts and such in Los Angeles. Their extremely detailed blog entry shows that they are sincerely working hard to try to mitigate any future disasters, and gives some insight into just how tough it is to run a massive hosting operation such as Dreamhost. It’s a great read, highly recommended. Technorati Tags: dreamhost, power outage [...]
August 1st, 2006 at 9:36 pm
[...] DreamHost Blog » Anatomy of a(n ongoing) Disaster. dreamhost. del.icio.us this! [...]
August 1st, 2006 at 9:39 pm
I am hosting with dreamhost despite living in Kenya, East Africa. DH experiences sound just like what happens to us in East Africa all the time, so I sympathise.
Good luck, keep at it, and thanks for the support.
August 1st, 2006 at 9:48 pm
# Daniel James Says:
I’ve been with DH for ~6 years now and have no intentions of leaving, but I would expect in this context to see something of a bone thrown to your customers. We recently experienced some downtime on Puzzle Pirates and ended up crediting subscribers with four days of free play.
Oddly, as I was reading this, I was thinking exactly the same thing - down to ‘With the recent issues on Puzzle Pirates, they…’
In the past, dreamhost has been excellent about renumeration for issues. I don’t feel ‘entitled’ to anything in particular, but I’ve spent a lot of time recently having to post to my blog (hosted elsewhere), ‘ok, guys, I can’t send mail again, so Rachel, if you’re reading this….’ and I admit starting to wonder if it was time to leave DH and head for some young, hungry company… like the one I signed up with several years ago.
-Stephanie.
August 1st, 2006 at 9:51 pm
While I don’t like downtime I prefer an honest “I screwed up!” anyday. Also, it is little memos like this that give DreamHost character–that is why I stick around.
August 1st, 2006 at 9:55 pm
Dreamhost - Thank you so much for the detailed explaination. I have over 60 of my clients hosted with you guys so I have been feeling the heat (no pun intended) from them as well.
Overall, Dreamhost has been an incredible business partner for me and I continue to partner exclusively with them.
August 1st, 2006 at 9:56 pm
It is very interesting to see all of these comments. The range is quite astounding. I am a recent new customer, so I haven’t had experience with previous outages, and therefore I can’t comment on the reliability.
My opinion is that the nice friendly explanation is nice, refreshing even. The lack of phone support does bite, but then that is why your clients hire you because you are supposed to have a clue, you are support. The prices for dreamhost are good, great even, what is the uptime average over the past 5 years?
A rough spot? possibly. A tech who is learning? possibly. Terminate the wrong guy? possibly. Downtime is certainly part of the datacenter’s contract, and I am sure that with patience DH will make the issues right, and we will all get properly compensated for the event.
The hardware replacement really bites, I wonder if the new switch wasn’t listening to STP priorities, or if they just weren’t set, or if it just started having it’s own party. That type of network problem is pernicious, difficult to track, and generally a pain in the you know what (especially in multi vendor environments, there are some things that syslog and snmp just can’t tell you)
All in all I am pretty happy.
And on a side note: my cable modem went down more often this past month, and the cable company knows they’re the only thing going in the neighborhood… so no credits here. Regardless of what it would do for customer loyalty and that general “hey, you got any vaseline?” felling I get when I think about my utility providers.
If your clients are making such big money off their site, they should put in their own pipe, run their own servers, and see how much fun it can be. Can’t get email? well, if you ran another MX at your location (and you probably should…), you wouldn’t have to worry about that. Drop a lower priority on it and you only get hit when it goes down.
You criticise the disaster plan? where is yours? what are you gonna do when the unlikely happens to you? oh, wait. it just did. Your plan was to sit back and whine that your clients were chewing you out. Great plan. I hope you pay your clients for that.
My mail servers (and my client’s) fall back to me, it is slower, but it’s still up. business gets done.
DH, I will stay. It’s not five nines, but I am betting that it is still in the mid 90’s.
Good luck. Hang tough. The admin is never appreciated.
August 1st, 2006 at 9:57 pm
As a systems admin, I understand the problems that the not quite so happy Dreamhost crew has been going through.
I also remember the OMG FUN times I had when I hosted all my sites from a box at my house. Script kiddies from Brazil, getting blacklisted because some spammer fell in love with my domain names, etc.
So, do the you will be happy again Dreamhost crew, thank you for being honest and letting us know what’s up. I truly appreciate your honesty.
And to those who posted negatively as “Anonymous”… move on if you must, but know you’ll most likely not be missed.
August 1st, 2006 at 10:01 pm
Thank you thank you, thank you!
I have been searching for another hosting company for the last couple of weeks. I love dreamhost. You guys have been great for a long time and here recently it’s been crap. I get hard times but I just wanted to know what was going on. Now that I know then I have hope things will get better. I’ll stick it out a few more months and hopefully it will get better.
Thanks again!
August 1st, 2006 at 10:08 pm
[...] As noted over in Half Baked, Dreamhost is starting to talk about the horrible things that have happened to it in the past month. That makes the issues a little less unsettling, just because someone, at least, is admitting to some sort of incompetence mixed with bad luck. Yay them. [...]
August 1st, 2006 at 10:13 pm
First of all, thanks, DH, for the openness and honesty. I’ve been bullshitted constantly by numerous tech companies (most notably, my local ISP, Alltel, who once tried to convince me that the entire Internet slows to 250kbps every evening at 10pm); it’s refreshing to hear someone say “This is what happened. It was [at least partially] our fault.”
I understand completely why businesses who rely on their web presence and e-mail would be angry about last month’s events. I also understand the demands for reimbursement; if you lose money due to someone else’s mistake, that’s a reasonable request. It’s also easy to understand why some DH customers would feel justified in taking their business elsewhere.
With that said, however, some of you guys need to cut the attitude. Shit happens. The world’s an imperfect place. Everyone on this board has screwed up at one point or another. The major difference is that *our* screwups aren’t visible by 300,000+ customers and even more web users. So cut them some slack; when you’re perfect, you can start giving them crap. Until then, get over yourselves.
Also, the fact that DH called a Cisco consultant *does* say something: it says that they’re smart enough to know when they need help. Tell me, folks… Who would you rather have working on the Cisco router hooked up to the server your site is located on: A DH engineer, or an engineer who works for and was trained by the company that ACTUALLY MADE THE ROUTER? I don’t expect DH to know everything about Cisco routers. There are Cisco engineers who are there for that.
Just my 2 cents. OK, maybe more like $2. Either way, I’m sticking with you guys. After all, things can only go up from here…
August 1st, 2006 at 10:33 pm
While I have to agree with the vast majority that Dreamhost’s honesty and integrity are not in question, the performance of the network is definitely something to be concerned about.
I wouldn’t consider moving at this point, but until we get several consecutive months of solid (and I don’t mean perfect) performance out of the network, I can’t in all good concience recommend the service to anyone else.
August 1st, 2006 at 10:44 pm
[...] Dreamhost Blog: Anatomy of a(n ongoing) disaster [...]
August 1st, 2006 at 10:47 pm
It’s a semi-decent explanation of things but I’m afraid it just doesn’t cut it with me. You’ve explained a few of the problems and it’s interesting to see your note end on a kinda ‘yeah well we’ve fixed a lot of problems but there’s still a few gremlins in the system that we’re trying to fix. we promise it won’t happen again… bla bla… please stay’… yawn.
You’re trying to put a lot of the blame on the infrastructure and the building. Let’s see. Media Temple are hosted there. I know a lot of their customers and they have had a near-faultless system for years, especially over the last few months. MySpace was only offline for an afternoon and service has resumed with no snags. Why on earth is it hitting Dreamhost so hard? Yes you’ve explained it yourselves. It’s part building part your own incompetence but a line really has to be drawn.
I’ve been with you guys for over 7 years and the last 3-4 months have been a JOKE! E-mail up and down like a yoyo, just like my sites. Plus (and here’s the biggie). I’ve spent the last 2 years telling everyone how great you are and have recommended clients and friends to move. Lately I’ve looked like a fool with phonecall after phonecall asking what’s going on with my website… what’s going on with my mail!
Enough!
I’m off elsewhere. To Texas actually, where the power grid seems to be a lot more stable. I’m not going to mention what host I’m off to as I don’t want this rant to turn into an advert. It’s been a great six and a half years but the last few months have been rediculous and people who buy into this ‘ohh we’re sorry’ explanation need a slap, especially those of us who have recommended you in the past and are looking foolish at this time. NOT a nice position to be in I’ll tell you!
August 1st, 2006 at 10:50 pm
Thanks for the communication. The reason we left our last service provider is because they had a rash of bad problems but their communication skills were ABYSMAL. We had no idea what was happening and any attempts to get a status were met with increasing hostility as the days wore on and the stress levels rose. So thanks for this and please keep us posted. That’s what separates you guys from everybody else.
August 1st, 2006 at 11:00 pm
I am in agreement with most of what has been posted here… the blog and friendly customer service is outstanding… so are your services and the level of control you offer your customers. That said, the level of uptime is not acceptable.
I have no intentions of leaving DreamHost, but I will definitely be evaluating the hosting over the next six months… non-mission critical sites will stay regardless but I may have to move e-commerce sites elsewhere if I have to go through another episode of this magnitude. I would like to recommend that you dedicate some of your efforts towards the infrastructure side of your operations (UPS, Generators…etc.) and perhaps focus less on Goodies and one-click installs and a little more on server backups and redundant architecture.
I used to host mostly with Interland (now Web.com). They are extremely overpriced, customers have hardly any control over their sites and their tech support is in India… however, I don’t recall ever having an crash and burn episode like this one. Still, I am very happy with DreamHost and hope you can get your act together in a reasonable time.
August 1st, 2006 at 11:22 pm
Where did the wild dogs come in?
August 1st, 2006 at 11:31 pm
Dreamhost, thank you for being honest, which is rather refreshing. It does remind me of a quote from Yes Prime Minister, talking about being an executive in the City of London:
“The basic rule of the City was that if you are incompetent you have to be honest, and if you are crooked you have to be clever. The reasoning is that, if you are honest, the chaps will rally round and help you if you make a pig’s breakfast out of your business dealings. Conversely, if you are crooked, no one will ask questions so long as you are making substantial profits. The ideal City firm was both honest and clever, although these were in short supply.” (Yes Prime Minister II, p. 109)
You seem to have made your mind up about the data centre - I would have thought instead of buying your own UPS it would be cheaper to move elsewhere (I’m sure we could live with the downtime!) but there’s probably something you know that we don’t :)
August 1st, 2006 at 11:31 pm
I joined Dreamhost at the start of July.
Hmmm - I’ve had a really bad string of luck in my life recently - maybe it’s *me*?!?!? ;)
I will be staying because no host is perfect but you guys are at least human and are trying your best.
And you’re funny. :)
August 1st, 2006 at 11:44 pm
[...] 如果一家公司给它的客户惹了不少麻烦，事后还能获得“谀”词如潮，想知道它是怎么做的吗？这家公司就是我这个网站的服务商Dreamhost。7月份是Dreamhost的灾难月，数据中心断电、UPS不工作、文件服务器不正常、路由器损坏……一连串的事件接连发生，让Dreamhost应付不暇。但是发生在这一切的同时，Dreamhost通过网站向用户通报事情进展，而在风平浪静之后又发了一篇详细的日志，坦诚的交代事情的前后经过。 [...]
August 1st, 2006 at 11:47 pm
Thanks for the awesome post!
August 1st, 2006 at 11:48 pm
The problem with moving elsewhere is that…
…unless I go for a dedicated server or a VPS, there’s nowhere that offers the same features. Even on a shared hosting plan twice the price.
August 1st, 2006 at 11:50 pm
It is looking better today. Hope you can fix all problems and I would suggest you invest a good amount of time to find a much more reliable data center. As a customer I would be happy to spend a bit more for better stability.
The company I am working for is switching colocotation for the 3rd time in 4 years and I am very familiar with many of your problems.
August 1st, 2006 at 11:53 pm
Thanks for the update Dreamhost, your honesty is appreciated.
August 2nd, 2006 at 12:01 am
Guys, keep up the good work. Honesty is the best policy. And I still wuv u!!! Better luck in the future!
August 2nd, 2006 at 12:06 am
You know, you guys are the unluckiest guys in the world of hosting… I hear of a few people slagging you off and saying that I’m leaving - those people really get on my nerves… you are the only “human” hosting company I know of, having such an informative company letting us know about every single in and out is such a nice feeling.
I put my trust in you with all of my sites and I will continue to do so, if something goes wrong I know you will be doing your best to fix it straight away, my complete confidence is in you guys.
Thank you for the great service and many laughs you have given to me…
August 2nd, 2006 at 12:11 am
[...] In recent weeks Dreamhost has been having some problems. Actually that’s an understatement. They’ve been having big problems. Not as bad as the problems my last hosting company had which drove me to leave (at least Dreamhost always have backups to restore), but enough to make me question hosting commercial sites with them. [...]
August 2nd, 2006 at 12:18 am
(Okay this turned out to rather long… but what the heck are you guys doing reading all the way down here at comment one hundred and something anyway?)
This is my second year with DreamHost - I just renewed. Right after I joined last time you guys had the LA power problem and I lost a couple of domains for a day or so. Inspite of that I still went ahead and started migrating all my multitude of domains and sites I hosted on my home server - I was able to do that because you guys added unlimited domains just after I joined up long before anyone else did, lets just say I was very happy. I was also able to stop worrying about a Slashdot effect taking out my home DSL line, or even running up an excess bandwidth bill because you guys effectively made that unlimited in my first year. You also added DNS access so I could stop trying to maintain separate DNS services elsewhere and save even more money.
This time around I have all my domains with you, twelve in one account alone (and another three in a separate account I manage for a non-profit). I’ve had a few comments from users of those domains, but they all have enough experience of computer systems to know that every IT department with a flawless uptime record is just one unexpected disaster or screwup from a reputation like s**t. In fact the longer an organization goes without some major problem to deal with the more complacent it gets. Everyone leaving DreamHost now will certainly rant and rave about how good their new place is right up until the time it fails them, and it will.
Right from the start I recognized that DreamHost services are homegrown, kind of folksy, and just plain different. But everyone, you know why you are here in the first place - its a deal, its a steal is sale of the ****king century - so what did you expect?
At the end of my second year I wont have paid more than $20 for my two years of hosting of all my domains and I’ll still have money left over in my rewards. And I’ll bet many of those few (I count less than 10% - hardly “everyone”) unhappy folks rubbed their hands together with glee over how much money they were going to save, or make reselling DreamHost services.
Well guys, you get what you pay for (remember what your el-cheapo plans are called - CRAZY DOMAIN INSANE - get it?) and anything more is gravy and to me DreamHost has delivered tons of gravy for almost nothing in the last year and they have removed from me the burden of self hosting and fixing all my OWN screwups at the most inconvenient times. What really did it was not even been able to go on vacation for worrying a disk would crash, network equipment fail or just something… Those things still go on but I’m more than happy to pay next to nothing every year for that to be SOMEONE ELSES problem no matter how long it takes.
Finally, the apology and complete honesty that screw-ups were made goes a long way with me. I just can’t stand a company or person that wont own up to their own mistakes and for every company that never issues such a thing but experiences even 0.001% downtime there’s a bunch of screw-ups and failures they never confess to, and because they never do you’ll never know if you’re just a gnats whisker away from the scale of problems DreamHost just had. Sure they may be of different nature, but they could have as big a consequence. And because DreamHost is very honest about its screwups all the information is right out there for buyers to make their decision to become customers - and yet they still do.
I don’t expect DreamHost confessionals to make my hosting problems go away - I never did, and never wil - but I admire them for having the balls to put it out there and realize they will be judged by how much they measure up to it in the future. A person or company that never does this is and just denies everything or pays off customers “out of court” so to speak to keep its reputation “clean” is just getting a free ride on our inherent greed and need to get “a good deal”.
My advice to DreamHost is - if you don’t want to keep being judged by the crappiness of your buildings power system, then do something about it. Its good enough for me (clearly better than what I had before) - but obviously not others. I’m glad to hear you are working on that and I’d love to hear more honest reports about progress in that direction and maybe even a timeline. I don’t care too much myself but I think it would make a lot of others happier and show good will on your part to follow through after today’s “confessional”.
August 2nd, 2006 at 12:31 am
[...] I’ve had more bad luck than you can shake a stick at with web hosts, lately. Read about what’s been happening to DreamHost, which hosts this self-same blog. [...]
August 2nd, 2006 at 1:07 am
Thanks for the info guys. I’m sitting here on the other side of the world and I still feel like I’m in the loop. Dreamhost’s service is the best I’ve found. My commiserations for all the bummers.
Peter.
August 2nd, 2006 at 1:38 am
Thanks for the info, I think this is your “open to customer” behaviour that makes you the best.
August 2nd, 2006 at 1:58 am
[...] Dreamhost has posted an extensive information update concerning the many difficulties they’ve been having in recent weeks. It’s fascinating behind-the-scenes detailed look at the issues they’ve had with everything from UPS failures, brownouts and server failures. Related Posts [...]
August 2nd, 2006 at 2:29 am
This is quite interesting, I just setup an account and started to move my site from Powweb to your servers last week & this happened.
I’m sympathetic to your plight, but I’m afraid I’ll have to reconsider the move to your service. Outages, mail bouncing.
The reason I was leaving Powweb was the instability after they were purchased and migrated to the east coast.
The only thing I ask of a web hosting firm is stability in hosting my sites. I have 4 of my own and I manage a significant number of other sites. If you cannot provide me with a stable platform for my test site…..
August 2nd, 2006 at 2:36 am
One of my dreamhost domains hasn’t been too bad… 5 minutes downtime a day on average from a UK address. Its not critical, and it uses so much bandwidth that it would cost a lot to host on a reliable site.
Another domain on dreamhost is a bit more alarming, 3 minutes downtime 20 times a day from a UK address. But its also not critical.
It costs peanuts to host here compared with the super reliable hosts like PAIR. But if downtime loses you a lot of money, thats WHY you need to pay more.
I’m glad I get 20 GB of disk space here, because I got r*ped for $450 this month on disk space overage at PAIR.
August 2nd, 2006 at 3:06 am
Thanks for your honesty and the information you provide. I hosted my domains at several other providers before I moved to DH, some were better (but a lot more expensive) but none had such a good customer information system like you have.
August 2nd, 2006 at 3:30 am
[...] Zapraszam do lektury anatomii (trwającej) katastrofy. [...]
August 2nd, 2006 at 3:35 am
I need to know about the wild dogs.
August 2nd, 2006 at 4:05 am
Since California hasn’t been having good luck with energy in general for the past … when was it Enron first started mucking around… several years… do it not make sense to have a datacenter somewhere in the middle of nowhere USA where there are not a million air conditioners running 24/7?
August 2nd, 2006 at 4:13 am
Next time is it possible to have all down URL’s post a page that says “Experiencing technical difficulties” or something?
August 2nd, 2006 at 4:14 am
I appreciate the honesty and transparency of DreamHost. Anyways, it’s cool, I only noticed the downtime once.
August 2nd, 2006 at 4:24 am
It has been really satisfactory to read this from you.
It’s good to know exactly what happened.
I hope it all get better now.
August 2nd, 2006 at 4:29 am
I’ll second the suggestion to bail out of LA. I don’t understand the attraction. It just doesn’t make sense to stay there when there are perfectly ample infrastructures throughout the United States.
Move to Kansas.
August 2nd, 2006 at 4:29 am
[...] A few weeks ago, I had some problems with this site going down, as my host Dreamhost has been having some issues. I’ve even reccomended Dreamhost to Clint Ivy, who I believe is using it as well. The Dreamhost blog explains all in a very lengthy post –read for yourself. “As I’m sure most of you already know, we’ve had nothing but troubles, large troubles, for pretty much the last three weeks. A lot of these troubles were our fault, a couple of them were at least ostensibly beyond our control, and they all compounded each other. [...]
August 2nd, 2006 at 4:34 am
Outages? What outages?
I hit Digg frontpage, and you guys kept me online all the time. Despite the masses of traffic.
Thanks again :)
August 2nd, 2006 at 4:36 am
Dito to a lot of whats been said. I appreciate the honesty and being upfront.
August 2nd, 2006 at 4:40 am
Greetings from the sweltering/muggy/sticky east coast,
Thanks for the highly informative blog post! You guys are my first ‘real’ hosting company and I couldn’t be happier with the service.
Stay cool.
—
A hollow voice says, “Fool.”
August 2nd, 2006 at 4:40 am
Perhaps is Josh spent more time leading this company away from disaster and less time on the stupid newsletter’s rifed with sophmoric humor, then maybe this company has a chance to survive. Maybe now is the time to sell out.
August 2nd, 2006 at 4:47 am
Great to see this transparancy, honesty and authenticity! I blogged about it.
August 2nd, 2006 at 5:04 am
I am proud to be a customer of a company that is so transparent that even details shared here can help some companies to compete in its business, if they want to just use the information to break your customers or even reverse engineer based on details shared…
Apart from the details, the page layout is one of the coolest one I have ever come across. Keep Walking Guys, you rock!!
August 2nd, 2006 at 5:22 am
Dreamhost is one notch better than my old hosting company but then again, they got indicted for fraud (google “featureprice” for your worst nightmare). When I called them to cancel my account, they took my website down and replaced it with a new homepage that read “I am a Supporter of Osama Bin Lauden. Please feel free to charge whatever you like to my credit card, followed by my actual American Express credit card number! I truly wish I were joking but this is one of many horror stories with this particular hosting company (I think the previous owners have fled the country).
Okay, so Dreamhost is absolutely incredible compared to them BUT I still have some issues. I think as big as they’ve become, they need to hire more competent help. Yes, they admit when they have major downtimes but the fact is there seem to be constant issues with system stability at DH. If it’s not my client’s websites going down it’s webmail, the control panel, constant slow response times or some other glitch. I’ve been with Dreamhost now for several years and these problems never seem to go away, except now we have major outages to deal with. I can’t take many more hits like this with my business clients. I am getting creamed.
Another problem is trying to get a straight, timely answer from DH. Yes, they have an emergency announcement page but during the last outage, they didn’t keep it updated on a timely basis - which is really necessary on their part since they have such sloooow response to email or phone support. I don’t like getting a callback a day after I’ve put in a request, for example. This is way too late. I think they should offer an add-on charge if necessary for those who’d like a number to call for immediate assistance. The fact they can’t offer this service even for a fee speaks volumes about how short-staffed and under-trained they are and how many problems they’re constantly running around trying to solve. They need to commit themselves to investing more money to hire more competent help to come in and solve some seemingly fundamental technical issues they’ve had for a very long time. Unless they do, they’ll continue to keep running around putting a bandaid on problems that require major revision. Just my two cents worth.
August 2nd, 2006 at 5:24 am
As the french say: merde happens. Considering the magnatude of the issues you guys have been dealing with and the fact that many of the problems start outside the building, I think that uptimes and response times have been quite acceptible. Throughout this ordeal, I have marveled at the immediate notices in my RSS feed and the quickly-following notices of resolution.
You don’t get that from the telcos.
I have…what…3…4 sites on DH with more to come(hopefully). I, too, recommend DH to everyone, especially those just starting out with a website. Best damn deal around, on balance.
Because I am a little older(53) than most here and because I spent my life as a tech in the music industry, I have grown to know that Shit Happens. I have also grown to accept that Shit Happens. Things break. Sometimes things break in a chain, one after another. I also know that bad power breaks things(oh, the memories of a gig in an old quonset hut in the wilds of NC still make me wake up screaming at night…). One last thing I do know is that DH does not treat its customers like mushrooms.
At my age, I have learned that such commitment to openness, without obscuring and obfuscating marketroid verbiage is both rare and wonderful.
Nahhhhh…I ain’t going nowhere.
One more thing, a meta-thought if you will: I think that you guys who are moving to other parts of the US better take a look at the news. These same power grid issues are national. Many years of lack of capital investment in infrastructure, by the single most cashed-up industrial and corporate sector we have, is starting to take its toll nationally. LA is just the canary in the coal mine.
August 2nd, 2006 at 5:28 am
Thank you for the explanation! Things happens to all companies. So far, you are the best at keeping things working, and letting us know what’s going on when you can’t… Keep up the good work…
August 2nd, 2006 at 5:30 am
Great post, guys ! I’m not even a customer (yet) … just waiting on my iPowerWeb account to expire. I’m a datacenter admin for state gov’t and have about ~150 servers in there. I can feel some of your pain. But the important thing is you explained it all, the good and bad, which is something iPowerWeb would Never do. I doubt they would even admit to anything being their fault - it never is !
Keep it up and I look forward to being a customer soon.
~ Jerry
August 2nd, 2006 at 5:40 am
Thank you for this blog. I found it very informative and interesting. I’m a new customer too. Like Allan above, I moved from PowWeb because their migration took down my site. Their backup worked but I wasn’t able to upload anything new. It just kept reverting to the old site. I got frustrated and moved here. My account there hasn’t expired yet and I *still* can’t upload anything new. I tried because I thought about moving back there again after these recent problems DH had. This blog has made me change my mind and I’ll stick here for a while. Thanks for being so upfront. Great pictures in the blog too btw!
My site isn’t critical though, and I have mirrors elsewhere. It’s a personal podcast site and no businesses are involved (I don’t even take banner or Google ads), so I can understand those who have lost money and clients being upset. Still, for me, it’s working ok now, and I hope to be here for a while, though I’ll keep my mirrors since you’re in earthquake country.
August 2nd, 2006 at 5:41 am
This post comes at a really good time. Given the low traffic at my sites, as well as some additional work I want to do, I was seriously considering moving to a Xen-based hosting service. I had identified some that were cheaper than my current service, and was starting to make plans for the move. This was before your recent troubles.
However, based on your communication and this honest explination I’ve decided that it is well worth it to stay with dreamhost. This whole incident reminded me of how I ended up here. My previous host was down for over a week , provided no communication during that time, and his reaction when everything came back up was to attack the users for wanting information about what had happened. Even got outright hostile when asked what he’d be doing to prevent such an outage in the future. During that outage, I was looking for a new job and missed several very important emails!
At anyrate, while the disruption was annoying, I’m sticking around because you are doing the right thing. Fully disclosing the issues and telling us what you are doing to deal with them.
Thinking back to that event, I realized that something like that wouldn’t happen here. So, I’m sticking around. (However, let us all know if you are going to be offering xen-based hosting anytime soon….)
August 2nd, 2006 at 5:48 am
I just registered with DreamHost. As long as you guys are up for at least the next two years without any major dramas, and you don’t fold, … I’ll be happy :)
August 2nd, 2006 at 5:51 am
[...] Anatomy of an On-Going Disaster (Refreshingly frank post from Dreamhost; nicely illustrated, too [...]
August 2nd, 2006 at 6:19 am
why are you people thanking dreamhost? does it make you feel warm and fuzzy that they’re “transparent” or something?
you pay these people and they give you crap service and then you thank them for it! what is that?
August 2nd, 2006 at 6:26 am
I think all the people who are all pissy because they “lost thousands” because of the outage are pretty funny. If you “lost thousands of dollars” because of the disruptions in a two-week period, how about the thousands you’ll be making back in the next two-week period. Or the one after tbat. Cut your losses and give them a fucking break. DH cannot predict when these disasters happen, and now expect to take some losses from assholes like yourselves who won’t renew their services, and I’m sure are compensating (internally) for it. It sucks, a lot. but they’re being honest about it, and moving on. You should too.
Keep up the excellent work guys. I’ve only been a customer since April, but so far, with the service that I get for dollar-I-pay per month, it’ll take a lot more than some power disruptions (however irritating) to get me to switch to anyone else.
August 2nd, 2006 at 6:30 am
What a nightmare…
I’ve had whole weeks like that.
One thing after another, the problems layered into each other so deeply you can’t tell what is causing which, and everything you do to fix one exposes another. It’s a terrible feeling. I don’t use this outfit, h…
August 2nd, 2006 at 6:32 am
Thanks for the update. much appreciated
August 2nd, 2006 at 6:41 am
[...] DreamHost Blog » Anatomy of a(n ongoing) Disaster..: “As I’m sure most of you already know, we’ve had nothing but troubles, large troubles, for pretty much the last three weeks. A lot of these troubles were our fault, a couple of them were at least ostensibly beyond our control, and they all compounded each other. [...]
August 2nd, 2006 at 6:45 am
I’ve been with dreamhost for over a year now and I am satisfied with the service. Nobody is perfect and mistakes do happy. I’m not the least bit worried.
August 2nd, 2006 at 6:48 am
[...] 上面的事件Dreamhost 都一五一十,老老實實的在自己的官方blog公佈. [...]
August 2nd, 2006 at 6:53 am
Thanks for the update. Having been through a couple of these “events” myself (and sporting prematurely grey hair as a result), I can certainly “feel your pain”.
Keep up the good planning and let’s hope the weather cooperates and cools things off so that the power drain is not so great. My your luck soon change and may the force be with you!
Just out of curiosity, and since you ar located in L.A., what disaster recovery plans do you have in place to restore service in a timely manner in the event (God forbid!) the “Big One” hits?
August 2nd, 2006 at 6:56 am
Thanks for being honest, I feel confident you are taking things as seriously as you have always had. I’m a happy customer despite these past three weeks.
August 2nd, 2006 at 6:57 am
I’ve only been with dreamhost for a couple weeks, but the downtime for me has been fairly minimal, and the transparency of the dreamhost operations is fantastic.
At some of my other hosts that may have provided more reliable service more frequently, when there would be any issue, or my sites would start going really slowly, I would have NO idea what the problem was. I would email support or call them and they would say “this is a known problem and we are working on it.” At dreamhost, so far, it seems that they are more explicit about what the problem is.
For example, my pending domain additions were really slow, so I checked the status page and say that there was a problem with the machine that deals with this, and that it would take a few hours before it was fixed and everything would be fine. Even though it broke, I didn’t need to panic and waste my time calling support to find out what was wrong.
To be able to know quickly what is the problem is worth more than spending my time trying to convince support that something is wrong or trying to figure out what the problem is when its a global problem and my call to support is nothing more than a waste of time.
Also, to those people who “lost thousands,” you should take $100 out of those thousands and get a dedicated host. The only reason I don’t do that is because my sites are fairly minimal, and not much resources are required for them (but there are many of them so unlimited domains is great). I think its readily apparent that dreamhost isn’t meant to host a site like amazon.com, or anything crazy like that.
August 2nd, 2006 at 7:03 am
I’ve been with you guys for several years now and once again you’ve proven yourselves with honesty and an up front explanation. Keep up the good work, I’m not going anywhere. Loved the photo choices :)
August 2nd, 2006 at 7:08 am
Great to hear that you guys are on top of it and it’s nice to see that you explain what’s going on instead of giving a vague ‘prefab’ answer.
August 2nd, 2006 at 7:10 am
Oh, the huge manatee! http://www.flickr.com/photos/dancentury/192334925/
August 2nd, 2006 at 7:23 am
Yeh, I really like how honest and upfront you guys are. I have had 4 different hosts in the last 6 years and most of the time the companies would act like nothing was their fault.
August 2nd, 2006 at 7:27 am
That’s the kind of attitude that everybody should have with their customers.
Honesty, humbleness and courage.
Now I see that I made a good choice migrating to DreamHost.
August 2nd, 2006 at 7:34 am
I’m sure in the last 2-3 weeks computers started a war with us. I’ve had 2 backup servers dead and one main server went right after them taking all data with it…
I will have to repeat few of the posts, but the people loosing so much money in few days of downtime, what were you thinking when you signed up for $10 hosting? And where is YOUR recovery plan?
I’m not in favor of downtimes but my web site is being down about 0.62% of the time since I joined DH about 2 months ago.
In the end it is pointless to move from one hosting to another. I’ve done it already and it is just lost time. All shared hosting services fail it is just a matter of time and it is in the nature of computers and computer systems in general.
But then again I see very few PPL angry here, which is not that bad I guess. A free of charge month sounds like a good deal ( or a free domain registration or cpanel :) )
Thank you for explaining ( not for the problems as some ppl here think )
August 2nd, 2006 at 7:34 am
These problems where in fact a pain in the ass, but they’re still nothing compared to problems I’ve had with various previous hosts. I’m staying with you guys, that’s for sure, but it’ld be nice to see transatlantic connections on your priority list, because they are really, really slow in peek-hours.
but hey, can’t have it all, can I?
dreamhost rocks!
Belgian greetings
Sil
August 2nd, 2006 at 7:38 am
[...] By the way, you can read the official post by the DreamHost owner regarding the downtime and problems. [...]
August 2nd, 2006 at 7:40 am
Bad stuff happens sometimes, you guys are doing a good job and I won’t be going anywhere. Thanks for your transparency on it as well, it’s one of the reasons I stick around.
August 2nd, 2006 at 7:41 am
Dreamhost rocks! I don’t care about downtime, as long as I know you guys are always working on it. Actually, that terabyte of bandwidth makes up for everything. Thanks Dreamhost!
August 2nd, 2006 at 7:42 am
If I could do business with you guys I would. This is an excellent example of a company standing head up to problems. Good job!
August 2nd, 2006 at 7:49 am
Whew! That’s a mess. Nonetheless, nice recovery. I also appreciate your candor.
Now, I’d ask you about “lessons learned” from this experience.
Have you sent the landlord a bill for all the overtime [on the presumption you had to pay it] pay it cost you?
When you write the next contract have you considered a Service Level Agreement with penalty clauses that are painful enough to get the landlord’s attention?
Did you ask MySpace if they’d rent you some space?
August 2nd, 2006 at 7:50 am
I’ve been with dreamhost forever - at least 5 years - and in comparison to the years of excellent service you’ve provided, a 3 week span of up-and-downery isn’t catastrophic. Your name is still good, I still recommend you to every web client I get, and I’m still a more than happy client of yours.
Do what you guys gotta do to make it right - we’ll wait for ya. Thanks for all the work!
August 2nd, 2006 at 7:58 am
I previously worked on Satellite Communications and thus a great deal with UPS’s and generators. We never had significant problems, but we always had backups for the backups. It surprises me that we could maintain adequate power levels, even in the Middle East, and your building cannot provide you (us) with realiable, quality power. Definitely keep looking for move opportunities. I’m sure we’re all more than willing to take some downtime to eliminate future power issues. Voltage fluctuations can wreak havoc on electronics, so that definitely didn’t help the situation. Good luck with everything!
August 2nd, 2006 at 8:09 am
Mistakes and Moments of Incompetence are Understandable as long as you are honest…
An interesting story about DreamHost being publicly, brutally honest.Read the comments, they are funny………
August 2nd, 2006 at 8:12 am
And email is down again!
August 2nd, 2006 at 8:15 am
Thanks for the detailed update. Many other sites were so cryptic about what happened there were rumors circling that the Garland Building caught fire and burned down… thanks for getting everything back up and running so quickly.
and let me put it this way to all the people whining.. would you rather that all of this happend in a week (so you’ve got some site troubles for seven days?) or it happen through the course of a year (so you’ve got troubles every other week?) quite honestly I’d rather have it happen all at once, that way it’s done and over with..
August 2nd, 2006 at 8:17 am
Some feedback to “Office of the Building”. Providing a great many details with respect to the equipment failure only provides the customer with a curiosity of wondering how much more can go wrong. By providing a great amount of detail regarding the outage, but no personalization or contact information (simply signing “Office of the Building”) is insincere and shows that you do not want to face the issue with your customer. That’s disconcerting.
My company had an extended outage last year that cost us hundreds of thousands of dollars in renewals. However, it could have cost us millions of dollars had we not called in employees, put up an alternative contact page (a sign-up page with a phone number where we could reach our customers at) and personally contacted each and EVERY client who was impacted.
Hiding behind a meaningless signature is terrible. I would not put up with this.
Doug
August 2nd, 2006 at 8:22 am
[...] August 2, 2006Anatomy of An Ongoing Disaster This site has had some downtime during July because of some unfortunate problems with our hosting service, Dreamhost. Fortunately for us, July is our slowest month of the year. [...]
August 2nd, 2006 at 8:26 am
Could Global Warming Kill the Internet?…
The current summer heat wave has been blamed for taking out MySpace for 12 hours, and more anecdotally the internet does not seem to be weathering the weather to well. The few mailing lists I subscribe to are filling up……
August 2nd, 2006 at 8:28 am
[...] The email newsletter I mentioned has a link to an extraordinary post on the Official Dreamhost Blog that chronicles in exhaustive detail what’s been behind most of the quite serious service issues the company has experienced in the past few weeks: As I’m sure most of you already know, we’ve had nothing but troubles, large troubles, for pretty much the last three weeks. A lot of these troubles were our fault, a couple of them were at least ostensibly beyond our control, and they all compounded each other. [...]
August 2nd, 2006 at 8:35 am
DreamHost: Should I stay or should I go?…
The answer, Bob, is I should STAY!!!!111!!1!!one!.
GoDaddy and Yahoo! wouldn’t of given me this detailed of an explanation….
August 2nd, 2006 at 8:44 am
82. Tim is 100% correct. My situation is a little different as I have been here 7 years or so but the bottom line is that our clients don’t care what the excuse is.
This blog post was cute and answers some questions but it doesn’t keep our clients warm at night.
August 2nd, 2006 at 8:46 am
Let me join in the chorus of customers thanking you for your candor and honesty in what happened, especially flagging up your own faults.
This was a really frustrating time for me since I was trying to set up a new business site during this whole commotion. And at the time, I couldn’t get the whole story, so it was tough. I’m glad now that I know and all my sites are humming along fine, so I’m still a dedicated customer. So, thanks.
August 2nd, 2006 at 8:46 am
The photos really make that whole well-written summary very funny. It really got the point across.
August 2nd, 2006 at 8:48 am
As a loyal DreamHoster through a few turbulent times, I appreciate the familiar folksy honesty and transparency. Not because it’s cute and it’s respectful (though of course that helps), but because it actually convinces me that issues like this are less likely to occur in the future. DH knows where the problems are (in the building’s infrastructure, and in their own processes), so this is an actual learning experience.
I agree with those who suggest that irate customers who are losing significant profits due to downtime should seek a new solution. There are definitely options if you’re looking for backups on top of backups and redundancy on top of redundancy. They may cost a little more, but compared to the “thousands” you’ll lose in just three weeks of intermittent outages, it sounds like a steal. Pick your new hosting company with a quick Google search for “guaranteed 100% uptime” — it’s not possible, but at least you’ll get a couple dollars in pro-rated refunds if you stay up late and submit trouble tickets for every momentary network glitch you find at 4am. Apparently that sort of thing makes you feel better.
If you’re like me, you selected DreamHost based on its stellar feature offerings, excellent prices, and very-good support. I’ve experienced some disappointing outages from DreamHost, and the occasional slow support reply, but they’ve also gone out of their way to assist with silly glitches in my own code or setup, and the level of access and configurability are unmatched in my experience. But I host a dopey blog; if you host a mission-critical e-commerce hotspot, you have to live with your decision to save money on your hosting bill.
If your personal zero-tolerance policy for downtime mandates that you leave DreamHost (and you’ll leave your new host in a week after that), I’m glad to see you go. For those of us who stick around for the improved service as DH learns from its mistakes, the support turnaround will be even quicker now that there are fewer people clogging the tubes with 20 goddamn support requests in six months, just because once or twice a week it takes more than three minutes to check e-mail.
Thanks, as always, Josh, for an enjoyable and informative read. (!)
August 2nd, 2006 at 8:54 am
Thank you guys for being honest and detailed in your problem assessment. It makes me really happy to be a dreamhost customer, even when things aren’t going well, because at least you guys are very transparent about what happens.
August 2nd, 2006 at 8:56 am
thanks for the honest explanation.
just a heads up: it was VERY frustrating to read the status page and read that things were all fixed and i STILL couldn’t get email, and webmail, and FTP, etc. that happened a couple times during that weekend.
please watch what you promise on the status page.
also, at one point last week i needed to contact support but the web panel was down. when i tried to email support i got an autoreply saying i needed to contact them through the web panel. heh. yeah.
thanks for working hard for us all
August 2nd, 2006 at 9:17 am
Wow, 200 comments lol!
I am proud that Dreamhost is working hard into solving everything :)
I’m not going anywhere! I am staying with Dreamhost!
August 2nd, 2006 at 9:22 am
I guess I fit somewhere in between. While I appreicate the explanation, I really do find the recent service completely unacceptable. That said, I’m not looking to move to another host immediately. You guys are at the end of the rope, but not quite past it yet.
August 2nd, 2006 at 9:24 am
I think all of customers appreciate your blunt honesty and effort to do everything you can. I also appreciate the fact that when you need to make a system policy change you announce it ahead of time unlike some other ****y hosting companies I’ve dealt with in the past.
Jeff
August 2nd, 2006 at 9:25 am
[...] Monster hosting service, Dreamhost, is living the SaaS nightmare right now. It appears the dominoes of disaster all decided to line up in a continous chain of events for them. While I worked as a maintenance manager at a major newspaper facility, I told people that there were an infinite amount of things that could go wrong between getting the pages and putting the paper out on the street. My job was to manage that risk by attacking every possible point of weakness in the system. Sometimes, those things were outside of our control, though! Dreamhost is finding this out right now… but they are actively communicating the disaster through their corporate blog (clog). [...]
August 2nd, 2006 at 9:29 am
I love you guys and all, but can you just fix the fucking Control Panel, for once and for all? Really. Just fix it. Because it sucks and I’m tired of it, and I can’t contact support because I can’t fucking login. It’s not Firefox, it’s not my connection. Three browser and hang hang hang.
August 2nd, 2006 at 9:29 am
bunch of clowns.
August 2nd, 2006 at 9:30 am
I’m a Sr. Network engineer and have hosted with DH for a while now. [Not looking for a job, sorry. Allergic to the LA area.]
Don’t use Extreme gear, while not utterly abysmal, they have too many problems to be part of your carrier grade network. Cisco or Foundry are the only vendors I’ve seen to be adaquate (juniper might also, but we haven’t really had them long enough to be sure). Single vendor isn’t necessarily always the way to go, but it does tend to simplify matters, and unless you have a really compelling reason, I do advise that you stick with cisco end to end. Or shift entirely to another vendor, as painful as that may be.
Secondly, push hard on cisco’s TAC to get the weird issues resolved. Hire a network engineer. Have critical spares (which is a better strategy than redundancy, it’s actually less downtime with the simpler network arrangement when you have replacements ready to go). Generally, keep the network as simple as possible and plan expansion. Don’t let it grow organically by itself, that way lies future suffering.
I’m very impressed with your honesty, and while I’m generally supportive of your office culture, you need to become more aggressive in protecting your business, both with contracts with the building (downtime fines) and with cisco’s TAC to resolve the weird network issues. I know from personal experience that you have to esclate the ticket up to third tier support sometimes to solve the weird ones. (Hire a real bastard and keep him locked up until needed?)
Make Cisco aware that you have a very visible network, and their performance on this issue will have a larger impact than just your network, but the perception of their company as well.
Downtime, while sometimes unavoidable (asteroid strike) is generally unacceptable. I suggest that you take a fresh look at your vulnerablities and see what you can do to address them. UPS’s fail. Power fails. Networks, especially overly-complicated ones, fail. You probably do have some equipment issues related to power spikes, so your own UPS’s are a start, but I’d have replacements for all your network gear ready to go on no notice (images and configurations updated regularly) on hand and tested good.
I hope that you look at your recovery procedures and training, and overhaul it regularly. Make procedure review a regular procedure.
Looking forward to hearing about what steps you’re taking. I realize that you’re being vague about them now, as you should be, but once you have a better plan etc, do share it with the same openness you’ve been exhibiting. It will only solidify further your customer support, as you can see here.
-R
August 2nd, 2006 at 9:31 am
I also greatly appreciate how forthcoming you were during the whole process. Not knowing why my site went down is so much worse, and what I had come to expect from other web hosts.
August 2nd, 2006 at 9:35 am
Don’t be too upset, Dreamhost employees. In fifty years you can say things like, “Yep, sonny-boy, I remember the great power outages in the summer of aught-six!”.
You know, any way you can work the word ‘aught’ into conversation.
August 2nd, 2006 at 9:38 am
[...] Here’s a link to the post: http://blog.dreamhost.com/2006/08/01/anatomy-of-an-ongoing-disaster/ [...]
August 2nd, 2006 at 9:38 am
[...] Josh Jones over at DreamHost blogged about the recent interuptions brilliantly. He didn’t defer responsibility. He didn’t belittle its impact on the customer. He took it on the chin and explained the situation clearly, as if their customers were sitting in the boardroom with him getting the brief. He also used images of infamous disasters well in the post. If you read the comments from the post, you will see the positive impact that this type of transparency can have. It might be easier for the smaller guys to make it work for them. But it is possible for everyone. You just have to get real! tags: customer service corporate blogging Dell DreamHost Seth Godin blogservice MySpace iPowerWeb Media Temple BroadSpire transparency Posted by Tom O’Leary, on August 2, 2006 @ 5:38 pm Digg This | Bookmark on del.icio.us | Trackback | Permalink | [...]
August 2nd, 2006 at 9:50 am
Well I´ve noticed my blog was off, and I got mad about it. That kind of problems was the main reason I switched my old host provider for DreamHost.
After reading your post, I´m not mad anymore. I work with Information Security, and I can advise you to purchase a BCP (Business Continuity Plan) and implement a DRP (Disaster and Recovery Plan). It´s not cheap but It can be very usefull for you to handle this kind of situation.
I´m not mad at Dreamhost anymore. I learned to trust companies that share the true picture with their customers. I like the way you do business and I wish luck on solving your (our) network problems.
Best Regards,
Vinicius
August 2nd, 2006 at 9:59 am
“just a heads up: it was VERY frustrating to read the status page and read that things were all fixed and i STILL couldn’t get email, and webmail, and FTP, etc. that happened a couple times during that weekend.”
THIS is the problem. Don’t say things are fixed when they aren’t. This drove a friend of mine away from you guys, and may yet drive me. When there’s a problem, we need to know just a few things:
- You understand the impact on us and have correctly prioritized it
- What is your diagnosis
- Expected time to recovery
- when you’ve implemented a solution (that’s different than fixing the problem; don’t say it’s “fixed” until ***WE*** say it’s fixed. It’s not fixed until we are able to do what we need to do, and your declarations of things being up when they’re not just demonstrate that you’re on a different page than us and our needs.)
Sometimes there’s such a thing as too much information, and constant updates of “this is fixed” when it clearly isn’t is bad, bad communication. We have plan according to what you say, so your word is very important. Sure, all providers hit problems like this but it’s how you communicate that make the difference - not just THAT you communicate.
And throwing some financial incentives our way to stay with you guys wouldn’t hurt either.
August 2nd, 2006 at 10:03 am
To hp (anonymity is wonderful, isn’t it?) and others who are raging and ragging and demanding compensation:
You. Get. What. You. Pay. For.
Dreamhost is incredibly cheap. You can NOT expect the best uptime, connectivity and service for a “Crazy Domain Insane” $30-per-year-for-unlimited-domains package. Period. End of discussion.
You chose not to pay more for a known-quantity four-nines host for whatever reason. Sorry, but it’s on you if you’re counting on your site for uninterrupted uptime. When you cheap out on your business, you deserve what you get.
(BTW, I recommend FutureQuest if you want a higher quality of service, they’re rock-solid and service is personal and fast, but you can’t host 35 domains for $30 or anything close to that.)
This is not a defense for what happened (although FWIW, for an ultracheap host, Dreamhost rocks simply because they TRY), merely a commentary on unrealistic expectations.
August 2nd, 2006 at 10:06 am
[...] Here’s a link to the post: http://blog.dreamhost.com/2006/08/01/anatomy-of-an-ongoing-disaster/ [...]
August 2nd, 2006 at 10:11 am
The problem, sorry, one of the problems with DH is that they do not own their data center, they have no control over what the building does. They also have no 24/7 phone support, well OK no phone support at all, DH is fine for personal sites like most of the people praising them here for taking their personal sites down. but quite frankly I am tired of the “Whoops we lost 30 days of your data sorry” (yes they did that in the past) thats not going to cut it, I don’t care how sorry you are, I run a business and so does dreamhost. I will be moving to a host in the near future that is serious about hosting and that owns their own data center and that also has 24/7 support, so a lot of these problems will go away, no host is perfect, but there are many that are better, I make my living from my websites so everytime they go down I loose. I wouldn’t have a problem if I had a site dedicated to my cat fluffy or something but thats not the case. lol
August 2nd, 2006 at 10:13 am
Datacenters in Middle America?…
Reading about all of Dreamhost’s hellish issues makes me wonder: why are all these major datacenters in major metropolitan areas? The provider I use has DCs in San Antonio and Dallas, which is generally a good idea, I guess, but both of those pl…
August 2nd, 2006 at 10:15 am
I also appreciate the honestly and admission of fault. It is indeed refreshing to see this from a business. However, I have lost an important client because of this issue and I feel that at the very least we should be re-imbursed for the down-time and hassle of all this.
I will be staying with Dreamhost, but most likely on a trial basis because I cannot afford to lose another client as a result of hosting problems or emails not getting through.
August 2nd, 2006 at 10:28 am
As usual, Dreamhost’s honesty saves the day. I’ve been a customer since January, and although the service has been a little patchy, there is always a comforting newsletter every month (nearly :P), and a very open explanation to the more serious issues. Dreamhoststatus.com has also been very useful. Thanks for your ongoing support.
August 2nd, 2006 at 10:59 am
I can’t seem to get any answers related to any of the problems I’ve had with installing software over the past four days, despite repeated requests for assistance.
I appreciate that those of you who may not be in the midst of migrating to a new hosting account or trying to install software, are experiencing only modest outages and feel like the price makes up for it.
But as someone who had the misfortune to decide to take a client’s business to dreamhost in the last week (and since dreamhost doesn’t highlight either of its blogs anywhere that a prospective customer would be likely to find them), this has been frustrating, and highlighted by a lack of competent information-dissemination about estimated times of completion or what specific services are impacted on anything other than a weekly basis, far too infrequently for an ongoing crisis.
I know this is a trial by fire, almost literally, for the Dreamhost team. I’d really like to see them bring in some crisis-management leadership TODAY and start rectifying both the technology problems and the customer-service/information-dissemination problems.
Best of luck with it, guys and gals!
-Sara
August 2nd, 2006 at 11:08 am
I can see that you and your team are working very hard to keep things in order. you made a couple of mistakes, but still clearly know this field and generally know what you’re doing.
Mistakes happen, and how well you all are dealing with it gives Dreamhost a better name to me.
August 2nd, 2006 at 11:49 am
3 weeks of pain and suffering. We are suffering, my customers are suffering. I use my business website as a communication tool between me and my customers. If my site is unavailable, then communication has broken down. This is bad.
I do appreciate the honesty, but I feel like you will need to throw us a bone (like a free month or something) to keep many of us happy.
I believe that what a company does when something goes wrong is more important than whether or not something goes wrong. I understand that things go wrong sometimes. You guys have done the right thing, so far. I am inclined to stay with you for now, but I may start to shop around for a new webhost if this is not fixed soon, and permanently.
August 2nd, 2006 at 12:03 pm
I am very new with DH but I had some experiences with unexpected power down chain failures and with other web hosting solutions too. I think that you’ve done the best that you could profesionaly do in this situation.
I would like though to share a few of my thoughts
1) From my point of view DH has an incredible good quality of services/price ratio.
2) The power down problems are not in the resposability of DH in the first place - but I trust that DH will do their best in the future about this, even relocating if necessary. This kind of failure can create a real mass no matter whatever config and data backup you would have. I’ll stick with you even if another similar thing would happen( I hope it won’t anyway!)
3) The honest explanations where complete, profesional and showed respect and apreciations for the clients. That is a very important thing !!! - but I guess you have to be a little profesional to understand that. I hope you will not give up this style ( you can keep the jokes too; they are funny ) for some marketing considerations! I know a lot of guys that want to marry only rich, good looking maids; let them go and find out where the bitch might be hidden!
4) You could ofer “bones” (as someone said ) like 0 or 1 month free hosting ( I would chose 0) but don’t forget to ask to get paid for damage that the data center created to your image!
5) I will recomend you and rely on you as long as you will be as open and profesional as you proved to be in this unwanted situation.
August 2nd, 2006 at 12:04 pm
Really appreciate the honesty - it’s refreshing! Keep improving on your competence *grin*
August 2nd, 2006 at 12:07 pm
Power Woes Continue at LA’s Garland Building…
The Garland Building in LA has suffered several building-wide power outages this month, knocking MySpace and other clients offline. …
August 2nd, 2006 at 12:10 pm
I can’t tell you how hugely relieved I am to see you ‘fess up so thoroughly. It’s the not-knowing that kills.
Keeping my fingers crossed that the problems are behind you and that I can start recommending DH to people again…
August 2nd, 2006 at 12:12 pm
[...] Effettivamente è successodi tutto e di più e le precauzioni adottate dai sysadmin non sono bastate. [...]
August 2nd, 2006 at 12:15 pm
I should only know so much detail about where all of my online information is stored. I truly appreciate the transparency and straightforwardness. Best of luck; I’m hanging in there with you.
August 2nd, 2006 at 12:26 pm
I blew up my router by pluggin it into 4000 volts son.
August 2nd, 2006 at 12:30 pm
right on except the part about saying “we’re now basically 95% of their data center”. Dream host is big but you would still be lying if you said you occupied anything more then a few % of that 20,000 foot room.
August 2nd, 2006 at 12:41 pm
[...] Check out their blog and read someone elses nighmare. [...]
August 2nd, 2006 at 12:48 pm
You might want to consult a good electrician BEFORE you buy your own UPS’s. One problem is that if you have several UPS on the power line, when power comes back on, they will ALL SWITCH TO HOUSE POWER AT THE SAME TIME. Unless the people who designed your building’s generator backups are on the ball, and unless they control the number and size of the UPS’s in the building, the following can happen:
Upon power loss, all the UPS’s switch to battery power. Once either the generator or outside power comes back, the UPS’s all switch back on to external power AT THE SAME TIME. Once that happens, the voltage WILL sag, and they will all switch back to battery AT THE SAME TIME. If you are on generator power, and the total power allocation for all UPS’s and other emergency-power buss loads is above about 50 to 60% of the running power capacity, the generator voltage will sag, and all the UPS’s will switch back to battery again, probably at the same instant. They will then sense ‘good’ power once the generator power has stabilized (frequency may have sagged as well if they are synchronous generators), and, you guessed it, switch BACK to the GENERATOR power at the SAME TIME. Back and forth until the UPS batteries are totally dead. And then you will have equipment damage.
Trust me on this. It’s worth your time to work with your building engineers and electricians to make sure your added UPS’s don’t exceed their engineered levels. There are ways to stagger UPS switch times, and unless you do that, this is a very likely scenario. I suspect that the rhetoric from your building people might be a result of overloaded generators rather than actual equipment failure.
But good for you and I am happy to hear so much honesty.
Regards, Jim
August 2nd, 2006 at 12:56 pm
I am really impressed with you guys for sharing all of this information. I would normally expect a company to try to write it off as a bizarre collision of someone elses problem, some good intentions, and a stray cosmic ray. I don’t expect you to be perfect, but I expect you to own up to your problems and try to fix them.
Thanks for that. I’m sticking around.
-db
August 2nd, 2006 at 1:06 pm
I went with Dreamhost last year and regretted it almost instantly. The site was up and down, slow and fast, no solutions, only “it will be up soon” (if that). My website is down and no status updates? You gotta be kidding me.
I quickly left. I sold my bosses on Dreamhost’s good name, and they ruined my own.
It was…sad. And disappointing.
Plain and simple: DH, you can do better than this. Get yourself a real Disaster Recovery Plan, the kind that can survive an earthquake and not just a brown out, and a network engineer who knows what they’re doing.
After reading 150+ comments, the smart ones realize that DH is a cheap place for a website that doesn’t need to be up 100% of the time.
These same smart people realize that DH is not good for businesses who rely on e-commerce to pay (at least some) of the bills. I learned this the hard way. This sort of event was bound to happen, because of lack of planning.
You’ve shown your weakness DH, now prove everyone wrong.
August 2nd, 2006 at 1:09 pm
Like someone else said, “I’m not going anywhere, and this blog post is a big reason why. Thanks for explaining what’s going on and being up front about everything.”
August 2nd, 2006 at 1:18 pm
People are losing thousands of dollars of business over this? If you’re making thousands of dollars per week from a website, you could probably afford (and should probably move to) a host with 99.9999% uptime guarantees in the SLA…
I’m sticking with dreamhost for value for money; it seems hosts do features *or* reliability, and I need dreamhost’s features. Looking for places with all dreamhost’s features *and* more reliability, prices start at ten times as much and rise sharply :-/
August 2nd, 2006 at 1:31 pm
[...] …as long as you listen. That’s the lesson DreamHost is hopefully learning right now. For those who don’t know DreamHost is a web hosting service. We host with DH and have a few of our client sites hosted on DH. Why is DH learning a lesson right now (and why should everyone else). Simple, they spoke (truthfully) in hopes that the people and their customers would listen. [...]
August 2nd, 2006 at 1:42 pm
[...] Nu har de udviklet sig til en virksomhed, der viser vejen, når det kommer til åbenhed og kontakt til kunderne. Prøv at læse deres hudløst ærlige forklaring på de seneste ugers problemer. Hvor mange virksomheder har egentlig nosserne til dette niveau af ærlighed - inklusiv at indrømme, at ens egen kompetence ikke har slået til? [...]
August 2nd, 2006 at 2:17 pm
These comments are ridiculous. Yes, let’s applaud DH for fessing up to how incompetent they were (mixing the wrong capacity disks on their file servers, configuring their routers without following proper procedure — i.e. storing the settings in the database). Let’s also applaud them for fessing up to depending on other incompetent companies (the building). In the end however, it doesn’t change the fact that they are completely incompetent and have chosen to deal with incompetent companies.
Unavoidable disaster? Not the fault of DH? Here’s a counter-example from a friend: “My registrar (directnic.com) has their headquarters in New Orleans. They never went down during Katrina nor was their service impaired in any way. Later I found that the first floor of their building was under water and they kept things going with their UPS systems and Diesel generators for weeks. [...] their connection to the rest of the Internet was underwater and somehow it kept working.”
Hurricanes are unavoidable disasters. Incompetence is not. A great company is one that survives an unavoidable disaster. Creating a disaster through your own incompetence and apologizing for it does not make a great company.
DH has a well-documented history of poor reliability and customer satisfaction. This is why they received an “F” from the California Better Business Bureau:
http://www.labbb.org/BBBWeb/Forms/Business/CompanyReportPage_Expository.aspx?CompanyID=13131294
In the end, DH is cheap and in the end you get what you pay for. Does anyone have suggestions for really reliable hosts with the same features (even if they cost several times more)?
August 2nd, 2006 at 2:45 pm
I just want to say thanks for keeping us in the loop and the way that you are so open and honest is very refreshing these days. Thanks!
August 2nd, 2006 at 2:50 pm
Thanks for this post.
I always knew Dreamhost was a special company: friendly and professional support that helps even when *I* did mistakes; announcements before changing policies, honesty about thair business, continuous improvement of services without increasing their prices.
Thia post shows how right I’m by choosing Dreamhost for my personal and work sites. There may be issues from time to time but when a company goes so far explaining problems and taking responsability for their own mistakes, I feel satisfied by the service and so are people which I work with and for. It would be great to get more companies work in this way, especially outside the IT world.
I only hope they’ll be able to offer dedicated servers again (or at least VPS). Until then, this service is the best one I can find on the market for what they offer, and it’s not just a matter of disk space, bandwidth and domains.
Thanks DreamHost, continue your good work.
I’ll stay and with me the ones I refered.
August 2nd, 2006 at 2:57 pm
The reasons for the BBB rating has been given in a previous post, don’t remember where… it basically boiled down to the problems being fixed but the BBB not recording them as fixed, IIRC.
even MySpace, which currently ranks in the top three visited websites in the US (actual position is disputed), went down here. There wasn’t much that could be done about it. The other problems are hopefully now fixed. We’ll have to wait and see, I suppose, but it looks like the problems have stopped occuring one after another.
August 2nd, 2006 at 3:10 pm
[...] I was pretty miffed about the outages, but I don’t feel so bad now, seeing that all of Rupert Murdoch’s money couldn’t keep the most popular web site in the world online. Dreamhost yesterday published a lengthy entry on their blog with all the gory details, which is also a good thing- I like companies that are honest and open about their mistakes (hint hint USPS?). [...]
August 2nd, 2006 at 3:16 pm
I’m with DreamHost for the features — WebDAV, R-on-R, & unlimited domains among others — that I can’t find elsewhere. However, I need more RELIABILITY. I’d be happy to pay more for redundant servers outside of California.
August 2nd, 2006 at 3:24 pm
[...] Seth points me to a posting by DreamHost - a company with some recent troubles and a very straightforward way of talking about them. The comments to the post and reactions of other bloggers (bizsolutionsplus, MessagingTimes, and Simon Wakeman, among others) are illuminating. From the incident, Seth comes up with three "lessons": don’t try to hide your errors, make sure you deliver what you are selling, and don’t expect power issues to get better any time soon. [...]
August 2nd, 2006 at 3:27 pm
[...] The other thing to keep in mind is that this was sort of the straw that broke the camel’s back. Many people at DH had been suffering from a series of semi-unrelated issues that all resulted in periodic downtime, sluggishness and internal services going down. As a result, DreamHost published this article on their blog: Anatomy of an ongoing disaster. It’s one of the best business letters I’ve ever read. If there was ever a way to reassure my confidence in DreamHost, this was definitely it. [...]
August 2nd, 2006 at 4:24 pm
I’m going to side with the “cut the family and friends tone” people.
You’ve written a friendly, supposedly amusing, and detailed account of the problems.
But friendly doesn’t help, humour is a very subjective thing, and who cares about the details? For me the whacky pictures and the “aw shucks” tone really grated. If you’re already angry at Dreamhost, it’s not going to help.
Personally I think the details are there to cover up some of the more annoying parts of the recent problems.
If you read through again, it comes down to this — there were power problems, and there were network problems.
1) The power problems could have been solved with better preparation and less complacency about trusting third parties.
2) Some of the network problems are STILL undiagnosed and unsolved.
oh and
3) No mention of compensation.
August 2nd, 2006 at 4:52 pm
I had a bunch of problems over the past two weeks. The time spent went I had no service I spent looking at other hosting options. I think that is the main effect of this debacle. Faithful Happy Customers consider looking elsewhere. I spent a week trying to convince tech support that the problem was not caused by me repeatedly closing my account — there was a misconfigured switch doing this automatically, causing me big downtime, big revenue loss and big credibility loss. That said, when the problem was eventually figured out, the support staff were sympathetic and did ensure that related problems were solved. The fact is that Dreamhost has, I hope, learned a lesson here. Learned it the hard way yes. But I believe that the company will implement protocols and processes not only to lessen the likelihood of this happening but to improve how they deal with customers as a result. This second part is critical. It is CRM — a basic tenet of modern business practice. So I am sticking with Dreamhost, at least until the next time which I, like Josh (and I am sure many of DH’s staff), hope never happens again.
August 2nd, 2006 at 5:01 pm
I like the friendly tone. There’s plently of faceless corps out there that would only give you the official statement on such an issue and not necessarily devulge anymore information.
August 2nd, 2006 at 5:04 pm
Thanks for the honesty!
I’ve worked in IT for the last 10 years and sympathized all the way through your post.
I still think Dreamhost is a great value. If I have to deal with a bit of downtime now and then, I can. (I’m not hosting anything that keeps peace in the world. Well for that matter, no one else appears to be either based on what’s happening in the middle east.)
August 2nd, 2006 at 5:30 pm
As a former IT guy, I can sympathize with most of this - except for the part about editing routes by hand and NOT making those changes permanent. What a great way to set a gigantic land mine for your future self or co-workers…
I know it’s hard to recruit really top-notch IT talent, but things like this (and some less than stellar conversations with DH support) makes me wonder if DH management is trying to hire the top-notch people or if they just want a staff they can have a good time with. Nothing says you can’t try to do both, but as a customer, PLEASE get some star-quality IT talent in the house. I’m sure you can afford it.
August 2nd, 2006 at 5:44 pm
Josh,
Ignoring your somewhat corney newsletters, I’ve been generally quite satisfied with DH (although our few sites are quite small and not critical to anything).
In my real life, I manage a data center for a large Pennsylvania state agency and have been in an IT area for more than 25 years. Additionally, for a few years, I ran my own web hosting business (until the job and the business left me less than 15 minutes of sleep per night) and I was paying top dollar for a dedicated server at the best company I could find. I’ve learned that Murphy’s Laws apply to data centers 24×7 and especially after 2:00 PM on Fridays. So, I sympathize with your problems. Even if you owned your own data center, even if you owned your own generators, even if you had a thousand UPSs, something would fail - ie: the failover systems would fail, the AC chillers would spring a leak and create a hole the size of Kansas in the ozone layer, the subfloor would begin to crack from the weight of blade server racks and threaten to move the data center to Bejing, or Jupiter would align with Mars without peace to guide the planet (never mind, you’re probably too young to remember).
The only places I know of that don’t have outages are ones that can afford to have fully hot failover data centers where millions of dollars per hour are lost during an outage. Places like Citibank and other large credit card processors, etc. can afford the big $$$ to have multiple data centers located all around the world and the technology to ensure that all of them are 100% up to date 100% of the time. In my opinion, if DH could afford that type of environment, I would say that you are charging us “little guys” too much.
The best you can do is:
- Always do your best,
- Learn from the problems,
- Explain the situation to your customers and ask for forgiveness, and
- Then turn around an try to do your best again.
However, you must always remember that your customers depend on your performance - I’m sure you have some who even depend on your performance for the money to buy their groceries. You need to decide what kind of obligation and commitment you have to your customers and what is a fair settlement when you can’t meet those obligations or commitments. It doesn’t matter who’s fault it is, the buck needs to stop with you!
As echoed in many of the other responses, you all seem to be trying your best and do a pretty good job of keeping us informed.
Thanks,
Chuck
August 2nd, 2006 at 6:07 pm
Wow. I’ve never read any apologia with such humor, tact, and honesty in any corporate blog, magazine, or in-house organ.
You rock. That’s all I can say. Blessings on your heads.
August 2nd, 2006 at 6:44 pm
While I did not notice any effects from the massive series of problems over the past month, I was getting concerned about them. I was seriously considering starting the search for another provider, not something I really wanted to do but something I felt might be necessary n light of the recent problems.
Then came the newsletter and the link to this blog.
The fact that DH has been so forthcoming with the causes of the problems, including admitting guilt, is refreshing. I will stay, at least for now.
I know they don’t need to hear more suggestions, but I cannot help but echo some earlier comments….
First - pick a network gear vendor and stick with them. At my place of employment, we are a Cisco shop, so I am somewhat biased in ther favor. And to the person complaining that DH calling in Cisco shows that they are not competent, my employer is easily a Fortune 50 (I think we may be in the top 25) company. We have a dedicated IT staff of hundreds. WQe open TAC cases with Cisco on a REGULAR basis. We have monthly meetings with Cisco reps that include a review of open and recently closed cases. That is why we go with a company such as Cisco. For the support.
Second - be careful of adding your own UPS units to the building infrastructure. If you are going that route, you have to closely coordinate with the engineers that know the building backup power plant - it might not seem like it, but the backup power plants can be sensitive. They were designed with a specific capacity in mind, and may not be able to accomodate the additional units you plan to install without affecting the reliability of the whole system. If you truly want to go your own backup route, I would suggest doing so from end to end. Start with redundant diverse utility feeds into your own redundant generators into your own redundant UPS systems.
Third - To those that are leaving. Go. It has been said before, but it bears repeating…what do you expect from a bargain basement (no offence intended, DH) hosting company? If you want 5 9’s of uptime, go to a dedicated hosting company that operates on multiple continents with multiple diverse paths…that’s the only way you will achieve that. IMHO, I am getting a bargain with my hosting. I do not EXPECT 99.999% uptime from a host that only costs a few dollars per month.
Fourth - If I were you, DH, I would seriously consider moving out of LA…and California in general. As others have said, there is plenty of cheap space available elsewhere in the country, with plentiful power and access to bandwidth. You mentioned you are not big enough for your own data center, only using about 1000 sq. ft. of space (IIRC), but I would consider moving into a better managed collocation facility in an area not prone to power disruptions.
Fifth - I would take a close look at your power situation in your present building. It seems to me that there were SEVERAL single points of failure. I recently finished working on a data network that required 99.999% uptime or better. When addressing power, I made sure that all critical core hardware had redundant power supplies with redundant feeds fed from physically different power sources via physically different pathways and distribution points. The only place there were single points of failure were in a few of the colo facilities that only had a single utility power feed. From that point on, everything else was redundant - to the point where the UPS systems were not even in the same room with each other - literally they are on opposite sides of the building.
Last - Keep up the good work with the notifications. I really appreciate the honest, down-home approach to notifying us about issues.
Let’s hope things improve from here.
August 2nd, 2006 at 6:56 pm
[...] Luego de todos los problemas de Dreamhost que se reflejaban en Technorati, la gente de Dreamhost publicó su explicación y disculpas. [...]
August 2nd, 2006 at 7:08 pm
[...] But, I kept sending e-mails to their admins, and they kept responding promptly; giving me concrete reasons for the outages. It all made sense to me in a language that I could understand: heat wave in LA, brown-outs, filer problems - they took responsibility where they should have. Then, they posted this detailed account of their failures: “As I’m sure most of you already know, we’ve had nothing but troubles, large troubles, for pretty much the last three weeks. A lot of these troubles were our fault, a couple of them were at least ostensibly beyond our control, and they all compounded each other. [...]
August 2nd, 2006 at 8:34 pm
I have nevern been happier than the day I moved my company off of dreamhost’s mail servers
August 2nd, 2006 at 8:41 pm
Well, I have to say…
I was looking at switching to Dreamhost from Media Temple because I’ve been wanting to run Django. Media Temple’s still running an ancient version of Python which makes it impossible. After reading this though, I’ve changed my mind.
While I appreciate the honesty, the parts of this post that interest me are -
“A lot of these troubles were our fault [...]”
“It turned out the filer problem seemed to stem from the fact that we had one shelf of 300GB disks and one shelf of 150GB disks on it. Apparently they’re not supposed to be able to support this, or at least it’s a bad idea. So, this was entirely our fault.”
“Rather, I’d say it’s the result of bad luck. And incompetence on our (and the building’s) part.
I don’t know if we’ll be able to change our luck, but hopefully we’ve at least learned something and will be able to become a tiny bit less incompetent in the future.”
What not to say on a company blog -
1) You’re incompetent (and provide examples)
2) You’re using your incompetence as a learning experience.
3) You’re not sure if you’ll ever be competent but hope that someday you’ll be a tiny bit less incompetent. (and then emphisize ‘tiny’)
Anyway, while the other hosts in the building did experience the power failure, here’s what at least mt is doing differently -
1) They’re still providing dedicated servers.
2) They had no hand in additional downtime.
3) They’re building their own DC so they can get the hell out of there.
They also claim they’re not oversold and that they assume clients will use 100% of the resources they’re paying for.
So this is why, at least for now, I’m not switching to Dreamhost. mt is migrating their customers to new servers so I’m gonna wait and hope for a better version of Python. They’re adding PostgreSQL too which Django is better tested with anyway.
August 2nd, 2006 at 9:07 pm
Good luck in the future! I read only half of this entry, but I could see where it was going. We are all playing the probability game . . .
August 2nd, 2006 at 10:16 pm
DH,
Nice art work in the post, but it seems pretty clear that lots of people depend on you, maybe more than they should. Your reward for building a popular service is that you now have a moral obligation to ensure the mail goes through - netiher rain nor snow nor dark of night, etc.
The mail problems have been ongoing for the last several months. Tech support hasn’t been forthcoming about this. You’ve also made some especially annoying downgrades to the mail system, e.g. I can’t reply to some messages with many people cc’d because your SMTP server says “too many recipients” (there were 35).
A partnership with a highly reliable mail provider and a one-click “change the DH MX record to point to our five-9 mail partner” would be a good thing to offer. Twice the price/three times the reliability sort of a deal for hosting would be good too.
Reporting that problems are solved when they are not is bad. Several people have complained about this, I’m sure you want to hope for the best, but please make a greater effort not to report things as resolved when they are not. If you really want to be transparent, maybe it would be fair to prospective and current customers to list the number of hours of downtime so far this year.
Overall, it doesn’t seem like you’ve yet made up for the pain you’ve inflicted on people this month. Maybe take a 10 percent cut in your salaries and apply it to getting some reliable equipment. You’ve certainly increased the working hours for a bunch of your users at least that much.
August 2nd, 2006 at 10:29 pm
I’ve been with Dreamhost since June 2005, and one thing that has kept me here (other than 2 years prepaid :P ) is the great customer service and the wonderful uptime. My visitors and clients know to go to dreamhoststatus.com if they ever have problems - BEFORE they send me an email.
As for the root of your problems, sounds like it’s time to seriously look into moving out of the Garland Building. You could do what most Californians who have moved in the last 5 years have done - come to Colorado. Adequate power for even the hottest days, rare blackouts, burnouts, or outages. Heck, our biggest problems are from snowstorms, and we don’t even get those very often, and we still usually have power citywide.
August 2nd, 2006 at 11:18 pm
dreamhost: hot or not?…
So as I mentioned, I just moved the site from “that old server under the desk upstairs” to Dreamhost last week. I was pretty psyched, because I got a great deal and it was nice to be out of the……
August 2nd, 2006 at 11:30 pm
[...] outages judging from the email I received and their blog. Posted in BlahBlah by Administrator RSS 2.0 Name [...]
August 2nd, 2006 at 11:55 pm
You people that host multiple clients here for a couple bucks a month and expect ridiculously high uptime (down 4 days in 2 years?! Oh my god, thats nearly 99.5% uptime! Heaven forbid! You guys have amazing deals, amazing customer service, but apparently some intolerable customers. Reading all these comments have convinced me to never start my own business, for fear of these same people jumping over me when a mistake (which, by the way, was mostly out of their control). Even if these people go somewhere else, they will soon realize that 100% uptime is never, ever, EVER guaranteed. They might ADVERTISE 99%, but never guarantee it.
And please dear God, stop telling dreamhost to be LESS honest. I like it when companies admit they screwed up. If you want a company that would rather lie to you or just keep their mouths shut when stuff like this happens, go ahead and switch. I’ll stay here.
August 2nd, 2006 at 11:59 pm
Also, what are people expecting for compensation? At the rate I got DH, compensation for one month might comes out to about 50 cents.
August 3rd, 2006 at 12:04 am
Thoughts on The Dreamhost Meltdown…
As many readers of this blog know, I’ve been an enthusiastic user and supporter of Dreamhost web hosting since getting turned on to it by Stan a couple of years ago. The service has been great, uptime has been excellent, and you simply can’t find th…
August 3rd, 2006 at 2:15 am
[...] Now, how annoying is that cheerfullness now, really, with MySpace STILL acting up and not delivering what it should. I’m not a MS adept, certainly not -ugly & cumbersome-, but I can image the heartbreak and frustration, since a blip.tv cross-posting faux-pas of mine resulted in this ’starting from scratch’ blog/vlog. Anyhoo, I sure was eager to know what was really going on behind the scenes of MS. A power outage is controllable, certainly with proper backups and generators up to the task. It didn’t take me long to find out that Dreamhost is in the same building as MS, sharing the same facilities, aka the same devistation. And waddayaknow, Dreamhost has a blog. Even better: it maticulously describes what has transpired the last few weeks. Kudos to Dreamhost. MS, not so much. The irony is obvious. [...]
August 3rd, 2006 at 3:33 am
[...] I’ve seen this entry from DreamHosts (my web hosting provider) blog referenced a few times on the network. I finally got a chance this morning to sit down and read it. I have to say, I’m impressed with both the honesty and the transparency that DreamHost provides to their customers. [...]
August 3rd, 2006 at 4:05 am
[...] I am much impressed, though, by a long explatation of the recent problems over on the Dreamhost Blog: DreamHost Blog » Anatomy of a(n ongoing) Disaster.. [...]
August 3rd, 2006 at 5:25 am
I signed up for an account on 31 of july … it’s 3rd August and I still can’t login in the ftp …or email …or whatever …But I stay with them for now … I’m sure they will fix it … I’m waiting my turn cause I know they got other problems bigger than mine.
Good luck dreamhost!
August 3rd, 2006 at 5:25 am
[...] Dreamhost, a web hosting provider, had a miserable July due to multiple service failures. For a hosting company, this is about as serious an issue as there is. That said, the company’s explanation, posted on their corporate blog, is superb: they clearly describe both what happened and their future plans to avoid new problems, taking responsibility where appropriate. They even included a few nice photos (like the one above). [...]
August 3rd, 2006 at 5:30 am
[...] That was until I stumbled upon Seth Godin’s post on how DreamHost dealt with the fiasco, which led me to the DreamHost blog post on the situation. I’ve (infrequently) read the DreamHost blog in the past and am currently subscribed to their status RSS feed and monthly newsletter, so I know they are good communicators. However, the level of detail and honesty in their story was very refreshing (and rare) to see on a corporate blog. Especially one that sells uptime and, without a doubt, angered a good chunk of their customer base with all their recent problems. [...]
August 3rd, 2006 at 5:34 am
As dumb as it may sound as a response to this, I wish I worked for a company like yours…
I’m a sysadmin myself and had a similar (but much smaller on scale) situation. While on my vacation (my honeymoon no less), the company I work for (and sadly still do) decided to call in a contractor to “assess the network.” It just so happened that the following day, a redundant ps on the companies main file server blew… To the point of taking the entire system down and killing a failover in the process.
They decided to call this guy back and fix it… Never called me once… But when I get back, I get damn near fired because I should have prevented it from happening in the first place!
-C
[Once your datacenter problems blow over (hooray for LA heat), I'm seriously considering moving my sites to your network!]
August 3rd, 2006 at 5:34 am
I’ll have to agree with all the other comments. Posts like this one are the reason I use Dreamhost. The general honesty and “down-to-earth personality” you (Dreamhost as a whole) express makes me feel less like I’m working with a bunch of really smart people who would laugh watching me try to set up webmail, and more like I’m working with friends.
(That’s aside from the support and features and everything else, of course.)
I will continue telling my friends about Dreamhost, and this is exactly why. Thanks for your post!
August 3rd, 2006 at 5:53 am
[...] Why MMW Has Been So Slow or Down Lately - DreamHost Blog Our hosting company, Dreamhost, is killing me. MMW has been either down or unbearably slow every day for about 2 weeks straight now. See their blog for an explanation. [...]
August 3rd, 2006 at 7:31 am
I am getting less enchanted with dreamhost, sigh. my home dirs were offline for 2 hours yesterday and only most are now on, some are still offline. their admin’s aren’t particularly good or observant. I am starting to reconsider my loyalty. sigh. they do call back within a few hours when your sites (in my case 20 or so) are down but even then, apparently, can’t fix the problems!!!!!!! BAD SYSADMIN BAD SYSADMIN.
August 3rd, 2006 at 8:16 am
Josh, thanks for the post. I for one am standing behind ya.
And shame on any of you in the community saying things like ‘bad sysadmin’ — you probably have no idea what it takes to run a massively shared hosting datacenter like DH. Neither do I, and I’m humbled by the DH staff who work hard every day.
August 3rd, 2006 at 8:17 am
DH received the following letter a year ago and was told that they would be kept in the loop regarding the building owner’s investigation into the generator failure. I don’t ever recall hearing what their answer was?
“On Monday, September 12, 2005, at approximately 12:35 p.m., the building experienced a total loss of electrical power from the DWP on their primary grid. At this time, the building generators started and began supplying adequate power to the tenants.
At approximately 12:55 p.m., the building power was partially restored by the DWP. At approximately 1:05 p.m., the building experienced another total power failure from the DWP.
During this period of 12:55 p.m. to 1:05 p.m., two of the five generators failed. The remaining three generators were unable to sustain the power requirements of the building causing the emergency electrical systems to transfer into a “load shedding mode” and the building’s UPS system to turn itself off, thus preventing permanent UPS and related equipment damage.
The Office of the Building is conducting an investigation into both generator failures and will keep you apprised of the results.
At this time, the building is functioning at full capacity with four generators to support any future outages.
On behalf of the Office of the Building, we would like to thank all of you for your cooperation during the building evactuation and your patience with building personnel during this emergency.”
August 3rd, 2006 at 9:09 am
I just switched to DreamHost a couple months ago.
Compared to my previous host, DH’s uptime sucks. (Previous host was down maybe ten hours over the five years I was with them, at least half of that scheduled.)
But I sort of expected that. Why? Because DH costs ~25% what the other host cost, for 10x the diskspace and 100x the bandwidth.
A) You get what you pay for.
B) If your website is mission critical, you should think about moving to a host that will include minimum downtime guarantees in your contract.
But don’t forget your checkbook, you’ll need it.
August 3rd, 2006 at 9:44 am
I *KNOW* what it takes to be a sysadmin for a large company - and dreamhost sysadmins are not doing a good job lately. They are failing misserably in fact. Dreamhost is starting to feel a lot like Earthlink and AOL - horrible service and tech support. I am SO frustrated.
August 3rd, 2006 at 10:03 am
You know, I’m just glad they got all of these problems resolved … oh wait … Our site is down again. Well that’s ok. I don’t really care if the site is down as long as I can read a blog entry with lots pretty pictures and a few jokes. My website may be down, but at least this blog is still up! Awesome!
I think a lot of people would admire companies for providing great service, but not me! I, ostensibly like many other people in this blog, admire companies who provide crappy service, but are really honest and forthcoming about being crappy. Yay Dreamhost!!
August 3rd, 2006 at 10:21 am
fab blog - thanks - 90% of my problems with other hosts has been not knowing what’s happening - but tho i check my DH sites regularly (several times a day) i didn’t even notice there’d been a problem - whatever, i still rate DH as best host i’ve used
August 3rd, 2006 at 11:25 am
At least you’re forthcoming with the issues. Obviously, the downtime has been very frustrating for us as well, but at least knowing WHY helps to mitigate that frustration somewhat.
Now I’m just waiting for a post that gives an ETA on when you’ll have these issues worked out… (hopefully before I have to renew)
August 3rd, 2006 at 11:43 am
[...] Alright. So dreamhost has been a little off recently, but I don’t mind. My site’s not that important and if it goes down it goes down. Mike Davidson at Mike Industries just wrote about it and there’s a blog post on the Dreamhost blog, and for the amazing deal I’m getting I’m going to stick with them. What was incredible to find out, is that Mike is making $30,000 a year in Dreamhost referrals. Basically, when you refer someone you get 97$ one time. Mike has referred 647 people over 2 years. Unbelievable. In fact, $97 of those dollars were from me. So if you end up signing up… Share:These icons link to social bookmarking sites where readers can share and discover new web pages. [...]
August 3rd, 2006 at 12:11 pm
[...] The last two weeks this site has been down quite a number of times; responsible for the mayhem was some Big Trouble at our webhost, Dreamhost. I quite like Dreamhost for the features they’re offering (Ruby on Rails, unlimited domains etc - onontbeerlijk voor Het Voorlopig Nog Geheim Project!) and they’re generally very responsive and communicative. They for instance published a lengthy blog post chronicling their problems, illustrating it with images of the exploding Hindenburg, earthquakes and fires, generating some comic relief. It even prompted corporate blogging mogul Robert Scoble to compliment them on their efforts (and also generated an entry in Valleywag). This fast and open communication is quite nice for us customers, though it may occasionally backfire (check what they posted a year ago). Even nicer of course would be enjoying a webhost offering the same features, communication *and* less downtime. [...]
August 3rd, 2006 at 12:29 pm
[...] In response to my caught-on-tape customer service nightmares involving T-Mobile, Ticketmaster, and Bank of America, several ZDNet readers asked that I not focus just on the negative and to look for some positive stories as well. It’s a good point given the overwhelming flood of negativity found in today’s headlines — be they in a newspaper, on a Web site, or on the Six O’Clock News. So, while I don’t have a caught-on-tape episode of a great customer service experience, I found Josh Jones’ account of what has been going wrong at the immensly popular hosting outfit Dreamhosters to be refreshingly blunt, apologetic, and accepting of accountability. [...]
August 3rd, 2006 at 1:17 pm
[...] The huge amount of downloads has sometimes caused technical problems and forced me to switch hosting provider earlier this year. Unfortunately the new provider had some serious problems in past weeks as well, but we are confident that they will be able to solve them. My apologies if the site has been unavailable and or very slow. [...]
August 3rd, 2006 at 1:44 pm
redundancy ReDunDancY REDUNDANCY reDUNDancy redunDANCY REdundancY redundancy ReDunDancY REDUNDANCY reDUNDancy redunDANCY REdundancY redundancy ReDunDancY REDUNDANCY reDUNDancy redunDANCY REdundancY redundancy ReDunDancY REDUNDANCY reDUNDancy redunDANCY REdundancY redundancy ReDunDancY REDUNDANCY reDUNDancy redunDANCY REdundancY redundancy ReDunDancY REDUNDANCY reDUNDancy redunDANCY REdundancY redundancy ReDunDancY REDUNDANCY reDUNDancy redunDANCY REdundancY redundancy ReDunDancY REDUNDANCY reDUNDancy redunDANCY REdundancY … and hey, did most of this happen *after* m y s p a c e moved in? hmmmmmmm
Ya scared me, I didn’t notice the problems because I just doubled my service and actually had plans to rely on it *soon*
But overall I’m a teacher (dhuh) and love that you are both learning and explaining. I just won’t move my main site here yet (the email) until I’m a little more secure… or you are… or we all are… and hey, ditto on the L.A. thing.
It’s nice up here in WA State, duplicate :O)
August 3rd, 2006 at 2:57 pm
Michael says … “You. Get. What. You. Pay. For.
Dreamhost is incredibly cheap. You can NOT expect the best uptime, connectivity and service for a “Crazy Domain Insane” $30-per-year-for-unlimited-domains package. Period. End of discussion.”
Yeah. Nifty. See, here’s the problem. 3 years ago we bought a $50 a month shared hosting package supposedly ideal for running an online business. I researched hosting companies, and everything I read raved about DH. And for the first year - they really were that good. Diamond in the rough, that sort of thing.
In the two years since, yes, they’ve reduced that price … but also reduced their reliability and everything else that is important to someone not running some BS website about their cat ‘fluffy’ (as someone else so elequently prased earlier).
When I say we’ve lost thousands, I mean that as a total for all the times their incompetence has shown through over the last 2 years. If our site is off the air for a few hours, we lose sales and profit. And more than likely, it’s a customer that is never coming back if they then find someone else they like.
And again, not *once* has DH ever included an offer of compensation along with their “Happy-happy-we-screwed-up-again” messages. Not once. Ever.
Plain and simple - it’s not just this one big disaster, it’s every other week (or day). Even now our site is crawling and email still sucks (as it has for months - you’d be amazed at how much email you’re not getting, we ran some tests). If they can’t even figure out how to run email servers properly after months of problems … I don’t think there’s really hope in them fixing any of their other serious issues.
They’re basically clueless. And should stop promoting themselves as anything more than a budget, unreliable hosting company.
- BR
August 3rd, 2006 at 3:29 pm
Keep making excuses for them… I’m sitting here wondering why I’m bothering to pay them anything at all for IMAP servers that have not been functional for the last three weeks. I love timeouts when opening folders.
If the money I’m paying isn’t enough to provide stable service then you are basically scamming people by offering services at a price point where you can’t provide the advertised services.
August 3rd, 2006 at 4:22 pm
A Perfect Example of How a Blog Benefits a Company…
I proudly host my Web sites with Dreamhost and lately I’ve been a little perterbed with them regarding their Webmail service. It has been slow, timing out, erroring out, and overall frustrating. It is with their blog that I learned……
August 3rd, 2006 at 4:24 pm
My site is down again. Is anyone else having trouble?
August 3rd, 2006 at 4:38 pm
You just can’t make everybody happy all the time. It would be nice, but not real. Yeah, I hate time-outs and everything else but you have to get over it. DH still impresses me with their in-your-face honesty. I’m not even looking over the fence.
August 3rd, 2006 at 4:43 pm
Some of my http sites are up… one is still down. Anyone else having this problem?
August 3rd, 2006 at 5:47 pm
Even with all the recent problems DreamHost is still the best host we’ve ever had, by far. Like #293, we’re not looking over the fence. We’re not even THINKING about it!
August 3rd, 2006 at 6:21 pm
[...] Look at how a Hosting Services provider, is sharing complexities in operating Internet Data Centers and thus related uncertainities in service levels. [...]
August 3rd, 2006 at 6:26 pm
Wow! This honest, readable, detailed, and humble explanation makes me much more likely to go with Dreamhost than some other vendor. It’s how people deal with things when the chips are down that counts for me.
August 3rd, 2006 at 8:05 pm
Hosting disasters…
nice…..
August 3rd, 2006 at 10:07 pm
I have heard great things about dreamhost. Other hosts can learn alot form what happened here.
August 3rd, 2006 at 10:35 pm
Here is a though from myself, who works for a data management company that owns 4 data centers, and also has a Dreamhost account. 1) I appreciate the honesty, most people in business are very sly about these situations. 2) hosting my personal website, I am fairly unaffected by this, so it is somewhat an oversight for me. 3) I have a responsibility in my daily job to maintain systems in our datacenter with thousands of concurrent users, and we are penalized if we do not meet 99.9% contractual agreements, and we are about to finalize 99.99%agreements. What does this mean? We pay for every MINUTE that we do not meet that mark. For a SERVICE industry, I think this is a high quality to live up to, and most hosting providers would not step close to this number. 4) Dreamhost makes no claim that they will provide services 24/7, but they are a very reliable service for the needs of a customer such as myself. If you want a high quality provider, be ready to fork up the big bucks, and help pay for many more salaries.
I’ll stick with my $8/mo hosting package, pretty good deal in my book.
August 3rd, 2006 at 11:02 pm
June and July being the first months I really started depending on Dreamhost (ShareTheTruth launched!), I was seriously considering leaving as well.
Your honesty inspires my confidence in you. I’m happy to stick around and will continue to recommend you to others.
Thank you!
August 4th, 2006 at 1:30 am
You know what? I’m not hosted with you, but this post makes me want to be - expect me to move over in the next few months!
August 4th, 2006 at 2:09 am
Well done DH. This type of open honesty is totally refreshing. I’m in the middle of migrating my sites from Site5. This kind of outtage, while not ideal, happens every day. Most hosts chose to hide instead of being honest.
August 4th, 2006 at 2:13 am
When a company can come completely out in the open and say exactly what happened and why it happened, I think that’s highly commendable.
In fact, inspirational to make me want to buy the product when I renew, really.
August 4th, 2006 at 5:21 am
[...] So what did they do? They wrote a blog post about the problems and admitted which ones were their fault. Here’s what Josh has to say: Here I’ll try and go into as much detail as possible about what happened, why, and the steps we’re taking to stop this sort of thing from ever happening again. I can’t excuse what happened, just apologize and hopefully elucidate. [...]
August 4th, 2006 at 6:58 am
You guys are the most honest and reasonable people in hosting business.
August 4th, 2006 at 6:59 am
I’m only a recent customer, and my own usage of the service[s] has been very minimally impacted by the recent operational stresses. When the ‘cart gets running, things could be different.
The important thing at our shop is that we left the last place because they became less and less responsive to queries and totally devoid of proactive information. Eventually it became a situation wherein the customer had NO clue about the status of services.
I have been really favorably impressed with DreamHost’s communications policy. I can live with an occasional inconvenience when I am not behing shunned like a member of the wrong political party.
Jim V
August 4th, 2006 at 7:30 am
I just love reading all these comments from these assclowns who use the standard bullshit line of “I’ve lost so much business” and “being down for 2 hours is costing me thousands of dollars” and the like.
If your website is your only source of income and 100% uptime is mission critical, then you should be taking the responsibility yourself to manage and operate the web services yourself, even if that means bringing the bandwidth to your location and hiring someone to manage a machine. The real business professionals understand that circumstances happen beyond control of the operators of systems. It happens to every one regardless of the level of proactive maintenance.
I have worked with web hosting customers in the past and found the ones that complain the most are the ones that are running the pyramid scam, free internet lotto or any other county fair/carnival/confidence game coded in PHP.
So to Dreamhost, kudos for taking ownership of the issues. Honesty is indeed always the best policy.
August 4th, 2006 at 7:36 am
I only recently came to DH after reading many glowing reports (one from a friend). However, in light of the downtime over the last few weeks have decided to NOT recommend DH as a hosting provider for a VERY ACTIVE professional club website that is looking for a new home.
I came to DH from another provider due to their slowness in propogating DNS changes (even to their own servers) and have only been partially glad I moved here.
DH has the best priced plans on the market, from what my own research has shown. HOWEVER, that’s irrelevant with them having this level of problems.
I was willing move on with my life thinking that it was a one time issue. However after reading this post and several of the comments and learning that these problems have existed for a year, I’m deeply having second thoughts.
The sad thing is, that I only recently decided to move a new site off of my own servers because I knew I would need higher bandwidth than what my Comcast connection would allow. However, the connection to my personal site that I still host at home has had a FASTER connection and more reliable uptime than DH.
Like everyone else seems to be saying: THANKS for the explanations. However words are just words. I’m going to hold off recommending DH to anyone for a while as I wait to see WHAT THEY DO NEXT. Only time will tell whether they are worth staying with or if I’ll need to shop for YET ANOTHER hosting provider.
I realize that moving to another datacenter, building, or city is a BIG undertaking that takes LOTS of planning, coordination, and does cause additional downtime — my employer just moved to a different datacenter across town — however if that is what it’ll take to prevent further future problems, then that is what MUST be done.
ONE BIG SUGGESTION: Please list the location of the datacenter in the sign-up documentation. Anyone in this country knows that L.A. has CRAPPY power with brownouts that occur every summer. Had I known ahead of time that the datacenter was in L.A., I would’ve shopped elsewhere.
August 4th, 2006 at 8:22 am
Right.
I’ve been a happy Dreamhost customer for over 18 months and host both my own shop’s site and those of several clients with them. I’m primarily a designer and Apple tech, not an IT guy or full-time webmaster, and I REALLY appreciate the access and control I have over registration, mail accounts, DNS, etc. After wrestling with AOL, Yahoo, Earthlink and other ISP’s to get clients’ domains transferred, their mail forwarded and goodies like streaming and stores working, I’ve no intention of moving or recommending to my clients that they move.
No, I’m not thrilled at service outages, but I have a different perspective. I live and work in Big Sur, on a 56K dialup, where downed power and phone lines in wintertime are the rule and cell phones rarely work. My clients expect service interruptions and (usually) have realistic expections of technology. Another web designer here also offers web hosting for the websites he builds, with guaranteed uptime, at a price
August 4th, 2006 at 8:27 am
It has been 23 straight interuptable days that webmail has been unusable and IMAP suffers timeouts every other time I open a mailbox… All you people making excuses are pathetic, hell I’d love you as customers, practically worshipping a company that can’t provide the services it advertises.
You are all a scammers wet dream which is a bad thing considering how badly the spam filters at Dreamhost seem to work…
August 4th, 2006 at 8:29 am
Not going anywhere. If at all possible, I move all my clients to DreamHost. If they are hosting elsewhere, I don’t have any answer for them when their site goes down. With DreamHost, it rarely is an issue and on when it does, I have an answer. We are staying here.
August 4th, 2006 at 8:45 am
The War on Lebanon and the Battle for Oil
by Michel Chossudovsky
July 26, 2006
GlobalResearch.ca
Is there a relationship between the bombing of Lebanon and the inauguration of the World’s largest strategic pipeline, which will channel more than a million barrels of oil a day to Western markets?
http://www.globalresearch.ca/index.php?context=viewArticle&code=CHO20060726&articleId=2824
August 4th, 2006 at 9:30 am
I don’t host with you guys, but if I needed to I would because of your honesty, hard work and you seem like a great bunch of dudes. To all the people that are complaining because their service was down you’re going to have to get a clue about how technology works in this world. A vendor can sell you the most reliable equipment in the world and it’s still going to fail. Even if you have redundant equipment it still might not failover correctly. The same goes for colo providers because EVERYONE sells stuff they cannot deliver on. It sounds like Dreamhost is trying the best that is possible with what technology is available. If you’re not happy with the service then by all means walk over to Savvis, AT&T or Equinix and rent some rackspace. Then you can get ahold of some vendors and purchase some Netapp filers, a couple Cisco 6509 switches, a 535 pix, a rack full of servers, maybe some nexsans for backend disk. By the way this will probably cost you close to a million for the equipment and service contracts along with the 15-20K+ a month you’ll be paying for bandwidth, power, cooling, etc. Then you’ll have to monitor this equipment 24/7 so WHEN something fails you can replace it right away with hopefully no downtime to your customers. Then you break out the chickens and voodoo dolls and pray that your vendors actually deliver on their 4 hour turn around time for dead equipment.
So I guess if you take away anything from this. Please understand the amount of time, money and effort that goes into keeping these stupid computers up and running. So chill out and remember that these guys are at the mercy of their colo provider and their vendors. Truly all you can ask for is that these guys are busting their butts 24/7 to get your services up and running again and I think you have that. Dreamhost, hopefully you guys can leave the cage soon. ;-)
Peace Out.
P.S. I hope the file corruption wasn’t too bad on your db servers. That shit sucks…
August 4th, 2006 at 9:42 am
Where do I report that the panel isnt working, when the panel isnt working? Quite the predicament. (By the way, the panel isnt working)
August 4th, 2006 at 10:10 am
Part of the problem is that DH has changed its customer focus over the last couple years without really saying so. When I signed up in 2003, DH offered a lot of Quality (reliability, uptime) at reasonable price. They did not offer the most Quantity (disk space, bandwidth, etc)
What I’ve noticed is all the “improvements” since then have been Quantity, not Quality. Now I have more disk space & bandwidth than I can ever use, but the reliability has eroded.
So, to those who say “you get what you pay for”, that’s true, but the cost/benefit equation these days is different than it was when I came onboard.
August 4th, 2006 at 10:59 am
Simple. You guys suck. Lot’s of other ISP out their who aren’t falling over like. Power outage may interrupt service, but its YOUR responsibiliy that wehn power comes back on to back online almost instanyly. MySpace did just that. Sorry to say we’re leaving DH. I’m not identfying my company because we already have suffered enough, and don’t want to have any more association with you. We went to a new provider that was a little more expensive, but whil you chasing your tail last week or so - our company site got records hits due to some marketing efforts and our site didn’t miss a beat. If we were still with DH - it would have been hell. EOM
August 4th, 2006 at 11:16 am
I’ve been with Dreamhost since 1999 and have been very pleased with them for *most* of that time. The response turnaround from customer support has gotten much worse in the past year, but overall I have enjoyed being a Dreamhost customer. Luckily, I don’t host business sites with Dreamhost that would impact my bottom line if they went down. However, the performance of WebMail over the past few weeks has been abysmal. I even resorted to installing RoundCube on my own as an alternate WebMail client, hoping that would fix the problem. Alas, it is a tad bit faster, but the problem definitely seems to be with Dreamhost and not with SquirrelMail.
August 4th, 2006 at 11:26 am
Just wanted to say thank you for being upfront and honest. As a Sysadmin myself, I can relate to all the quirks and problems with hosting. The fact that you are detailed and straight-forward makes me feel like I’m giving my money to like-minded people. Keep up the good work!
August 4th, 2006 at 12:04 pm
[...] There were some other good examples of people managing downtime this week, Dreamhost & Blip.tv. Mike Hudack, of Blip.tv, kept constant posts going explaining what was going on, and then sent personal emails to those affected. Now, we know shit happens. Google goes down, right? It’s all in how you handle it. And these guys point the way to great community management. [...]
August 4th, 2006 at 2:05 pm
It would be nice if you could at least answer your emails when someones sites is not online for 2 days. I have now lost 2 days messing with you trying to get my stuff back online. Which is a loss of both money from sites being off line and then loss of time dealing with you.
August 4th, 2006 at 5:39 pm
http://space4commerce.blogspot.com/2006/08/hey-dreamhost.html
I was nice to you in the comments.
Now will you please fix the *^%&*** error that has my databases down? I only really care about the ones running the Liftport Blog and Forums.
I admit that they are bugging me in different ways. The blog because it’s important. The Forum, well it’s important but there is another reason. It’s configured so that everytime there is a DB error it sends email to me. Such errors occur everytime someone hits the Forum link and it’s flooding my inbox.
August 4th, 2006 at 10:26 pm
Great to see some honesty, as has already been said by many… I’ll not be moving away from DH, but I will be putting a ‘reserve plan’ into place - site backups on lower-specced hosting in an entirely different country, which I can email clients to point them at in case of any future outages.
As a side note, it’d be great if I (and indeed anyone else who’s interested) could be automatically mailed about any major outages, system-wide at least but preferably even server specific, so I know straight away rather than finding out from cutsomers or trying to visit the sites!
August 4th, 2006 at 11:26 pm
Thanks for the info…and owning up to it, as you normally do….that’s why I’ve stuck with you guys since ‘98
August 5th, 2006 at 1:17 am
Wouldn’t think of moving. You guys have been good to me over the last few years. Wish there was something we could do to help.
August 5th, 2006 at 1:50 am
MySpace could well have had just as many problems as DH, but they only have one service with a ton of redundancy — if half their servers had died, the service would carry on, with the other half at double load. With DH, each server is important on it’s own, and a single one going down will be noticed~
August 5th, 2006 at 7:17 am
Dreamhost, my web host has had some difficulty lately…
DreamHost Blog Anatomy of a(n ongoing) Disaster.. Damn.. I am glad I no longer work in that industry…….
August 5th, 2006 at 9:05 am
[...] Good crisis management: Anatomy of a(n ongoing) Disaster [...]
August 5th, 2006 at 9:13 am
As your customer, I am deeply appreciative of your honesty here. It is reassuring to know that there is a level of trustworthy transparency.
I also like your writing style.
August 5th, 2006 at 1:57 pm
To echo some of the above posts, I really appreciate yall stepping up and being upfront with everything that has been going on. As has been said, in a perfect world this wouldn’t happen but….
I’m more than pleased to be doing business with a company that is willing to be so transparent. I’m quite confident that my money has been well spent with DreamHost!
August 5th, 2006 at 2:46 pm
Thanks, Josh. As has been said, I love DreamHost because you guys are so up front with your clients. I’ll be staying with you for years to come, no matter how bad things at Garland get.
August 5th, 2006 at 5:35 pm
[...] Anatomy of an Ongoing Disaster — An entry on the official Dreamhost blog written by Josh Jones, the company’s founder. [...]
August 5th, 2006 at 9:43 pm
Just wanted to n-th my appreciation for the explanation. You guys are great, and I look forward to hosting several more sites with you in the future. Sh*t happens, and hopefully this epsiode of sh*t is about over…
August 6th, 2006 at 1:23 am
Although it is good that this is being explained isn’t it about time that things were back up an running? As of 09.59 GMT today my website and email are STILL both down. I can ftp so the site is still safe but there is clearly some way to go before these problems are fixed.
Please DH, get this sorted out. If this is going to be an ongoing situation, with sites going down over and over again then please let us know.
August 6th, 2006 at 3:05 am
Hehe- I understand what those photos are for now!
Click my ads please: http://nadinetannous.blogspot.com/
Nadine.
August 6th, 2006 at 8:51 am
Seth Godin’s lessons from Dreamhost…
From Seth Godin’s blog: Three important lessons from Dreamhost Zeljko points us to: DreamHost Blog � Anatomy of a(n ongoing) Disaster… Lesson one: when things get messed up, being clear, self-critical and apologetic is really the only way to deal….
August 6th, 2006 at 10:13 am
I just switched over from another webhost. They also had big problems, the only problem was they did not tell you what was going on!
You guys are great, I will never need to look for another hosting company, keep up the good work!
August 6th, 2006 at 1:13 pm
[...] My absence apparently did not interrupt excitement here at the blog, nor at my fellow tenants’ places. I’ll save comments towards our common host for a time of obvious stress, as my own abode has not mustered such abundant tourism. - Break - The image you presumably see above, represents a cropping of an effort begun about a month ago, on a familiar medium, namely Winamp. The evidence of my vacation from the skinning or theming “scene” has become increasingly palpable to me and created an appetite I am trying to satiate through this creation. What you see here, is still in progress and elucidates very little of the fluid transitions I want in this design. However, I am hoping to transcend my previous (and modestly successful) concept with familiar help and some new ideas I have developed for increasingly concise interaction. [...]
August 6th, 2006 at 8:33 pm
I feel for Dreamhost and though this was a rather annoying outbreak of misfortune for all (customers and employees), I think you learned a lot on this. And how to start servers nearly errorless after power outages ;]
And, no worries, I will be with Dreamhost for quite a while. *nod*
Thanks you guys!
Logan
August 6th, 2006 at 9:14 pm
I’ve been a customer since 2003. I love Josh’s writing style and I really liked Dreamhost. But the last 6 months or so there has been NO reliability. Yes, DH has a fabulous package, which is the only thing that kept me from leaving 3-4 months ago after a seriously frustrating episode.
Now, after this seriously frustrating episode, I AM leaving. It’s a massive hassle, but there is nothing DH can do to compensate me for the issues. My hosting plan is $20 a month…which is admittedly dirt cheap for the features. But even if DH gave me a month free hosting, that doesn’t help me with the 30 clients I have who were down too. I’m not losing “thousands” of dollars, but I am losing respectability with my clients.
I came to DH after horrible experiences with Hostway (who took 72 hours to do anything and were snotty if you asked a question) and Yahoo! Geocities, which had the worst support ever. DH was WONDERFUL compared to them…emails were answered usually within 2 hours, things happened right away, and the control panel was easy to use. But…lately the support has been shoddy and takes forever to get a response. And the reliability is just not there.
I just can’t afford to wait anymore for DH to get it’s act together. And let’s be realistic - will DH really care that we left? Prolly not.
August 7th, 2006 at 2:57 am
How the heck do we report the panel is down? There’s no way to contact support (as far as I can tell) except THROUGH the panel? Emails to support@dreamhost.com get returned as unread for not having a valid ticket # attached.
August 7th, 2006 at 5:09 am
DH must have a lot of customers not running a business off their sites. There’s no way you would be as understanding if these outages were costing you any significant amount of money. The fact is DH provides impressive features but crappy service. It’s like owning a great car that only starts 4 days a week.
August 7th, 2006 at 8:01 am
El Apocalipsis de Dreamhost…
El mes de Julio será recordado por los administradores de Dreamhost como el infierno en la tierra (o al menos en su Datacenter):
Empecemos viendo el suceso desde un post de Überbin:
Luego de todos los problemas de Dreamhost que se reflejaban en …
August 7th, 2006 at 8:16 am
No matter how honest you are at admitting faults, they’re still faults. I can’t sit on my ass and wait for (almost weekly) email downtime to sort itself out. And because of the slow support I become the help line to the clients I have referred to DH.
Which is why after 6 or so years I and many of the clients I have referred, are gone and moving to more reliable service with quicker support help.
August 7th, 2006 at 8:42 am
This is hurting our business. Big SIGH!
August 7th, 2006 at 8:42 am
[...] If you haven’t noticed, my blog has been extremely slow to load for the last couple of weeks. Apparently, Dreamhost has been hit by a variety of network and power problems that caused massive outages across their network. Their offical blog explains it all or at least most of it. Coincidentally, before the series of troubles began for Dreamhost, I noticed the slow load times for this site as well as Synergos Design. Both sites are on the same database servers and both sites use CMS. When I first reported the issue, a support representative responded by letting me know that he did notice a “higher-than-normal resource usage” on the database server. But then too my dismay, Dreamhost was hit with a wack load of problems. Here is what they wrote to me: Hello, [...]
August 7th, 2006 at 10:15 am
And you guys are down. AGAIN.
This is intolerable. I’m outta here as soon as I can line up an account with WiredHub or a similar competitor.
August 7th, 2006 at 10:24 am
[...] Just last week, Dreamhost experienced a major outage - you can read details on their blog. [...]
August 7th, 2006 at 11:17 am
What is the point of the DreamHost Status page and this blog, if the information is not up-to-date? (can’t reach the panel either)
NightmareHost: you are down again. Would like to know why and and how you plan to increase stability on short term.
August 7th, 2006 at 11:44 am
My email is still not working. I can’t send anything. So I tried to email support@dreamhost.com like it said to on the status page from a yahoo account and it bounces back saying support@dreamhost.com is down. I’m going to try to fill this out on the Panel, but I don’t know what to do. We’ve been without email since 5am!
Is anyone else having email problems still even after DH said it was up and running?!
August 7th, 2006 at 11:49 am
wow, the geniuses fucked me over again. no outgoing email for 5 hours and counting. i can’t wait to hear your next cute explanation, and outpouring of empathy from people WHO DON’T USE THEIR DREAMHOST FOR ANY KIND OF REAL WORK. as of tonight, i am backing up my site and leaving. urging other friends to do so, and they are.
August 7th, 2006 at 1:55 pm
We are happy and staying. We found Dreamhost after getting screwed by some other hosting companies. Everyone has problems occasionally, but you do not try to hide yours; and yours are accidents - some companies’ are not.
August 7th, 2006 at 2:11 pm
The biggest joke so far is having to report a side-wide outage throug the ‘award winning’ pannel twice in a couple of hours; receiving a “we believe this issue to be fixed” mail and when checking your site having no access to it and timeouts…
My contract ends in a week and it will just be their luck that I don’t have the time to do all the moves or I would seriously consider moving to another host. I’m happy of the support but I wish I would not need it in the first place.
August 7th, 2006 at 8:12 pm
[...] Er, yeah. I was also motivated to do this because of the useful/informative nature of blogs like this [...]
August 8th, 2006 at 1:46 am
My contract ends in a while, but I´m staying another year. It´s been alot of shit recently and once they´ve sorted out, which understandably could take a while, I believe they´ll get back on track.
What keeps me here is the dreamhoststatus/blog/honesty, the panel (when it´s working) and not having to worry about bandwith/storage.
The support is great, even tho there were 350+ tickets open, I got my issue tested, fixed and replied with in ~2h.
Best regards,
Ossi
August 8th, 2006 at 2:10 am
This has happened to us before (we offer co-lo services) and it caused us abject misery for 6 weeks becuase all our kit overheated due to lack of air-con. Servers fell like dominoes not immediately, but randomly over a space of time.
This was due to hidden fatigue in the hardware caused by the heat problems. There is no way a visual check from a sys admin can show these problems up. I’d urge you to get your hardware vendors in to do specialist integrity checks on them if excessive heat was an issue at any time.
Being in europe, our room cooled itself as night drew in and our diesel genny kept us going for the 8 hours we were off power. We were lucky with the issue itself, if not the imapact.
Total sympathies DH - this stuff happens and its outside of your control. But get your hardware looked at. And if you get your own UPS, dont buy anything but APC UPS boxes. deploying the cheaper stuff is actually worse then depolying no UPS at all. We’ve had diffs with some UPS boxes.
btw - i think you did the right thing in being 100% honest with your customers.
August 8th, 2006 at 2:16 am
Okay, what is the refund policy at dream host? All the apologist in the world and all of the moronic “what do you expect for …$ / month..” comments are not going to rectify the apparent incompetence of the dreamhost technical staff.
After the Powweb sale I thought that the problems there were bad; but these guys make Endurance look like geniuses
August 8th, 2006 at 4:37 am
Dreamhost and their problems…
DreamHost Status pages list all kinds of problems DreamHost is having with their service. A lengty explanation about their now-famous problems can ……
August 8th, 2006 at 9:22 am
Know what? Everyone has a valid point, be it pro or con DreamHost at this point. And believe me, i have landed in almost all of the camps the past 3 weeks. For each one of us our sites are Mission Critical, regardless of what we pay for the hosting services. Everyone has an opinion, and for the most part, they are valid. What is the solution? If you are interested, I wrote a blog about what I am doing. DreamHost team, i hope you read it.
http://www.spiritonthejob.com/content/view/216/44/
August 8th, 2006 at 10:36 am
“If you are interested, I wrote a blog about what I am doing. DreamHost team, i hope you read it.”
Great, so now I have to pray for my email to work again.
Well, nothing else has fixed it, maybe that’ll help.
Oh wait … I’m an athiest without working email at DH. I guess I’m hosed :)
August 8th, 2006 at 11:12 am
People need to understand that type of service that they are getting. A dedicated server is a dedicated server. Paying for redundancy is redundancy. Going with the cheapest service and expecting SLA redundant 99.9% service at a fraction of the price dosen’t exsist. You get what u pay for. No matter who you go with things like this happen.
August 8th, 2006 at 11:33 am
While I truly sympathize with Dreamhost’s hard working IT staff, I must confess I’m delighted that my company made the decision last year to leave Dreamhost and host our own mail and website. It was hardly a difficult decision to make. The once-a-year big outages like the one detailed in this blog are one thing, but we were having email outages lasting from 15 minutes to over an hour almost every single day. Since bringing our hosting in-house we’ve had exactly zero downtime. Zero. Downtime. Let me repeat that again–ZERO DOWNTIME. Had we stayed with Dreamhost, frankly I don’t know how we could have stayed in business.
August 8th, 2006 at 12:14 pm
“Going with the cheapest service and expecting SLA redundant 99.9% service at a fraction of the price dosen’t exsist.”
You’re absolutely correct, but that’s not what we’re talking about here.
We’re talking about a hosting company that in the last year has gotten to the point where your email doesn’t work multiple times per week, sometimes for hours at a time (if not an entire day). Added to the miriad of other problems they’re not even worth the $20 a month we pay them (originally $40 a month). Oh, and I just noticed on the front page that they’ve got a “special” on our plan (”Code Monster”) again and are charging us $4 more than someone who joins right now. Nice.
Again - they didn’t used to be a low-cost, low-reliability hosting company. Apparently, now they are.
If that’s what you want, and you’re paying $8, good for you. There are people here paying far more per month and not getting what they are paying for. We’ve also seen the service degrade in the last year to what you see now.
We now have to find someone who isn’t a bargain-bin hosting company and go to the trouble of moving our site, neither of which we ever wanted to do.
DH *used* to be awesome. It’s sad to see them go down the drain.
August 8th, 2006 at 5:50 pm
I see from these comments that many people love dreamhost so much because they are “so honest” but I don’t understand why anyone thinks that! Their home page does not mention anything to new customers of the troubles they are surely in for. Of course they are honest with the people who are already customers. It’s not like people aren’t going to notice that their sites are down, their email is not working, and that they are losing clients and respectabilty.
Dreamhost is making so much money from all of us and all the new customers that believe the hype, so why can’t they put some of it back into their company to provide realiable service for their customers?
Many people here also seem to think that this “just happens”. It does not just happen; not to this extent. I have been reselling for another company for 6 years, and can count the number of hours of downtime (including email) on the fingers of one hand.
August 8th, 2006 at 7:52 pm
The SMTP server was down yesterday and it took Dreamhost 15 hours to fix the problem?! And the email service has been having frequent problems in the previous two months.
Has Dreamhost given up on improving the reliability? It seems like I’ve better setup my own server, so when something goes wrong at least I can do something about it, instead of waiting and waiting for Dreamhost to fix it.
Now the POP/IMAP server is down, for over 12 hours already and still not fixed. I can’t tolerate it any longer.
I’ve been using Dreamhost for years, but in the last 3 months it kept having email outages, and each time it took them hours to fix the problem. It doesn’t “just happen” when it happens almost weekly.
August 8th, 2006 at 8:48 pm
I’m not going anywhere. Keep fighting the good fight and I’ll keep referring people. I’ve had much worse… at least you guys are honest with us about the problems, and this isn’t your side job, so I know you’re all dedicated to making it better. Thanks - John
August 8th, 2006 at 9:03 pm
It’s nessary tounderstand how such hosting provider can help in the business. I had tried many cheap (that means Bad) hostings and lost my data and my money. And so i recommend to use only professional, not cheap, hosting company, as Dreamhost is. Thank u!
August 9th, 2006 at 6:27 am
[...] Spurred to action by my web host’s recent spate of problems, I have attempted several measures recently to speed up my website. Upgrading from PHP 4.4 to PHP 5.1 and adding FastCGI support seems to have helped. (I am running PHP as CGI instead of an Apache module to facilitate one-click web-based installs of WordPress and the like.) I also made sure my third party software is up to date. I do appreciate Dreamhost’s honesty about and detailed explanation of the hellish debacle they faced during July, but I expect it not to happen again. Methinks a discount on next year’s hosting might be an appropriate conciliatory measure towards their customers as well. [...]
August 9th, 2006 at 7:12 am
“Hey my email doesn’t work, but I’m staying with you because you’re honest.”
What type of logic is that? Think about the reliability that is expected when you signed up. I signed up 7 years ago and got what I paid for, which was great service and support. The last 6 months I have had to contiuously wait for dreamhost to fix mail problems, then has only provided vague “should be working now!” emails that don’t make up for the hours upon hours I could not send important and non-important mail. They’ve been less than helpful, or informative each time… nor have the even provided any kind of reimbursement for down time.
As of this weekend we’re washing our hands of this place, it’s been a memorable ride but we’ve got shit to do.
August 9th, 2006 at 8:10 am
Could you please fix your damn IMAP servers so I can at least get a copy of my personal data? For some reason on your servers all my Maildirs in the shell accounts are owned by root and I can’t even make a copy.
August 9th, 2006 at 8:18 am
hey, just wanted to say your honesty and detailed explanation is such a breath of fresh air. I’ve had similar outages with other hosts and they wouldn’t give any explanation or acknowledge any wrong doing.
August 9th, 2006 at 8:47 am
My site is down yet AGAIN! I can’t take too much more of this. I’m only hosting a few images and I’m freaking out. I can image what business owners are going through. I’ve been with you guys for a few years and I feel really let down. :(
August 9th, 2006 at 10:26 am
How about another honest answer to these questions…
Any chance of receiving the email that was and is still being lost during these outages? How about other people receiving the mail that was sent?
August 9th, 2006 at 11:41 am
It this kind of honesty that will keep me with Dreamhost. Every web host has problems. Not every web host tells you what is really going on.
Good luck. Things can only get better!
August 9th, 2006 at 11:50 am
Wow, your detailed and *honest* description to your customers makes me *want* to host with you!
August 9th, 2006 at 12:34 pm
“Good luck. Things can only get better!”
Actually no. Since DH has “fixed” everything, our site has been down almost every other day. That doesn’t even count the email outages. If this keeps up, August will be even worse than July.
Your assumption would be true if the people at DH were at least as competent as most other web hosting companies, but I can see from the mistakes they made (and continue to make) that they are not.
Fundamentally the problem is that DH keeps adding all of these whiz-bang features, but doesn’t address systemic reliability problems. I frankly don’t care about “1 click install” this or that. I care about our site being up 8-5 M-F. Is that really so hard to do??
No, it’s not easy to be a web a host. But why is it that other hosts have not had these problems? All of the problems listed were the result of bad decisions by DH.
It’s really amazing to me how much customer loyalty these mistakes have *created*. In fact this has generated more loyalty than if DH had done their job right and not had any of these problems. Amazing.
Good luck indeed. DH needs all they can get.
August 9th, 2006 at 12:40 pm
Wow, Next time I need a host for my meaningless blog with next to no traffic and an email account that I do not need to rely on it being up during peak hours of a productive *work* day, I’m coming to dreamhost because they are *honest*!
August 9th, 2006 at 1:07 pm
I have been up for the past 2 days moving my clients sites AWAY from Dreamhost.com, I will not mention here because I am not trying to promote another hosting company on DreamHosts website, because I would not want someone elses business posted on my website.
I will say this.. For being a dedicated customer and hosting over 103 sites with DreamHost.com you can have the friendliest, nicest staff and all the greatest features in the world, but if it doesnt work its all meaningless.
Can you imagine moving 103 Website for customers? With Databases and paths to change in each site?
Regardless of the honesty in which Dreamhost shares with us all here there are still many truths left out of the story and while its better to tell the truth than lie… Telling the half truth is just as bad…
Dreamhost has Oversold their services for almost a year now due to this problem, they were not anywhere near prepared to handled mulitple issues that have become compounded in the last few days.
Its like clockwork… around 11:am eastern time.. sites begin to creep along and then finally time out.
Well Dreamhost… I have timed out! Im outta here!
For any future DreamHost.com customers… think twice! Unless you are completely fine with Massive hard drive space and Bandwidth… that is served up slow and part of the time.
Good bye Dreamhost
Please do a search on Google for xxxxxxxx Sucks on any host and see what people really have to say.
Did I mention Dreamhost has an F with the Better Business Bureau?
August 9th, 2006 at 1:13 pm
Tell me about it.
The past 3 weeks we have been loosing money because of this hosting. We don’t know how long this is going to go on.
We have telemarketers that have been seating behind their desks and not being able to sell anything because of the server being down, being slow, and not being able to send or receive emails.
Compared to my previous host, Dreamhost’s uptime sucks.
August 9th, 2006 at 1:39 pm
This is ridiculous… Your IMAP servers have been throwing errors for over 3 weeks. The only reason I am still getting my mail is because I have my account setup to forward mail to a HOTMAIL account.
I agree with the last 2 posts, I work with the guy from the first post. This has been terrible for our business and moving is something I would really not like to do again. Dreamhost has ruined it with me and I will not be coming back.
All I want is my sites to work.. Is that too much to ask?
August 9th, 2006 at 2:00 pm
I guess it’s okay to uproot the email system during business hours, again and again…
Maybe it’s time to hang it up and sell out to a company who can handle volume as well as handle an emergency without throwing everyone offline.
You really need to get it together, or simply get out of the business.
August 9th, 2006 at 2:28 pm
I couldn’t agree more with the unhappy legions; in fact I agreed with them last week and the week before, finding myself subjected to the snarky replies of some dude name Mike who called it ‘whining’.
At any rate, I’m jumping ship but need a recommendation for a MUCH better (i.e., halfway decent) webpage/email host.
Any serious suggestions heartily appreciated.
August 9th, 2006 at 3:05 pm
i switched to DH from one of those “lets charge you up the ass and promise you 99% uptime” web hosting companies.
Had more problems during that 1% downtime than anything DH has had. And when you got to customer service, you were routed to India and the towelhead would say in broken english “what problem?? We have a problem?? It must be on your side”
Again, I side with those with some smarts, if you are making thousands of dollars an bhour from your website, and still host at the bargain basement companies, you are an idiot and get what you deserve….
Want reliablilty and make thousands of dollars an hour? Why dont you spend some of that money and build your own servers?? Than you would see what it takes to make it all work 99% of the time…
You get what you pay for… And those who bitch loudest are usually those asking for everything, but demanding to pay nothing.
August 9th, 2006 at 3:06 pm
I can’t wait until we’re off Dreamhost. Could someone please forward me any reliable hosts (even if they are several times more expensive).
My email: spolo (at) designfm.net
August 9th, 2006 at 4:41 pm
Quoting Armen (comment 379):
“We have telemarketers that have been seating behind their desks and not being able to sell anything because of the server being down”
I guess every cloud *does* have a silver lining! While I am as frustrated as everyone else with the recent troubles at Dreamhost, I derive some satisfaction from *any* event that interferes with the business of telemarketers.
August 9th, 2006 at 5:08 pm
Haha. I agree with rlparker. I don’t see how you can call something bad if it’s hurting telemarketers. Hooray for downtime! :)
August 9th, 2006 at 5:13 pm
I see Barry’s still a whiner.
August 9th, 2006 at 5:13 pm
If you spent your time fixing the problem instead of writing this article, the problems may have been corrected by now
August 9th, 2006 at 6:42 pm
As I write, my website just went down because:
Warning: mysql_connect() [function.mysql-connect]: Lost connection to MySQL server during query in /home/.ferris/xxxx/xxxxxxx.com/classes/drivers/adodb-mysql.inc.php on line 346
could not connect to database
And ofcourse,
- my ADWORDS are running and charging to me.
- you guys are just going to write explanation.
- things will happen over and over again.
I think might as well change name to cheaphost. It is no longer a dream, or is it nightmare?
August 9th, 2006 at 6:47 pm
Then pause your ads, genius. You should be doing that instead of posting here.
August 9th, 2006 at 6:53 pm
apart from the fact that my website just temporarily went down just now, I hadn’t noticed all these problems.
Keep up the good work dreamhost!
August 9th, 2006 at 8:12 pm
I guess you could divide these comments into 1 of 2 camps; those with expectations of service and those without. Probably these correspond to those of us who have referred others to DreamHost or have clients and those who haven’t. I’ve been burned - badly - in the last four weeks and I cannot explain what that feels like to anyone who has not hosted clients. I was not expecting many 9s of uptime but uptime! This was nothing mission-critical nor would a more expensive provider with an SLA have been appropriate. As John said above - like clockwork, my site gets slow and goes down at ~12:00 CDT - down with an “unverified” outage for about 2 hours this afternoon. This is bad and I’m getting off. My own site will be the last to go but if this doesn’t improve by an order of magnitude in the next week I’m going with. Please, PLEASE, do something!
August 9th, 2006 at 8:31 pm
[...] DreamHost Blog » Anatomy of a(n ongoing) Disaster.. [...]
August 9th, 2006 at 9:06 pm
写在千篇时的一律…
这是第一千篇日志，之前有很多篇日志一直停留在草稿状态，故而这篇日志的序号也就不是1000了。不想把这篇本该像阶段性报告一样的日志真的写成阶段性报告，不过最近有几件事情，…
August 10th, 2006 at 8:58 am
STUPIDEST FUCKEN HOSTING COMPANY EVER.
August 10th, 2006 at 9:07 am
I think everyone should just walk away from dreamhost as we are in few days.
We sell advertising space on our site (directory) and the past 3 weeks has been nothing but down time. And it really looks bad front of our customers.
So do yourself a favor and just walk away.
August 10th, 2006 at 11:09 am
I appreciate that at least the admins are willing to let us know what the hell happened. That’s more than any company will give you when there’s a problem. Trust me, I work for a company that doesn’t tell me sweet shit all, but im supposed to tell you that “were working on it and we have no ETA…”.
Thanks Dreamhost, and now more than ever, you have my support. In fact - I just pulled over ANOTHER domain from GoDaddy.
August 10th, 2006 at 1:34 pm
I’ve been doing my best to move my sites off DH. Moving the email has been a total horror. They’re all on IMAP and I can’t get logged in to DH about 90% of the time the last 3 days.
I’ve got 50+ clients and I’m administering DH accts for several more. I get a call every time something doesn’t work.
I did a mass edit to change the DNS on several sites. Guess what? They’re magically still pointed at DH. Gee thanks. So I told my clients to expect downtime a few days ago (which they probably got because DH has been so wonky) and now I have to tell them again. Oh yay.
Now I’m trying to access the panel and change the DNS *AGAIN*, but it’s down. I own a web dev company, and the load of downtime followed by migration is seriously impacting our delivery dates. But what choice do we have?
August 10th, 2006 at 3:15 pm
I agree with Moschops. I’ve been with Dreamhost I think for about 4 years now, and have had nothing but great experiences with them. You guys have generally been forthcoming about issues, and a little levity and honesty goes a long way to reducing concerns. Granted, I’m sure some of the people who are hosting with you for the intentions of making a profit (or, worse, an actual living) may not be so forgiving, but so what. You guys are awesome, and your support team is, as they say, fanatical.
August 10th, 2006 at 4:54 pm
Like some have mentioned, I’m still here also. I’m really pissed off, but still here. This little fubar is killing my business by destroying both my credibility and yours.
PLEASE. Fix it so we can all get back to doing what we love.
August 11th, 2006 at 2:12 am
[...] Dreamhost, a web hosting provider, had a miserable July due to multiple service failures. For a hosting company, this is about as serious an issue as there is. That said, the company’s explanation, posted on their corporate blog, is superb: they clearly describe both what happened and their future plans to avoid new problems, taking responsibility where appropriate. They even included a few nice photos (like the one above). [...]
August 11th, 2006 at 10:28 am
Server hillmont is having problems.
That one’s ours. Errrr!
August 11th, 2006 at 11:00 am
I write this as my 10 hosted sites are down (I’m on Hillmont) and webmail takes over 30 seconds per click… I feel for you guys. That sucks. I’ve worked those nights and weekends and screamed at the computers. They aren’t fun.
But I’ve lost clients now. Three of our 10 clients in the last month, mostly due to their perceived instability of their sites and mail. I don’t want to have to switch companies, but if this persists, I’m going to have to.
Could you at least hire a high school kid to reply to your tickets or something? I never feel like I get a timely or accurate response through the support system.
August 11th, 2006 at 11:02 am
[...] We’ve pretty much got our whole network under control now.. the ongoing problem mentioned last post was finally figured out by Cisco support. It turns out it was a bug undocumented feature in IOS dealing with how they learn MAC addresses. [...]
August 11th, 2006 at 11:37 am
Thanks for your candor. It makes a big difference to know the full story.
August 11th, 2006 at 9:25 pm
Dreamhost is totally CRAPTASTIC!
FYI Post# 384 the man is an Dreamhost employee. Read the post and you will figure it out.
If you want better hosting go to 1and1.com I moved there and its great! I move my site away from CrapHost.com or Nightmarehost.com
Watch the little movie about their datacenter…
http://order.1and1.com/xml/order/DataCente
Why doesnt DreamHost have one of these?
Oh…because they would rather wait until a situation like this happens to make a move and fix things.
Well you know what folks…. Dreamhost will break again.. they just dont have the technical backing on the inside and they damn sure arent spending the money where they need to.
I have moved all my business away from this place.
Just do a search on Google… Type Dreamhost Sucks! You will see the light!
August 11th, 2006 at 11:21 pm
Ohhhhhh,
It was awfull, I lost part of my visitors, and positions in search engines. But I’m still with DH. Because I hope that all problems were temporary
August 12th, 2006 at 5:57 pm
Greetings. Can someone find out why the password for gary@garycaininsurance.com still isn’t working?????
It was changed half a day ago. The message said the update would take up to two hours. Well, its been seven hours and it still isn’t working.
This is a brand new account for me, and once again, Dreamhost is killing my credibility. Please, please, please!!!! Fix this right away and let me know what’s going on.
If this guy checks his email tomorrow morning and it still isn’t working, he will start asking for refunds, and it will be almost IMPOSSIBLE TO GET A REFERRAL FROM HIM!
My entire business revolves around referrals and credibility. Please, FOR GOD’S SAKE, HELP ME!!!!
Its been nothing but problems with my account, and your CEO promised this would get fixed last week!
You know what??? What you need to do now is start offering compensation. What would you people do if I didn’t pay my bill huh????? Would you be understanding and just write me one letter after another reminding me my payment was due for my hosting account????
Not a chance. Listen, You are killing my business with situations like this. If my customers make me refund their hosting fees, you need to refund me mine. Just like me, if you cannot provide the services you advertise, you need to refund your customers.
Its 9pm on a Saturday night and I’m sitting here working my ass off trying to stop my business from crashing down around my ears (thanks for you folks). You owe me two phone calls for support problems reported to you last week, and I still haven’t heard a word from you.
Make me understand why you should not refund every penny of mine and everyone else’s fees???
Enough is enough
With Regards,
Cody P. Skidmore
Chief Technology Wonk
Parsimony Software
http://www.parsimo.com
August 12th, 2006 at 11:48 pm
Five minutes after his hysterical post, Cody Skidmore shot himself in the head after realizing that life was too complicated for him.
This sentence shows that you don’t know how to run a business:
My entire business revolves around referrals and credibility. Please, FOR GOD’S SAKE, HELP ME!!!!
If your entire business relies on 100% uptime from a shared hosting account, your business is going to fold anyway.
Time for you to go back to flipping burgers.
August 13th, 2006 at 12:03 am
lol!
August 13th, 2006 at 12:24 pm
At the end of the day a business that can honestly own up to its part of any outage is a business committed to never repeat the past.
Team DH .. thanks for the info you guys are tops..
August 14th, 2006 at 3:14 am
[...] The recent buzz around iServe has made me more happy than ever that I chose to host most of my sites with Dreamhost in the States. Take a gander at each company’s approach to recent difficulties. Dreamhost, after ongoing outage and power issues, posted their Anatomy of a(n ongoing) Disaster blog post. Absolutely fin’ brilliant. Key: A lot of these troubles were our fault [...]
August 14th, 2006 at 11:01 am
Notice to readers on this site.
Any Comments that criticize your frustrations or resentment towards DreamHost.com are written by Employess of DreamHost.com
As you read through the comments you will occasionally find a remark written by what looks to be like another individual, but in fact it is a Dreamhost employee.
Why else would someone show up here to read this long excuse for problems and then post rebuttals or slams about someone having problems?
If you send Dreamhost.com support tickets that show your own frustration, tone, or overall current feelings about your service, you will notice that they refer to is as in whinning (SEE POST 383)
Most people here who are accepting Dreamhosts excuse for their 4 week excursion of problems are the simple customers who really dont rely on web business or their web business is very small and has very little traffic.
Honestly Folks Think About!
DreamHost.com announced a few months ago they hit 300,000 customers. Now not all customers go for the Cheap-Inexpensive $7.95 plans.. Their are plenty of customers who do the $63.95
plans.
So do the the Math:
300,000 customers x $7.95 = $2,385,000 Per Month
Or Average it out…by plan price 7.95+15.95+31.95+63.95= $119.00 (Lets find the mean) $119.00/4(plans)= $29.75 is their average monthly per customer.
So.. 300,000 x $29.75 = $8,925,000 per month
No one is really to know the exact figures Dreamhost.com is rolling in… but its in the Millions Monthly and their is NO EXCUSE FOR THEIR POOR SERVICE.
August 14th, 2006 at 2:52 pm
I am a current DreamHost employee. We won’t delete comments made here on our blog (no matter how incorrect), but we WILL respond to them. The simple truth is the above poster is wrong on every point.
Firstly, he is not an ex-employee and his lack of the understanding of our company exposes that pretty clearly.
DreamHost just passed 300,000 hosted domain names, not 300,000 customers. The very basis of this math example is incorrect. Naturally, our customers also do not evenly distribute themselves among our plans.. and we won’t even get into promo code discounts for their first year. 9 million a month is simply incorrect.
DreamHost employees always sign their own names on their posts. We conduct ourselves with integrity and respect for our customers. It would be truly shameful to disparage the very customers who make our business possible and none of these snippy posts are by us. We wouldn’t dream of doing it.
- Pete
August 14th, 2006 at 5:40 pm
The guy that wrote post #414 couldn’t multiply his way out of a 4th grade math book if his life depended on it. Not to mention, his reading & writing skills are just as impressive. Customers, domains… same difference!
I’m sure DH has earned millions, earn being the keyword. So what? Would you feel better if they made minimum wage like you?
Yet another post that proves there is a directly link between whining and being braindead.
August 14th, 2006 at 9:22 pm
[...] I made the decision to switch hosting providers for johan.org–from TextDrive to DreamHost. Although DreamHost have had their share of problems recently, I’m confident that they have taken the appropriate measures to remedy the issues. I particularly appreciated the candid response to their customers and prospective customers like myself regarding these unfortunate turn of events. [...]
August 15th, 2006 at 9:51 am
Stop whining!! Seriously. So you can’t send email?? So customers can’t access your website?? So what? Have you ever heard of something called the *Post Office*??
My site has gone down at least 46 times since I started with DH, so let me tell you what I do. When a customer emails me with some “urgent” request and I can’t respond because our website and email are down. I just whip out my typewriter, type my response, put it in an envelope and drop it in the mailbox. A few days later, voila! they have it on their desk. If they get upset because they needed information quicker for some “presentation” or something, I tell them “Hey, quit whining! Running a hosting company is hard!”
Here’s a quick tip for dealing with angry “braindead” customers: A lot of times I’ll just print out a paper copy of this blog and send it them (but you should probably block out the parts about DH getting an “F” from the Better Business Bureau). Who knows how many contracts we’ve won from customers who tell their bosses, “Well we don’t have any information from that company because their website and email are down, but I love their writing style and they have pretty pictures of the Hindenburg!”
Sure, there are hundreds of other companies that have been able to survive true disasters for a similar price (as in post 240), but our company actually sees more uptime as a disadvantage because then we don’t get as many humorous apologies. I say, “Dreamhost, don’t worry about being incompetent, just keep being honestly incompetent!!” (And keep up those pretty pictures. Man, our customers love those pretty pictures!).
August 15th, 2006 at 3:06 pm
[...] As if Dreamhost’s well-publicized problems during the last three weeks of July weren’t enough to frustrate a shared web hosting customer such as myself, with a single humble domain hosting a couple of lightly-used blogs and a photo gallery, lately my shared hosting server, happy.dreamhost.com, has been sluggish at best, and frequently just dead to the world. I’ll share a few statistics that are probably interesting to no one but me, and maybe the few other people who read or use the web services on auroralux.net: [...]
August 15th, 2006 at 5:04 pm
Data Center Outages Bring Headaches, Headlines…
An outage July 30 at Seattle’s Fisher Plaza was the latest in a series of data center outages caused by equipment failure. …
August 16th, 2006 at 5:35 am
I identify myself as a dissatisfied customer in this mess.
I won’t regurgitate what’s been said about it though. For a summary of the “thumbs down” position, I recommend reading posts 56, 79, 101 and 282.
I will add this:
Last night, I noticed my local e-mail app had forgotten my saved password and asked me to re-enter it. Shortly thereafter, I was contacted by a client of mine about the same problem. There’s something on the blog about “Relay Access Denied errors”, which may or may not be related to this issue, and how to resolve it by doing a lot of techincal stuff. Now I’m thinking, how is it that both Dreamhost and I both have to work to fix theese problems, but only one of us is getting paid?
Also, to all those people claiming there are no other companies offering what DH offers for such a low price - If you do some research, I think you’ll find that there are. I’ve found several, and I’ll be switching to one of them when I get home from work tonight.
August 17th, 2006 at 4:16 am
I was wondering why my website was down so many times.. but I have to say that I am happy for the explination and I hope you guys can figure out the problem. I have been with you guys for a little while now, and Im a satisfied customer. So hopefully you can fix everything soon!
August 18th, 2006 at 11:24 am
[...] Qué mes. Ha sido un mes movido debido a los fallos que ha habido en el servicio. La causa principal ha sido los cortes de electricidad y el fallo de los generadores de reserva del edificio donde se encuentran sus servidores. Nos lo cuentan en su blog oficial. [...]
August 18th, 2006 at 2:06 pm
[...] DreamHost har dessutom en skön webblogg där man bland mycket annat kan läsa om när det inte gick så bra för ett tag sen. [...]
August 18th, 2006 at 6:14 pm
[...] One of DreamHost’s head honchos Josh spilled the beans on all of the problems we’ve been experiencing lately in a blog post on the official DreamHost blog. Josh tells of internal DreamHost screw-ups, datacenter building management issues, and why we’ve had so much trouble with power lately! Consider this to be an end-call ‘FAQ’ to the current going-ons with DreamHost. It provides some great information as to what specifically went wrong, when it went wrong, and what else it caused to go wrong. While much of the article discusses issues that were caused or snowballed by the datacenter itself, he makes it clear as to what DreamHost screwed up itself and how it’s being fixed. [...]
August 21st, 2006 at 4:20 am
[...] So I have picked Dreamhost as my ‘hosting partner’, and used their one-click install of Wordpress to get up and running. Why did I choose Dreamhost? Yes I know they have had some issues over the last few months, but I found the fact that they are open and honest about their operation a positive (this being the best example). Plus in the end, what they offer is a fantastic deal (plenty of storage, plenty of bandwidth, reasonable price… what more do you want from a hosting company?) [...]
August 21st, 2006 at 11:52 am
[...] Posso recomendar-lhe uma coisa? Leia este postzinho aqui. É um bocado longo, mas vale bem a pena. Ah, e já agora, os comentários também, sim? Eu espero. [...]
August 24th, 2006 at 3:43 pm
[...] So I think I may be done with Dreamhost. I’ve been sponging my hosting for quite some time from a gracious friend, which is nice. Only problem is the server is one of Dreamhost’s. Which, up until the past few months, had been great. They have fantastic prices, excellent service, nice extras. Mike has found it in his heart to give them a break, and I had as well. But I’m seeing trouble again, whether it be mail or MySQL or just plain network outages, seems like it’s always something. For all the great stuff they’ve tried to do to mitigate these issues and all the damage control and transparency they’ve had, they just haven’t manage to fix the problems. [...]
August 24th, 2006 at 5:44 pm
[...] Well, DH has an official blog and has offered up a lengthy post explaining the root causes behind the recent issues and just how they're looking to prevent such things in the future. That's right up-front and neighborly of them, if I do say so myself and a complete rarity in my experience. Crick don't rise, I'll be sticking with Dreamhost for the forseeable future based on this alone. [...]
August 25th, 2006 at 1:12 am
[...] DreamHost blog [...]
August 29th, 2006 at 12:29 am
[...] Tras una hora de búsqueda por internet, me topé con dos quejas en concreto: las caídas constantes de los servidores durante las pasadas semanas y la limitación de uso de la CPU. A la primera cuestión encontré las disculpas de Dreamhost explicando lo acontecido. A la segunda cuestión también le encontré solución, pues habian dejado de limitar el uso de CPU, ahora si no molestabas no te llamaban la atención. [...]
August 29th, 2006 at 3:37 am
http://filmy-z-shemal.lolek.pl | filmy z shemal | [url]http://filmy-z-shemal.lolek.pl[/url]
August 29th, 2006 at 6:33 pm
Nice post, really nice, the better way, the true way.
Good luck,
August 30th, 2006 at 4:27 pm
After having hosted with dreamhost off and on for 5 years and watching your service slowly degrade over time I have decided to no longer recommend your hosting to any of my clients.
I have no qualms about mentioning who I *will* be recommending from this point on - 1and1 has been to me these past few months what you guys were when I first found you - courteous, timely and dependable.
August 31st, 2006 at 12:20 am
[...] S er Codemonster kommet. 60 GB plads, og 1.6 tb traffik. Og et hav af one-click installs. Og meget, meget mere. Denne post gjorde udslaget. [...]
September 2nd, 2006 at 2:27 am
[...] 8. Stay aware of Dreamhost Events - The Dreamhost status pages keeps you aware of what exactly is the problem going on in different web servers, what is the cause of downtimes and when scheduled upgrades are expected. You can subscribe their feed for every anouncement on Dreamhost Status. They are willing to truly convey their problems. [...]
September 6th, 2006 at 6:31 pm
[...] Data Center Disaster… One of the key elements of PACS administration is ensuring patient data is secure and safe. We take all the precautions we can (within budget), and hope for the best. Sometimes bad things happen. An ISP in the US had a month of hell in July. Read about it. Bookmark to: [...]
September 8th, 2006 at 11:14 pm
Is the service stable now ?
September 9th, 2006 at 7:21 am
[...] What a frustrating evening for the World’s Greatest Online Magazine. It seems that my server host was having some “technical difficulties” all evening — the worst night in a long series of bad days. They claim now that they’ve got things under control, but that there will be an outage sometime on Monday. So, for tonight, I am hoping that the server will be faster than it has been. If you are interested, they have posted a story about their troubles. I wouldn’t read the whole story, just look at the pictures. Pretty funny. [...]
September 12th, 2006 at 11:17 am
[...] c) una importante cuota de ingenio cuando piden disculpas por las interrupciones en el servicio. [...]
September 17th, 2006 at 12:00 am
[...] Anatomy of a(n ongoing) Disaster..What a three weeks As I m sure most of you already know, we ve had nothing but troubles, large troubles, for pretty much the last three weeks. A lot of these troubles were our fault, a couple of them were at least ostensibly beyond our … [...]
September 20th, 2006 at 12:00 am
[...] Anatomy of a(n ongoing) Disaster..What a three weeks As I m sure most of you already know, we ve had nothing but troubles, large troubles, for pretty much the last three weeks. A lot of these troubles were our fault, a couple of them were at least ostensibly beyond our … [...]
September 20th, 2006 at 6:20 am
[...] recommend bookmarking both if you are a current Dreamhost customer. Posted at 9:19 am | Comments (0) Post yourcomment: [...]
September 20th, 2006 at 1:35 pm
[...] Some of you may have noticed that the website has been sorta slow the last several weeks. There’s a reason for that. My host, Dreamhost, was having all sorts of issues from power outages to hardware funk. [...]
September 21st, 2006 at 4:49 am
A “Real” Example…
After my recent post on the Three R’s of Web Marketing, I was challenged on the premise of being real with customers. Is honesty always the best policy? Well, nothing is always the right approach in any situation, but being……
September 26th, 2006 at 3:10 pm
Why would anyone consider hosting with a company that *didn’t even think about needed multiple datacenters” until there was a power problem? And even then choose a host in the same building? Do you guys have any idea of the concept of disaster recovery? Not to mention that the power outage was only a small part of the outage. The major portion of the outage was misconfiguration on Dreamhosts part.
September 27th, 2006 at 11:24 pm
[...] On their part DreamHost openly discussed their problems on their blog (part 1, part 2) which they were lauded for (Mike Davidson, Robert Scoble) because it, of all things, kept the dialogue open between them and customers. [...]
October 3rd, 2006 at 11:28 am
[...] In less than a year, Dreamhost has already upgraded their already insane webhosting plans with up to 10 times more webspace and at least 2 times the bandwidth - an upgrade that comes to all existing Dreamhost customers. In other words, Level 1 plans start with 200 Gigs of webdisk and 2 TB of bandwidth. But is this really a good thing? I mean, we’ve already experienced a wackload of problems with Dreamhost that seem to be ongoing. Now, I’m not sure if the previous increase in bandwidth and diskspace was a direct correlation with the network issues that Dreamhost is facing now. One thing that’s for sure is that these “insane” deals are in fact bringing Dreamhost a lot of new customers. According to their blog and webhosting.info, Dreamhost hosts over 300, 000 domains and ranks 24th among the largest webhosts in the world - a nice jump from 200, 000 domains and ranked 39th back in January 2006. Having researched that, there is no doubt increasing bandwidth and diskspace means increased customers. Increased customers means increasing number of shared servers. Increasing shared servers means increasing network problems. Which begs the question again, is this upgrade really a good thing? I remember when some of my sites would take up to 20 seconds to load because of the high resource usage on the database server and there was little that Dreamhost Support could do except to tell me to wait as they tried figuring out the problems. [...]
October 3rd, 2006 at 8:57 pm
[...] Now, not everyone’s a fan of Dreamhost - but the one thing I do appreciate is that their support blog tends to air all their dirty laundry. This was probably the best entry ever: Anatomy of a Disaster. Especially with my network engineering background, I find it to be an interesting read. [...]
October 4th, 2006 at 1:53 am
First Europa NL
Uw online verzekeringsmakelaarEerst Europa Doelstellingen: De Ci2i Verzekering (Ci2i) zal het nummer een gebrandmerkte pan Europese commoditized online verzekeringsmakelaar door 2010 zijn.
First Europa ES
FirstEuropa.com - tu corredor de seguro en línea Primera declaración de la misión de Europa: El seguro de Ci2i (Ci2i) será el pan-europeo calificado del número uno commoditized el corredor de seguro en línea antes de 2010.
First Europa FR
votre assurance en ligne sponsorisent(Ci2i) être le pan-européen marqué du numéro un commoditised l’assurance en ligne sponsorisent d’ici 2010.
October 13th, 2006 at 3:37 pm
[...] Now, you may think it has something to do with this, this or maybe this. [...]
October 14th, 2006 at 3:50 pm
[...] But no commercial webhosting company I’m aware of currently offers grid-based hosting. This leads to outages and slowness for clients. DreamHost has had a number of recent problems. To their credit, they are quite open and honest about what went wrong. But there’s also a constant parade of announcements of servers going down for 10 minutes here, 10 minutes there for RAM upgrades, hardware replacements, etc. [...]
October 22nd, 2006 at 12:51 am
I am currently using two other hosting services and none has been problem-free. This post (and a promo code found in a Google ad ;) convinced me to buy some service from DreamHost and eventually migrate stuff I keep elsewhere. It pays to be honest :)
OTOH, grid-based hosting *would* make sense, wouldn’t it?
October 26th, 2006 at 6:05 pm
[...] One of DreamHost’s head honchos Josh spilled the beans on all of the problems we’ve been experiencing lately in a blog post on the official DreamHost blog. Josh tells of internal DreamHost screw-ups, datacenter building management issues, and why we’ve had so much trouble with power lately! Consider this to be an end-call ‘FAQ’ to the current going-ons with DreamHost. It provides some great information as to what specifically went wrong, when it went wrong, and what else it caused to go wrong. While much of the article discusses issues that were caused or snowballed by the datacenter itself, he makes it clear as to what DreamHost screwed up itself and how it’s being fixed. [...]
November 7th, 2006 at 4:22 am
Okay going to beat a lame horse to death, as its so much fun :) I have been a dreamhost customer for 3+ years now that I can remember, in that time, Never really had any issues that were not resolved quickly. One large power failure (note that THEY HAD NO CONTROL OVER POWER)(sorry for the caps but for god sake people Dreamhost is not the DWP) what looks like a handfull of people got really upset. What surprised me was how very much alike they all wrote, and very inaccurate about facts they seem to have been. Most likely the people doing the biggest complaining and keep mentioning the same other hosting company are the same person who isn’t smart enough to realize most people in this workd know a good thing when they have it. To you Mr Jerkoff who can not do math worth a damn, I say these five words:
Get a life you poozer.
To everyone else, I’m glad to see that so many people actually know what they have in this company, I deal with Linux systems myself and have a rather complicated network to oversee (in my house of all placed, but I like things complicated, and I mean complicated) I know how bad failure of one part can be, been there, done that, got a crappy T-Shirt.
To the Dreamhost staff, Honesty is the best policy you guys owned up to your faults, and really it seemed like there were a few people just trying to complain to save a quick buck…I hate that. Ignore them and just keep doing the best job you can with what space you can aford. I know exactly what you are dealing with not wanting to pay for space that will not be used until much later, so don’t if you have not, as its a waste of money and for the work you do, you deserve what you can make profit on it. Thats right folks I said the magic P word, profit, as in the money these hard working people get to take home with them instead of investing in more parts for the computers you all use, if the very few people who complained got a problem with that, they need to get a life, and grow up acting like a 6 year old who didn’t get his/her way solves nothing.
November 9th, 2006 at 7:18 am
[...] Wenn die Server eines Hosting Unternehmens “abrauchen”, ist das der Super GAU. Wie DreamHost darauf reagiert, ist ein wunderbares Beispiel für Kriesen-PR: DreamHost Blog » Anatomy of a(n ongoing) Disaster.. Michael Raich | Other | | Artikel drucken [...]
November 18th, 2006 at 11:12 pm
[...] I’ve had a nightmarish experience with my earlier web host, Page-Zone.com. They seem to be going through a Dreamhost period, but three weeks without any response to the numerous support tickets was a bit too frustrating at the end. Hopefully they’ll be able to fix things soon for their existing customers. As for me, I’ve moved elsewhere, and hope to [...]
November 23rd, 2006 at 10:16 am
[...] For me, it came down to MediaTemple and DreamHost. I have used DreamHost for a client site in the past, and found that they are a company that is very concientous of their customer’s well-being. They go out of their way to give easy access to different server features on their (somewhat bizzare looking but very useful) control panel, and overall are very open about different issues that are occuring with their servers (and are not afraid to take responsibility when things get messed up and it is their fault). Overall, I have been pretty impressed with their hosting, as well as customer service. (And am reassured that if problems occurred on a server, that they would take care of it ASAP, unlike my current webhosts). [...]
November 30th, 2006 at 1:09 pm
Excellent customer service from Media Temple…
My account has been a little flaky since I upgraded to the newgrid server. Nothing major, and nothing that wasn’t fixed in an hour or two. I had some trouble with my database, and mail has been spotty. When it……
December 12th, 2006 at 6:00 pm
[...] Dreamhost begs for mercy August 02nd, 2006 Dreamhost crawls out of the rubble and explains the reason(s) for their less than pleasant services over the past while. [...]
December 13th, 2006 at 5:02 am
[...] Al van voor Y2K staan mijn sites bij Dreamhost zonder enige problemen die naam waardig, van de vele problemen die ze gehad hebben in hun datacenter met netwerkproblemen en stroomonderbrekingen ben ik grotendeels gespaard gebleven maar blijkbaar heeft de server waarop mijn data momenteel staat het wel ‘vlaggen’. Vanmorgen bleek mijn mailbox leeg te zijn en een paar sites die ik host voor friends, family en bedrijven uit de lucht te zijn, very not funny. In de loop van de voormiddag kwam de data al terug in de mailbox en een paar van de sites weer online maar nog altijd niet alles… Alhoewel de support merendeel wel werkt is het natuurlijk weer 9 uur voorsprong op de west-coast mannen dus de 9-to-5 mensen gaan waarschijnlijk een email of 4/5 van ondergetekende in hun support box vinden. Natuurlijk heb ik veel liever dat de plaatselijke nerds zich bezig houden met het fixen van de servers waar mijn bits en bytes op gestokeerd worden maar er mag natuurlijk wel een of andere antwoordmens eens door de mailboxen gaan. Als ze nu ook hun statuspagina eens zouden update, ‘t zou ook al veel beter zijn infeite.. Soit. ‘K komt wel op z’n pootjes. « Semaine [...]
December 18th, 2006 at 9:27 pm
[...] Photo No, this isn’t a random photo. This photo of the Hindenburg disaster was Dreamhost CEO Josh Jones used in reference to server issues in July. [...]
January 3rd, 2007 at 6:53 pm
[...] As those of you who’ve been playing the home game know, we had some troubles in 2006. [...]
January 10th, 2007 at 11:18 am
[...] DreamHost Blog » Anatomy of a(n ongoing) Disaster.. Now that’s community management. I *heart* dreamhost. (tags: community, transparency) [...]
January 16th, 2007 at 3:02 am
[...] Quite surprisingly, customers were treated to a long and detailed post outlining all the apparent issues in August of 2006. The blog post juxtaposed striking images of various real-world disasters with an extremely candid and informative update. The mix of honesty and humour appeared to create a winning PR strategy: 400-odd mainly positive comments including It’s refreshing to see honesty and a company hold their hand up and say they made a mistake. Thank you. Lets hope your building has the same honesty in admitting their faults. [...]
January 17th, 2007 at 4:29 am
Over the past 12 years I have seen every hosting setup imaginable… and for the better part of the last 4 years I have been with dreamhost. During this time I have been locked in R&D, scratching my head as well as various other parts of my anatomy (no, not stroking, Josh!) trying to figure out what the definition of an ‘ethical brand’ is and working towards a solution that will (one day) help everyone to instantly identify and distinguish genuinely ‘good guys’ from… well, you know… all the others, in any market. Of course no branded organisation, or individual [ie: Madonna] is perfect… so ‘being ethical’ means something else and quite frankly, dreamhost epitomises that ’something else’. I don’t believe for a moment they have any particularly special skills or any earth-shattering technology that many, many other organisations don’t have. It’s all in the attitude. Maybe ‘cos they’re employee owned… dunno, but whatever it is they got, the whole friggin’ world needs more of.
January 19th, 2007 at 8:08 pm
If you want to consider a location 52 miliseconds from London, 11 to NYC & 7 to Boston, we have an ultra hardened site — Dataville — that might be a way to avoid your power, cooling and bandwidth problems.
We like your style.
Best wishes, -Anton.
902-482-6466
January 25th, 2007 at 1:53 am
i do not host with you , but continue to be interested in doing so. Your frank approach when you have problems will win you a lot of customers.
February 2nd, 2007 at 2:03 am
Nice blog. Great news!
域名注册虚拟主机
February 12th, 2007 at 3:22 pm
“How about a review of DreamHost?”…
Laughing Man comments:
[I]f you’re accepting requests for new articles, how about a review of your web hosting company DreamHost, like your plan, how many domains you use, pros and cons, and serviceability.
I’ve considered moving my blog to …
February 26th, 2007 at 5:37 am
[...] - Anatomy of a(n ongoing) Disaster. [...]
February 27th, 2007 at 6:02 am
[...] 文中提到2006年的一些麻烦导致 dreamhost 的声誉受损(的确如此，有段时间 yee 的 gt 签名都在诅咒它 -__- )，所以新年 dreamhost 将会赠送更多的空间更大的带宽给用户(可能高达原来的8倍)。 [...]
March 7th, 2007 at 3:14 pm
It is now March 7th yet the service at Dreamhost is getting worse daily.
Still a disaster…
March 15th, 2007 at 6:59 am
[...] wonders of overselling, they had tons of spare processing power, memory, and bandwidth. Aside from datacenter problems, they’ve been pretty [...]
March 31st, 2007 at 7:27 am
[...] Dreamhost, have been for a year and a half. While they do have issues now and then and have had a MAJOR malfunction last year, I don’t know how often other low cost hosts have problems, so I can’t [...]
April 19th, 2007 at 1:08 am
So I think I may be done with Dreamhost. I’ve been sponging my hosting for quite some time from a gracious friend, which is nice. Only problem is the server is one of Dreamhost’s. Which, up until the past few months, had been great. They have fantastic prices, excellent service, nice extras. Mike has found it in his heart to give them a break, and I had as well. But I’m seeing trouble again, whether it be mail or MySQL or just plain network outages, seems like it’s always something. For all the great stuff they’ve tried to do to mitigate these issues and all the damage control and transparency they’ve had, they just haven’t manage to fix the problems
June 6th, 2008 at 3:44 pm
[...] in light of all the recent mis-haps over at Dreamhost, I’ve moved my domain to another hosting provider. Dreamhost is a great company and [...]
December 2nd, 2008 at 7:55 am
[...] weblog: Anatomy of a(n ongoing) Disaster – достойный пример открытости бизнеса. Пожалуй, [...]
December 28th, 2008 at 7:17 pm
Where can I buy diet weight loss pills?