Anatomy of a(n ongoing) Disaster..
August 1, 2006 on 12:29 pm | In Foobars, Insider View, Updates by Josh Jones |
What a three weeks…
As I’m sure most of you already know, we’ve had nothing but troubles, large troubles, for pretty much the last three weeks. A lot of these troubles were our fault, a couple of them were at least ostensibly beyond our control, and they all compounded each other.
Here I’ll try and go into as much detail as possible about what happened, why, and the steps we’re taking to stop this sort of thing from ever happening again. I can’t excuse what happened, just apologize and hopefully elucidate.
Ironically, all the recent disasters stem somewhat from us attempting to take some proactive steps to head off any sort of future power outages like the kind we experienced last year.
The Back Story
As some of you may know, we are co-located with Switch and Data in The Garland Building in downtown L.A. To say we’re co-located is a bit misleading though, since we’re now basically 95% of their data center.
Why don’t we have our own data center?
Because, believe it or not, we’re still not big enough for it to make sense. Even now, we only use about 1000 sq ft of data center space.. for it to really start to make sense to get our own space, we’d have to be using around 2500 sq ft. Mainly because when you buy a data center, you want to get one big enough to handle a lot of growth.. and although it’s cheaper per square foot than co-locating, you have to pay for all the space you’re not using yet.
And really, The Garland Building is supposed to be an excellent place for data centers. There are more than a dozen in the building. Companies like iPowerWeb, Media Temple, BroadSpire, and even MySpace (now the most popular website in the whole US!) are in there. It’s got FIVE huge generators, UPS for the whole building, on two separate power grids, and a dedicated engineering staff to make it all work flawlessly. Or so we were all assured.
Around last June though, the building informed all its data center tenants that they had essentially run out of power! Not power altogether, but the “good” power that data centers need.. i.e. ups and generator-backed power. Because Wells Fargo, who holds the master lease on the building, wasn’t sure if they were going to renew the lease when it is up in three years, they didn’t want to invest the millions of dollars to add more generators and ups to increase capacity. This is in fact the primary reason we’re still not selling any more dedicated servers .. they use too much power per dollar!
Of course, none of that was supposed to have any affect on their ability to keep the current power going in the case of an outage. September 12th, 2005 we discovered they actually couldn’t… when two of the five generators failed!
However, since then, the building has repaired and replaced the faulty generators, and given all their tenants numerous assurances that what happened before would never ever happen again.
Why didn’t we move data centers right then?
That would have been a fairly massive undertaking, resulted in even more down time, been very expensive, and actually we did look around and there weren’t any really good options for moving… data center space is becoming pretty tight (in the LA area at least) and the Garland Building is still one of the best options, believe it or not. Also, this was the first time something like this had ever happened, and it seemed pretty reasonable that it wouldn’t happen again. We even asked around and none of those other tenants mentioned above were moving, so I guess it seemed like people were generally pretty confident it was a one-time freak occurrence.
Nevertheless, we started making contingency plans, searching around for another data center that had some power and would make sense for us. Eventually, we found Alchemy, just down the hall from S+D actually, and began making arrangements for getting some space from them. They had a little bit of power available because they were moving some of their clients out to El Segundo, and because they had gotten permission from the building to install their own generator. With that generator and some UPSes they were able to convert a “dirty” power feed into “clean” (i.e. good for data center use) power.
How the troubles began.
All this took a very, very, very long time. After months of searching and negotiating with Alchemy, we still had to get Switch and Data to allow us to put a cross-connect in from their data center over to their competitors down the hall. After even more months and teeth-pulling, we finally got that up and running. In fact, we finally got the first live server up in Alchemy a little less than a month ago.
All this in an attempt to head off future power problems.
Unfortunately, shortly after setting up the new footprint, we noticed something wasn’t right. Getting to Alchemy from Switch and Data we would lose huge buckets of packets. Just as we were trying to figure out the problem, we started to have problems with one of our file servers.
This resulted in a lot of problems across the board. The web servers that mounted that filer all had problems. The mail servers that mounted that filer all had problems. In fact, one of the mail servers was mis-configured and was logging thousands of errors a second to a remote logging machine… so many in fact that it was saturating its switch and clogging up a whole chunk of our network. Which in turn caused other machines to get slow and crashy because they couldn’t get to their filers, and so on and so on.
It turned out the filer problem seemed to stem from the fact that we had one shelf of 300GB disks and one shelf of 150GB disks on it. Apparently they’re not supposed to be able to support this, or at least it’s a bad idea. So, this was entirely our fault. However, we did have a number of other filers we did this on, and we’d never had problems before. Nonetheless, we will never mix disk shelf types on a file server again.
We eventually cleared all this up.
However, the Alchemy connection problems were still ongoing.
After trying all sorts of things, we eventually decided to replace one of our distribution switches that was acting strangely with a new one. This didn’t really seem to fix the problem either. This was on Friday, July 21st.
On Saturday, July 22nd, the building lost power.
This time, the generators actually worked, but the UPS failed! Honestly, it was much better than last year’s.. but unfortunately, even a brief power outage wreaks havoc on a data center. And this one wasn’t so brief.. here’s the building’s explanation:
At around 5:21pm, on Saturday July 22nd, a brown out occurred due to record high temperatures in downtown Los Angeles. Voltage dropped due to the high demand of electrical current along with equipment failure operated by the Department of Water and Power, City of Los Angeles. This condition caused failure of “ATS-B” switch and to UPS Module #3. Engineering crews were dispatched and began repair of this damaged equipment. A power interruption was required to replace contacts in “ATS-B”.
Repair of “ATS-B” failed contacts was completed on 7-24-06. Power was restored between 4:00am and 4:30am by the Engineering department.
Thank you,
Office of the Building
So, after all the emergency filer stuff going on the previous weekend, just about the entire admin team was back last weekend, working on getting everything back up when power came back on. Even when we had power, it was in a degraded state and so the cooling wasn’t working. As temperatures rose, file servers automatically shut themselves down rather than risk being damaged by the hostile environment. Apparently, MySpace made the decision to just keep all their servers off until cooling was restored.
More network troubles..
After the power outage, we decided to just yank everything back out of Alchemy (they lost power too!) until we could figure out what was going on with the network to there. Unfortunately, this didn’t seem to fix things, and our internal (”red”) network was still really fubar. When our red network isn’t working, the panel isn’t working, webmail isn’t working, and our server configuration system starts having problems (basically, anything that connects to our internal databases).
It took us just about all of Monday to figure out (and then fix) that a lot of the file servers had bad routes after being powercycled.. and so were sending ALL their traffic through the red network, saturating it. These things are generally pretty stable and a lot hadn’t been rebooted since September 12th, 2005.. and some had apparently had their networking set up by hand instead of correctly configured via our database. We’re making sure that doesn’t happen anymore either.
More network troubles..
Once that was fixed, things generally got better. Except there was STILL strange stuff going on (causing slowness and high loads around the system, but not an actual system-wide outage), even without NFS traffic going through red, and even without anything at Alchemy. It started to look like there was a problem with one of our core routers. We called our Cisco consultant and opened a trouble ticket with Cisco themselves..
More power problems..
On Friday, July 28th, we lost power again. The building wrote:
The Garland Building experienced a dead short which resulted in a brief power outage today, July 28, 2006. The air conditioning, elevators, and the electrical utility have all been restored.
While on generator power, a dead short occurred from one of our internal telecom users. We are investigating where the dead short occurred. A follow-up memo will be sent by the end of the business day reconfirming our transfer at 11:30pm tonight. We are currently on DWP power until further notice.
And then:
The Garland Building UPS System is back on-line supplied by DWP. Diesel generators have returned to an on-call status.
The 11:30pm transfer has been cancelled due to the dead short prematurely returning us to utility power. At 4:30pm the engineers engaged the UPS System to protect all tenants at the Garland Building.
Thank you,
Office of the Building
This time, we were able to get our entire system back up much quicker and with close to no problems. Of course, it had been less than a week since our last power outage.
Alchemy was the only data center in the building who did not lose power this time.
More network troubles..
Over the weekend (this last weekend), we kept having the same ongoing weird network problems I mentioned above, and Cisco hasn’t made much progress. Yesterday, we realized the new distribution switch (an extreme) was causing spanning tree problems with the older Ciscos. Jeremy got it all figured out, but in the process it erroneously blocked our “green” (public!) network for a few brief periods, taking down everything again.
Unfortunately, that fix STILL doesn’t seem to have fixed the ongoing core network problems. We were finally able to get our tickets escalated with Cisco yesterday. It is starting to look like something may have been damaged during the first power failure, although we’re not sure. The replacement/repair cost might be around $80,000 it looks like.
And that’s where things stand today.
Our number one priority right now is getting this nagging network problem understood and fixed. Once that’s the case, we should be able to put things back in Alchemy, who didn’t lose power on Friday at least. Once things are going good there, we’ll be able to add new servers and transition old ones slowly with little to no downtime.
We’re also going to be buying our own UPSes, since we have learned we can’t trust our data center OR our building to do it. We’ll start by putting the core routers on them, then our internal databases and servers, then our file servers, and finally the hundreds of customer mail, web, and database servers.
Finally…
We’re very sorry for what happened. We definitely don’t want it to happen again, and we’re trying to take all the practical steps we can to prevent it. We never want to have another July 2006 again.
Ironically, some of the network problems seem to have stemmed from us trying to better protect ourselves from power failures. I also want to say for the record that none of these problems in my opinion stemmed from “overselling”. Rather, I’d say it’s the result of bad luck. And incompetence on our (and the building’s) part.
I don’t know if we’ll be able to change our luck, but hopefully we’ve at least learned something and will be able to become a tiny bit less incompetent in the future.
I hope you’ll all stay with us to find out.
476 Comments
Pages: [1] 2 3 4 5 6 7 8 9 10 » Show All
Pages: [1] 2 3 4 5 6 7 8 9 10 » Show All
Sorry, the comment form is closed at this time.
Categories:
Archives:
- May 2008 (2)
- April 2008 (2)
- March 2008 (3)
- February 2008 (5)
- January 2008 (7)
- December 2007 (3)
- November 2007 (3)
- October 2007 (5)
- September 2007 (3)
- August 2007 (4)
- July 2007 (4)
- June 2007 (7)
- May 2007 (3)
- April 2007 (3)
- March 2007 (8)
- February 2007 (4)
- January 2007 (6)
- December 2006 (5)
- November 2006 (2)
- October 2006 (4)
- September 2006 (4)
- August 2006 (5)
- July 2006 (3)
- June 2006 (3)
- May 2006 (5)
- April 2006 (6)
- March 2006 (7)
- February 2006 (6)
- January 2006 (8)
- December 2005 (7)
- November 2005 (5)
- October 2005 (8)
- September 2005 (19)
- August 2005 (11)
- July 2005 (4)
Other Sites
- DreamHost Status - An off-network page for all DreamHost Announcements! (Not as silly as this blog.)
- DreamHost! - The host behind the blog.
- Forum - Discuss DreamHost and web hosting!
- Web Control Panel - Manage your account.
- Wiki - On DreamHost and web hosting.
Powered by WordPress. Pool theme by Borja Fernandez, modified by DreamHost.
Entries and comments feeds.
^Top^
DreamHost Power Outage…
Downtown Los Angeles was hit by another power outage last Friday. This affected DreamHost’s services and other tenants at the Garland Building (here among MySpace), because the UPS system failed.
DreamHost was affected by a similar incident last …
Trackback by Unofficial DreamHost Blog — August 1, 2006 #
thank you for explaining what happened over the past few days. these things happen, and you seem to be dealing with it.
some peole might say these things shouldn’t happen, and in a perfect world, they wouldn’t. but this world is far from perfect, and as long as you guys (and gals) are trying, that’s fine by me.
thanks again.
Comment by vince — August 1, 2006 #
It’s refreshing to see honesty and a company hold their hand up and say they made a mistake. Thank you.
Lets hope your building has the same honesty in admitting their faults.
Comment by Chris — August 1, 2006 #
ditto. thanks for the good post!
Comment by simplifiedmedia — August 1, 2006 #
Good to see the open and clear explanations. Thanks.
Glad it’s you running all this. ;-)
Comment by Neil — August 1, 2006 #
Josh, will the space with Alchemy allow you to start offering dedicated server again?
Comment by Will — August 1, 2006 #
Dedicated servers at dreamhost, ya!!!!
From,
Adam
Comment by Adam — August 1, 2006 #
I’m not going anywhere, and this blog post is a big reason why. Thanks for explaining what’s going on and being up front about everything. May your cluetrain never crash.
Comment by Kevin — August 1, 2006 #
I’m still here.. :)
Comment by pixiepurls — August 1, 2006 #
I am a sysadmin/networkadmin/etc.. these things happen. Usually they happen all at once making it seem like we are entirely incompetent, but in reality, it’s chaos taking over - strange attractors and all. Point being, for a year I experienced relatively little downtime, and all at once a bunch. In no way does that cancel out the year. I am staying here for now. - Yossie
P.s. And THANKS for the explanation - knowledge is power, and I feel a bit more powerful today :)
Comment by Joseph Silverman — August 1, 2006 #
I was about to make the switch to DreamHost last September when you had a major power outage. I monitored your status page and blog and decided that I would give you a try anyway. I’m glad I did and so are my clients. Your honesty about what is going on is definitely appreciated by me.
Comment by Karen of Scottsdale — August 1, 2006 #
thanks for keeping your customers in the loop. even with all the problems that you all have had, you’re still the best webhost that i’ve worked with.
thanks!!
Comment by ryan — August 1, 2006 #
Use one vendor for all your switches. Any other way lies pain.
Comment by cj — August 1, 2006 #
Posts like this are the reason that I love dreamhost and continue to pimp you guys out to all my friends.
Comment by Rachel — August 1, 2006 #
Great post :D… thanks for the explanation!
We only joined under a week ago, but since then have been astounded by the level of honesty and customer care you guys provide. They ought to learn a few things from you over here in the UK!!
Keep trying to fix the problems, and try not to beat yourselves up too much over this period of bad luck and trial and error. We’re not moving anywhere for the foreseeable future.
Tom :)
Comment by Thomas Hine — August 1, 2006 #
Thank you so much for putting all this out in the open! I feel hugely better now.
Comment by Daniel Drucker — August 1, 2006 #
The right way to do business…
I have been a Dreamhost customer since 1999, and I only recall one significant outage over that entire period. Until the last couple of weeks. Things have definately gotten pretty ugly. DreamHost Blog ｻ Anatomy of a(n ongoing) Disaster.. One……
Trackback by Fingertoe.com — August 1, 2006 #
We use DH for non-critical stuff, like our internal company wiki and email routing. It sounds like a number of problems were brought about by the building’s inability to provide good power.
My company has looked at many colo facilities, and I was very impressed by Equinix. When I took a tour of one of their facilties, I could see many big names (Google, Amazon) located there, and it seems like they know how to run a facility. They have 3 locations in the LA area — I’d recommend looking at moving there.
The fact that MySpace is in your building doesn’t mean much. They probably never expected to grow as much as they did, and are too busy dealing with that to focus on their datacenter situation. They started small and haven’t had time to evaluate if their current facility is appropriate for the level of uptime they require.
Comment by Justin — August 1, 2006 #
Hey, if the worst thing that can happen is Dreamhost owning up to their mistakes and promising to do better, I see no reason at all to even consider switching. The federal government should be so honest!
Comment by Patrick Lee — August 1, 2006 #
I’m not a DreamHost customer, but I just wanted to come here and leave a comment to say I admire the way you’re owning up to the problems that have been occurring recently. Laying everything out in the open for everyone to see is the way business should be done!
Comment by RisingSunofNihon — August 1, 2006 #
I just wanted to say how refreshing it is to get such honest information these days.
Thanks and all the best for the future.
Comment by Andrew B — August 1, 2006 #
I still love you guys.
Comment by Eric — August 1, 2006 #
Ah, good… it sounds like you are almost ready to have a local navvy put a shovel through the fibre outside the building…
Comment by Yada yada — August 1, 2006 #
I have to say that without this post, I’d be outta here. Thankfully, you guys sent out the newsletter with a pointer to this. Well done!
I subscribe to the emergency status feed and I was beginning to wonder if you had just started reporting things that had not been reported before (but happening), or if you had a new person that got chatty about emergency stuff.
Glad to hear this was an anomaly. I’ll stick with you.
Again, very well done on being so upfront about the issues; but do please try your best to prevent it (I’m sure you will).
Comment by Bob Monsour — August 1, 2006 #
Thanks for the updates and thet honest work. It’s appreciated and understood. Thanks again.
Comment by brian — August 1, 2006 #
Thanks for being so up front with all of us. It’s rare to find a company who is willing to own up to their mistakes and be honest with customers. I know I appreciate it, and I’m sure most of your other customers do, too.
I’ve been recommending Dreamhost to all of my friends for years, and I’m going to continue to do so.
Keep up the great work.
Comment by Tyler Hall — August 1, 2006 #
I too appreciate the update.
I’ve been with DH for ~6 years now and have no intentions of leaving, but I would expect in this context to see something of a bone thrown to your customers. We recently experienced some downtime on Puzzle Pirates and ended up crediting subscribers with four days of free play. The downtime was pretty minimal, measured in minutes rather than hours, but intermittant and annoying, so we ‘paid up’ to our players. I would respectfully suggest doing something similar.
Comment by Daniel James — August 1, 2006 #
Well, despite it all, I still love you guys. And I try to direct as many people as I can to you, too. That way, maybe you can get that fancy data center (or not).
Keep up the good work….
Nancy
Comment by Nancy — August 1, 2006 #
Thanks for being up front throughout this ordeal. I’m hoping that August goes better for you guys. Thanks!
Comment by Bocce Ball Rules — August 1, 2006 #
Garland Building en Los Angeles
http://flickr.com/photos/alvy/42973478/
Comment by George — August 1, 2006 #
Thanks for the explanation. After four years I’m not planning on going anywhere, but you could include some more information on the customer service issues that have come to light in this. All the problems felt were not technical, and the inaccurate and untimely information being put out was at least as frustrating as the outages themselves to me.
Comment by Chuck — August 1, 2006 #
You guys rock! I love the catostrophe pictures strewn about. Really conveys the mood :)
I used to work for PowWeb (before it was sold), who was also hosted at the Garland Building, so I hear where you guys are coming from.
This past three weeks has been rough on me with the outages, just like Im sure it has been for many customers, but knowing that you guys definitely dont WANT this to happen, and are constantly trying to make it better goes a long way.
Anyways, brighter days are ahead, and I know I am rooting for you.
Comment by Nate Cavanaugh — August 1, 2006 #
DreamHost…
I admire honesty where I see it. Too much of the world is built on lies or at the very least deception. Recently, DreamHost (the company that hosts DigiFiend and a couple of other sites I manage) has been having……
Trackback by DigiFiend — August 1, 2006 #
go to hell, dreamhost. ive been with you 7 years and the service has been downhill since i first got on. the cutesy “family and friends” tone isn’t cutting it anymore (unless you’re completely retarded), we’re all pretty aware of the dough being raked in. i can’t believe i’m getting lame copy/paste apology responses instead of credits to my account in some form or another. i pay out my nose for you (in comparison to an equally apt host like 1and1), have referred at least 10 people to you through the years, and keep operating under this illusion that i’m getting the “best” of customer service and features. prove me right, and cut out the cuteness. focus on service and re-imbursing your customers for gigantic boners like this. i don’t care what happened to your facilities, guess what, i have problems in my life, too. do your job as a supplier and amend the situation, and credit us ALL SOMETHING for the bullshit we had to go through for this pathetic display.
Comment by hp — August 1, 2006 #
Jeesh. Comp time for everyone!
Comment by bmwracer3 — August 1, 2006 #
It’s great to see such a detailed account of what happened. That’s why I’m still a customer.
Kind of off-topic, but: Why do internet companies constantly choose to locate in LA, which has chronic power problems in the summer? Why not Dallas, Atlanta, or Richmond? There are tons of other cities with great infrastructure, cheap land, adequate power/no brownouts, and a skilled labor force. But for some reason, LA is chosen despite its lack of adequate power during the summer. I don’t really understand that.
Comment by bwd — August 1, 2006 #
Thank you for the status report, I’m once again pleased I switched to DH earlier this year, excellent customer service and an entertaining newsletter! What more could we want?
Comment by Boldman UK — August 1, 2006 #
Thanks so much for taking the time to give us a status report. I won’t be moving hosting companies anytime soon. You guys really rock in terms of customer support.
Comment by Andrew M. Lin — August 1, 2006 #
I appreciate that a company owns up to its mistakes. Also I have to note that customer support was top-notch during these incidents. While I was a bit pissed off during the occurences, this explanation brings back the warm and fuzzies.
Comment by John — August 1, 2006 #
I have been with Dreamhost for about 4 years now and have recomended Dreamhost to around 20 others and I am happy that I have. Has all of this been hard on us? Yes. But LIFE DOES HAPPEN! Over all I have been VERY happy with Dreamhost and have watched them gorw (and learn) and feel like this will only help them come out a little stronger! Thanks for telling us all of this so that we really can understand what happened! That is all that I ask for!
Comment by Marie Duncan — August 1, 2006 #
Thank you guys. Your customer service is amazing.
4 years and counting.
Comment by Arturo Rojas — August 1, 2006 #
I’d say it’s time to hire a dedicated network engineer. You mention “consultant” in the article.
Comment by cj — August 1, 2006 #
The hosting solution i used before you guys recently had a bunch of problems, some of which stemmed from the terrible flooding in Louisiana. So I could forgive them for the circumstances, but i could not forgive them for the terrible level of communication during and after the initial problems.
Your forthrightness and honesty are absolutely golden, and you can count on me sticking with you guys through the trying times.
keep up the good work!
Comment by Jesse — August 1, 2006 #
Thanks for this write-up; we will definitely be staying with DreamHost. This is the kind of status report that keeps customers happy — well done!
Comment by David M. — August 1, 2006 #
[...] How do you treat your customers when you let them down? DreamHost did about as good a job as I can imagine after they were down during a power outage that they couldn’t control. Read the comments where customers are coming back — corporate blogging done right. Thanks to Dylan Bennett for emailing me this. [...]
Pingback by Crazy Factor » Bad hosting day blog — August 1, 2006 #
Without this post I would seriously consider moving. In fact I even hosted one site at Godaddy.com in anticipation of moving, however, I will stay for now. Personally, I think you should at least offer your customers some sort of compensation due to the lack of planning. speaking of which:
I live in Florida, I hosted with DH in CALIFORNIA. Why did I do this? HURRICANES. I know that every year hurricanes are going to take out a large part of Floridas infrastructure. I always laugh my ass off at the idiots driving hummers and mercedes to water and ice lines in the heat while I drive my beat up Saturn around and go home to hot water and a cold refrigerator sitting on my Air-conditioned ass Watching DVD’s on the projector. I bought a generator rather than a hummer. It just made mroe sense. Same thing with a gun. my mom always bugs me about owning a worthless gun I never shoot. But if someone ever attacks my family at home, I can protect my loved ones. And yes I keep it LOADED and COCKED and yes the kids know what it is and where it is and how to handle it. That way when they see one they will know how to handle it rather than blowing thier damn friends head off.
If you prepare for the worst you will be prepared for the best. Perhaps DH should consider hiring some BOY SCOUTS to help prevent problems rather than putting the flames out when it catches fire.
All in all, If i had been at DH when all the catastrophy happened I would be absolutely beside myself. Im sure you gues have been sitting around after a hard day at work and the lights go out all at once and someone said “OH YOU HAVE GOT TO BE FUCKING KIDDING ME!!” I worked for a company once that fired me after I told them that the worst can and would happen. They fired me right after I set up the generator plans and the day was set for it to be installed. The funny thing was that the generator was partially installed and all the parts were not available the day they lost power for a week.
They called me and being the nice guy I suggested they send the guy who replaced me to the Home Depot to buy every generator they had. Idiot forgot the extension cords. Anyways, good luck, and godspeed and please heed some advice. IF IT IS GOING TO HAPPEN TO ANYONE IT WILL HAPPEN TO ME. Say that everyday when you wake up and you will never get surprised.
Comment by Phillipx — August 1, 2006 #
Great job being so honest guys. I’m really happy that I switched to your service after a horrible experience with C I Host… who still owes me $40.
Comment by Pete Gilligan — August 1, 2006 #
You guys are honest, transparent, and rock our collective hosting world. I have been in your shoes (in NY, not LA) so I understand exactly what you mean — I found myself nodding through most of your post and I even had a flashback or two at various points. Like most of the customers here, I will use this as only further reason to sing your praises loud and long and to recruit more customers for you.
I’m glad you’re putting in your own UPSes. I second the earlier recommendation for a single vendor for your network fabric (I know, the last thing you want is a backseat sysadmin, but think of this as a friendly comment, not a whine), and on top of that, a consistent OS release on those network devices. (Yes, that’s the scar of an IOS version burn showing through.)
I’ll admit if I was spending $1000/mo on hosting with you I’d be expecting some kind of rebate… but then, it’d be in the SLA I’d signed with you. So if you felt like giving us all a free month I wouldn’t argue, but it sounds like you may have six figures of unexpected hardware expenses about to hit, so I understand if that precludes giving each small customer some kind of token of apology. Hopefully others will understand, too.
Thank you, Josh. Tell the rest of the team thanks, too. Best of luck in the coming months.
Comment by Scott Swanson — August 1, 2006 #
I really appreciate you guys! The downtime has not been much of an impact, but what HAS been an impact has been being able to go to “dreamhoststatus.com” and find out IMMEDIATELY what the problem was. My last hosting company would basically lie thru their teeth, and I appreciate that you don’t. The pictures made me die laughing, by the way… :)
Comment by Jonathan Feldman — August 1, 2006 #
I switched my company’s hosting over to DreamHost in March after our previous host had continued downtimes with no explanation or support. While I’ve experienced problems on DreamHost, they’ve always been taking care of quickly, friendly, and with an explanation.
Power outages happen. You don’t own the building, so you can’t ensure the generators and UPSes function correctly. Things happen. I understand.
Thanks for the explanation. Even without it, DH support and the features I am given keeps me here. I’m proud to work with DreamHost, and wholeheartedly recommend them to anyone who needs a host.
/dan mattia
Comment by Dan Mattia — August 1, 2006 #