Schadenfreude

July 24, 2007 on 11:30 pm | In Business, Foobars, Insider View, Musings, Tech News by Josh Jones |

I feel SOOOOO bad for them.

Almost exactly a year ago today, DreamHost experienced its last unplanned power outage.

Last ever?

Last ever so far! Who knows what the future holds? (Besides me.)

But for now, I’m just glad the present has been a little better for DreamHost customers than for 365 Main’s!

Because in case you hadn’t heard or noticed, power outages in San Francisco today caused downtime at Craigslist, Technorati, TypePad, LiveJournal, Yelp, RedEnvelope, and more!

San Francisco in August, 2007.

Who here is glad DreamHost is in sunny, safe, earthquake, mudslide, forest fire, riot, tsunami-free, Los Angeles now? And who here is publicly enjoying that 365 Main is not?

Here’s a big hint: he’s really good looking and wrote this post.

Of course, the real reason we had no problems is not because our data center is finally super reliable, or that Los Angeles itself never has so much as a cloudy day, or even that we’re just lucky.

It’s because I am in Chicago at HostingCon and so am temporarily unable to break anything.

Of course, that’s not really true either. I’m not in Chicago; as everyone knows, I’m a compulsive liar. In fact, this statement is a lie.

But, even if I was at hosting con (and everybody knows we don’t go to hosting conventions), my ability to break DreamHost systems knows no boundary of time or space, and strikes at any time, usually without warning and definitely without mercy.

Why were we were spared this time?

The honest truth is that any data center can, at any time and for any reason, no matter what precautions they take, have an outage! You’d think making a reliable data center would be a lot easier than making a reliable software service, seeing as how it’s all just power cables, air conditioning, and gasoline.

And yet somehow, it seems like all even the best and most expensive data centers can do is make the outages a little less frequent.

"Jem" is for "Josh: Everybodys Master".

What IS a poor host to do?

Nothing, really.

I mean, the only way you can really achieve “five nines” uptime is by having an entire architecture designed around the assumption that ANYTHING can fail… and at the worst possible time. Duh.

However, like most Las Vegas escorts, that sort of redundancy does not come easily. Or cheap. And the truth of the matter is unless you’re Google, most likely an entire day of downtime once a year is not going to cost you as much as it would to truly prevent it.

In fact, I wish there were some low-reliability data centers out there! I bet if somebody made an ultra low-cost data center, one that provided “adequate” cooling, network, and power capacity, but no UPS, fire-suppression, generators, crazy physical security, or extra earthquake protection, they would clean UP.

They could probably charge around half of any data center I’ve ever seen, and I bet with only twice the downtime… and that would be very appealing.

I mean, think about it… how many of you could deal with an extra day of downtime per year for half the price? Heck, you’d probably be fine with FOUR days of downtime a year if it meant 75% off.. but would you pay double to save 12 hours of downtime a year? Would you pay FOUR times as much to save 18? Eight times as much to save 21?

That’s pretty much how it works, and I’m guessing not a lot of you would.

Of course, maybe I’m over-estimating the cost savings of skimping on redundancy in a data center a little, and maybe I’m under-estimating the reliability hit a tiny bit. On the other hand, my blog posts have never been wrong before.

The more wires, the more porn.

AND, if somebody did come out with a “Crap-of-the-Art” data center, it’d make it a lot more feasible for those who really need reliability to get two; thereby keeping all their company’s eggs out of one risqué basket.

In fact, what we’ve been doing over the last year is breaking our system down into smaller and smaller isolated “clusters,” and distributing them between three data centers (all in LA). The idea being, data centers will go down.. let’s at least try and keep the eggs in our other baskets un-scrambled. And since we’re not really counting on much reliability from them anyway, it sure would be nice if those data centers all charged a lot less!

Of course our network still has a single (though redundant) point of failure, but we are working towards eventually making each data center a complete stand-alone “node”… some day.

This day, however, I think I’ll just go to bed… while taking pleasure in the fact that it was somebody else this time!

21 Responses to “Schadenfreude”

  1. Cailin Coilleach Says:
    July 24th, 2007 at 11:59 pm

    Well, as you can see from both your ticketing system and your status blog: some of your customers were affected by the outage. Not in a direct way, but because the trans-Atlantic network company Global Crossing (gblx.com) had some troubles. This provided a certain percentage of European folks with the disability to reach your and their servers :)

    To answer your question about whether customers would accept double the downtime for half the cost? I don’t know… Seriously, everybody simply expects “The Internet” to be available 60×60x24×7. If a site’s down for a day a lot of people may stop visiting the site entirely, thinking they’ve gone down completely. Hey, it takes all sorts…

    Lemme tell you: it sucked, having to miss my direct access to Wrangler for over twelve hours ^_^

  2. Steve Says:
    July 25th, 2007 at 2:23 am

    Well, having power to my server is one think, having a server that can actually serve my website is another !
    I’m sooo sick of the high load on my server (50+) that I’m moving on.
    Please don’t brag about not having a power failure for a year when so many of your servers are so overloaded that they’re not responsive anyway …

  3. Tom Says:
    July 25th, 2007 at 4:07 am

    Misery loves company… so to speak. The benefit of hosting in the same data center as all the big boys is if they all go down, no one is going to be doing much surfing on the internet anyways, right? At least that’s what you can tell your customers. Your cash register might not agree with this perspective, however.

  4. David Says:
    July 25th, 2007 at 5:27 am

    LOL, that prediction in the last blog post didn’t last long.

    Will the DreamHost Blog ever have another hosting-related blog post again?

    Reply Hazy, Try Again

    Will the DreamHost Blog ever have another hosting-related blog post again?

    Better Not Tell You Now

    Will the DreamHost Blog ever have another hosting-related blog post again?

    Concentrate and Ask Again

    WILL THE DREAMHOST BLOG EVER HAVE ANOTHER HOSTING-RELATED BLOG POST AGAIN?

    Don’t Count On It.

  5. Tim Says:
    July 25th, 2007 at 9:46 am

    >> “Of course our network still has a single (though redundant) point of failure”

    Now that might be the best quote I have read all year!

  6. Dino Says:
    July 25th, 2007 at 9:55 am

    I think you are much better than last year. The uptime stat is getting better and better. Good works !

  7. Dreamhost tickles me: AND, if somebody did come out with a “Crap-of-the-Art” data center… - Justinsomnia Says:
    July 25th, 2007 at 10:04 am

    [...] tickles me: AND, if somebody did come out with a “Crap-of-the-Art” data center… Trackback Wednesday, July 25, 2007 [...]

  8. Unofficial DreamHost Blog Says:
    July 25th, 2007 at 12:20 pm

    Ars Technica used DreamHost to serve iPhone video…

    The insanely popular blog Ars Technica (the 7th most popular blog in the world according to Technorati) recently did an iPhone review including a 5 minute video with their attempts to destroy the phone (sound like something Josh would have done).
    The i…

  9. Mike Says:
    July 25th, 2007 at 12:36 pm

    Why not have at least one data center outside of LA?

  10. Matt Says:
    July 25th, 2007 at 12:49 pm

    Seriously Josh, if you use one more bolded word I am going to hunt you down and something you.

  11. Josh Jones Says:
    July 25th, 2007 at 12:55 pm

    Uh oh.

  12. Larry Says:
    July 25th, 2007 at 1:18 pm

    The problem was that 365 Main angered the “hosting gods” by putting out a press release touting 2 years of 100% uptime for one of its customers:

    http://www.prnewswire.com/cgi-bin/stories.pl?ACCT=104&STORY=/www/story/07-24-2007/0004631109&EDATE=

  13. Dave O'Flynn Says:
    July 25th, 2007 at 8:39 pm

    Good-enough-but-way-cheaper is why I use dreamhost.com instead of, say, pair.com.

    A friend has had a total of 11min of downtime with pair in the last seven years, but his monthly bill is somewhere around half of what I pay for a year’s hosting.

    It’s a trade-off I’m comfortable making…

    …and he’s considering the switch too :-)

  14. ChiaLynn Says:
    July 27th, 2007 at 12:02 pm

    And this seems as good a place and time as any point out that it’s Sysadmin Day, and thank you all for everything you do to keep my sites up and running far more often than not.

    http://www.sysadminday.com

  15. Xopl Says:
    July 27th, 2007 at 12:15 pm

    I would like just the opposite of what you are proposing. Offer more expensive plans with uptimes guaranteed in writing.

  16. Jim Says:
    July 27th, 2007 at 4:23 pm

    Happy SysAdmin day!

  17. HansNL Says:
    July 29th, 2007 at 4:01 pm

    When is the data center in Europe coming along?
    That you wanna live in an earthquake zone
    is fine with me but not with my data.
    Cheers, HansNL

  18. Wouldn't Work for DH Says:
    July 31st, 2007 at 9:31 am

    This is a concept that is working well for me. I used to have several managed servers with RackSpace. Way too expensive and I had a server crash that took them 12 hours to fix. For the rates they charge, that was unacceptable.

    Now I have a T1 line at my office. I have replaced my two managed servers with four (they aren’t as powerful) and haven’t had more than 30 seconds of downtime in the six months since I started this.

    If/when my bandwith requirements exceed the capabilities of my T1, I would love a data center like this.

    This is a fine and dandy for someone like me who needs a few dedicated servers of our own but this wouldn’t work for DH. Your rates are already ridiculously cheap and I suspect people would rather pay what they pay now for the same service than have more downtime.

  19. peter Says:
    July 31st, 2007 at 12:38 pm

    well the power might not have gone out on my account but my site has been more or less unreachable for WEEKS now and all I get is a ‘we are working on it’

  20. Tiuca Says:
    September 10th, 2007 at 6:23 pm

    Interesting datacenter with interesting connections !

  21. Side effects of stopping zoloft. Says:
    May 14th, 2008 at 3:08 am

    Zoloft side affects….

    Zoloft sideeffects. Zoloft. Secret side effects of zoloft….

Leave a Reply

Powered by WordPress. Pool theme by Borja Fernandez, modified by DreamHost.
Entries and comments feeds. ^Top^