Good Reminiscing Friday

March 21, 2008 on 6:09 pm | In Foobars, Insider View, Updates by Josh Jones | 80 Comments

Those were the days!

Well, it was a little over two months ago that we had what I think is pretty safe to call the worst disaster in DreamHost history.

In retrospect to me, it’s kind of funny that the worst disaster didn’t turn out to be due to a security breach, a power outage, a loss of data, or actually anything related to our actual hosting service. I guess it shouldn’t be a surprise that people care a lot more about their bank accounts than they do their websites.

I have realized that billing is the one issue where how important we feel it is is completely at odds with how important you guys feel it is.

What I’m trying to say is, we’ve always been ultra-flexible and lax about how people pay, when people pay, or even about giving credits, discounts, or refunds. We figure, whatever, pay us when you’re ready, we’re not sending anybody to collections or ruining anybody’s credit over some measly bandwidth bill.

If everybody had just been paying by check!

What we’ve always tried to focus on more (even though it might not seem like it at times!) is our hosting system’s stability, performance, and features.

I guess I’ve always figured that any billing-related error can be easily undone (worst case scenario, it costs us a little money); there is no lasting harm done to the customer. Whereas having a website or email problem could potentially cause permanent damage to somebody’s business or personal life or something?

Well then, let’s go back and see just how little money a worst case scenario actually costs, shall we?

Credits and refunds to cover people’s bank fees: $52,000.

Sigh, if only everybody kept a big cushion of cash in their account! The main damage that can be caused by a billing snafu is for people who get their account overdrawn, and because of that aren’t able to make a critical purchase, or have a check bounce, causing hassles and incurring bank fees. We offered to pay people any amount their bank charged them for going negative, and in the end that total looks like it came to about $52,000.

Discover how much money I lost DreamHost!

Accidental refunds: $170,000.

The worst part of this whole process (for us) turned out to be just after the accidental billing, ironically when we were trying to make things right!

If you recall, our system was not actually charging about 75% of the time we thought it did.. and so we refunded thousands of people who were never charged (but, 75% of the refunds didn’t work either). Well, out of all that, and after two months, there are still about 600 accounts who were credited a total of $170,000 in excess of what we charged them that we haven’t been able to get back from them or their bank.

It is slightly annoying when the same guy who complains to the high heavens when he thought he’d been over-charged $9,000 by accident conveniently disappears when we realize that actually, he’s been over-refunded $9,000 by accident.

Extra credit card fees: $82,000.

Another slightly annoying thing is that credit card processors don’t credit you back any fees when you refund a transaction. Overall, the extra credit card processing we did resulted in extra fees of about $350,000! Fortunately, after a whole lot of groveling and explaining the situation (and waiting two months), we finally got all but $82,000 of that back from First Data, American Express, and Discover Card.

Apparently our snafu didn't screw up Visa's IPO too badly.

Extra support messages: 20,000.

As you may have surmised, people wrote to us about this thing. About 20,000 times… and it would have been tens of thousands more if we hadn’t put up an “emergency block” against new messages for a little while in there.

How much this extra support actually cost (in terms of your wased time, tech support overtime pay, and other questions taking longer to answer to) is hard to say, but normally we only get about 45,000 messages in a whole month!

Accounts canceled: 1000.

It’s also kind of hard to say how many people actually closed their account because of the incident, but in January we did have about 1,000 more accounts closed than average. Assuming each of those accounts would have stayed for maybe another year, that’s another $120,000 down the Intertubes. It’s crazy… from all our power problems back in 2006, we hardly lost any accounts at all.

mastercard.jpg

Goodwill lost: Priceless.

Yeah, it turns out this whole blog post is nothing more than another clichéd MasterCard commercial parody.

P.S. I guess it’s nice to know, less than two hours away from our biggest data center move ever, that we’ll cause a tiny fraction of the disruption to our customers that one unexpected fat finger did!

P.P.S. Thanks RIM, for scheduling a blackberry outage exactly at the same time. It makes us look better. And, maybe some of our Happy Customers will blame their lack of email tonight on you!

80 Responses to “Good Reminiscing Friday”

  1. Punny Money Says:

    Is it bad that I actually feel better knowing this little mistake cost you guys lots of money? :)

  2. BW Says:

    Holy cow that fiasco cost you guys a lot of money. 1000 customers is a lot– that IS crazy. If I did my numbers right, this whole thing ended up costing you nearly half a million dollars. Good thing you guys are a privately held company and not publicly traded– your investors would have had your heads on a stick!

  3. Andrew Says:

    I hope your supreme overlords go easy on you, Josh — I for one enjoy these posts and don’t want to see you go.

  4. Ian Clifton Says:

    Probably would have cost less to get liposuction on that finger before the error. Hell, I hear doctors are giving away free anuses these days.

    These sound like the most obvious costs but what about incidentals (e.g., all the people who are not going to renew when their contract is up, all the people who would have signed up but didn’t because they heard about the problem, etc.)?

  5. Viking Says:

    Holy smokes…
    that is indeed a lot of money.
    Contrary to what BW guesstimated in his comment, I’d say that it’s much more than a half mil.

    Let’s assume that, in average, each of the 1000 customers would have stayed (without the billing issue) for 3 years. Let’s also assume that 800 of them are “normal” shared accounts (120 bucks a year) and the remaining 200 are VPS accounts @300Mhz (30 bucks a month, or 360 bucks a year)

    So, this comes to a yearly loss of 168k

    Now let’s assume a discount rate of 12% and the income loss comes to a present value of 404k (the net present value of lost sales)

    Adding this to the other figures in the post, total net loss comes to almost 710,000 bucks. A LOT OF CASH

    Yes, it was a huge mishap. Yes, we all hope (and kinda know) it won’t happen again. But yes… most of us know better than running away from DH. You guys provide an AMAZING service for just a few bucks. Keep it up!!!
    So what if you lost almost a three quarters of a mil in the process. Money comes and goes… you guys can endure this.

    Go DH!

  6. Sam Greenfield Says:

    I still find it a bit sad that you consider the root problems due to a “fat finger” rather than poor financial controls. And it makes me even sadder that you haven’t written anything to explain how you will avoid these kinds of situations in the future.

    You wrote, “The worst part of this whole process (for us) turned out to be just after the accidental billing, ironically when we were trying to make things right!” I would think that the worst part was actually the loss of goodwill and returning customers. Yes, $170,000 is a lot of money. But it was a one-time potentially-recoverable fee. Personally, I would be much unhappier about the loss of $120,000 in recurring annual subscription fees. Or that customers will stop referring friends and family to your site. Or long-term customers who pay yearly in advance may decide to cancel their account at the end of their term. (Like I may decide to do.)

    You also wrote, “What I’m trying to say is, we’ve always been ultra-flexible and lax about how people pay[...]” That’s great–it’s your prerogative and a nice way to retain customers. How would you feel if your suppliers overcharged you, didn’t return your email messages, and generally acted snarky about the entire incident? There’s a big difference between Accounts Receivable and Accounts Payable. Given that we have a business relationship, I’m less concerned about you making money than I am about me not losing money. I doubt that you truly feel differently.

    So… Serious question: After $304K of one time fees and an estimated recurring loss of $10K/month, what steps have you taken to make sure this can’t happen again?

    On the bright side, you guys get to claim most of this lost money as business expenses. Lower tax liability is always nice, right?

  7. Sam Greenfield Says:

    I still find it a bit sad that you consider the root problems due to a “fat finger” rather than poor financial controls. And it makes me even sadder that you haven’t written anything to explain how you will avoid these kinds of situations in the future.

    You wrote, “The worst part of this whole process (for us) turned out to be just after the accidental billing, ironically when we were trying to make things right!” I would think that the worst part was actually the loss of goodwill and returning customers. Yes, $170,000 is a lot of money. But it was a one-time potentially-recoverable fee. Personally, I would be much unhappier about the loss of $120,000 in recurring annual subscription fees. Or that customers will stop referring friends and family to your site. Or long-term customers who pay yearly in advance may decide to cancel their account at the end of their term. (Like I may decide to do.)

    You also wrote, “What I’m trying to say is, we’ve always been ultra-flexible and lax about how people pay[...]” That’s great–it’s your prerogative and a nice way to retain customers. How would you feel if your suppliers overcharged you, didn’t return your email messages, and generally acted snarky about the entire incident? There’s a big difference between Accounts Receivable and Accounts Payable. Given that we have a business relationship, I’m less concerned about you making money than I am about me not losing money. I doubt that you truly feel differently.

    So… Serious question: After $304K of one time fees and an estimated recurring loss of $10K/month, what steps have you taken to make sure this can’t happen again?

  8. Andy Huang Says:

    Why is this one worser than June through August of 2006? Was it because DH suffered more financial losses, or was it because of something else? In honesty, from a client’s point of view, June through August of 2006 was probably significantly worser than this one :X

  9. Jeff T Says:

    @Sam (”And it makes me even sadder that you haven’t written anything to explain how you will avoid these kinds of situations in the future.”)

    Actually, didn’t they?
    http://blog.dreamhost.com/2008/01/16/the-aftermath/

  10. Nix Says:

    Bad karma on them for keeping that money. You guys did what you could to fix a situation that was shitty, yes, but could have been a hell of a lot worse.

    SHIT HAPPENS. MISTAKES HAPPEN. BOO FUCKING HOO.

    I’m so sick of people whining. Let me whine about the whiners.

  11. Matt A Says:

    It is slightly annoying when the same guy who complains to the high heavens when he thought he’d been over-charged $9,000 by accident conveniently disappears when we realize that actually, he’s been over-refunded $9,000 by accident.

    Is this an exaggerated example or is there really a dude out there who’s gained $9K from this?

  12. Josh Jones Says:

    There really is! :(

  13. ryan Says:

    On the up side, there are now less people on my server :-)

  14. Shawn Says:

    Its fine fellas. We all make mistakes and you guys did something about it in return instead of saying sorry, no refunds. People give you a lot of crap about it but their anger is just a psychological thing, probally something from their childhood. Forgive and Forget is my standard. I was refunded and I am still happy with dreamhost :-) Thats why I am now up to 10 domains :-D -Shawn

  15. T1 Says:

    I know you guys can swing the money that was lost and move on, but I find it pretty disturbing that while you were trying to make an ACCIDENT right, and even paying back more than you were required to… that you had customers trying to rob you blind.

    That $9,000 loudmouth is really pathetic, though.

    Is it bad that I actually feel better knowing this little mistake cost you guys lots of money? :)

    I think what makes you bad isn’t so much that that’s how you think, but that you aren’t even grown up enough to keep it in your head.

  16. Dallas Kashuba Says:

    @ #6 :

    Josh did mention the more longterm cost of lost customer growth and respect when he said…

    “Goodwill lost: Priceless.”

    The post is really only about summing up the near-term financial loss. The long-term losses, financially and otherwise, are very difficult to determine and potentially much more significant.

  17. Al Says:

    Huh, well yeah I lost some money due to the fact the euro is higher than the dollar right now. This whole thing did cost me a pizza and a coca cola! So I was thinking about complaining, but I didn’t know where to go. To Dreamhost or my Bank? I’d rather blame the Bank than Dreamhost, but I was too lazy to go there and draw a waterpistol and eh… eh… sh…! sh…! I’m just too peaceful for doing so.

  18. sdayman Says:

    The post really just summed up how much it costs to shoot yourself in the foot. I hope DreamHost has an awesome set of bulletproof shoes now.

  19. jimnter Says:

    You can use my laser pointer if you want one day.

  20. John Knowles Says:

    It’s a good job it was Josh’s mistake and not one of the techs/accounts staff otherwise they would be walking out the door with a P45 (if you have P45’s in the US) or a not walking and in an ambulance.

    You have the blokes address hunt him down with some big blokes in a blacked out van and get them to “inform” him how it upset you that he tried to con you.

  21. Grey Hodge Says:

    What Josh didn’t mention is that to recover those costs, he’ll be without pay this year (also known as being “fired”) and that the interns will be replacing him. They feel the intern’s can’t do any worse. ;)

  22. Melissa Says:

    Well, if it helps any — this incident actually made me glad to be a client with Dreamhost. Instead of blaming it on something beyond your control (software malfunction), you were completely honest and very upfront about the entire situation and worked very quickly to fix things on your end. You can’t help that the banking and credit card industries move like molasses. The brutal honesty policy you guys have here is what keeps me a customer, so that’s $120/year you guys can count on :)

  23. John Says:

    The thing I remember most about the whole screwup is that they explained pretty much immediately what went wrong. I’m so sick of corporate bullshit that every time DH actually claims responsibility I’m blown away.

    In fact, that’s pretty much my entire opinion of DH. I have incredible respect for them because they offer quality service at good prices and actually take responsibility for everything that goes wrong. Heck, they even gave a name and picture for the one who did it this time.

    When is the last time you saw a major company admit they were wrong and have the employee most directly responsible personally apologize? DH is the only place I’ve seen it.

  24. Matt Butts Says:

    Thanks for the great information. *stumbles*

  25. Lor Says:

    I am proud to be one of the 1,000 customers you lost due to your horrendous error. In my case, that is the reason my account is closed, and why I moved my domains and hosting. What happened was unacceptable, to me, the customer. And, you are still referring the problem to a “fat finger”, which reinforces why I am glad not to be a DH customer any longer.

    Am I sorry you lost the money you did. Not really, because it was your mistake and mistakes need to be rectified and corrected, even if it cost you a monetary outlay. If your post is a way of whining about the money you lost and the accounts you lost, then you still don’t get it, don’t get the seriousness and entirety of the situation, and the problems you caused the majority of your customers.

    And, for those of you who love DH and stayed, you will have one less to contend with on your server! Me. Good for me!

  26. david streever Says:

    I actually think the “worst” disaster in DH history is the generally awful mysql.

    I don’t understand why a company offers something (and even promotes it!) that is basically terrible.

    I have “copy pasted” the Content Management System & all content onto other servers, from other companies, that charge less, and amazingly? Seen page generation drop to under 1 second.

    Yet, on DH, it routinely hits 10+.

    You don’t take any pride in a basic component of your service. That’s why people left. Too many screw-ups & the billing was the last thing.

    I wish I’d left back then :(. Now I’ve paid another 2 years. What a dummy.

    It’s just hard to accept that 9 dollar a month hosting outperforms 15 dollar a month hosting: every day of the week! And after months of being told “it’s your web server let us move you no it’s not the mysql” I finally got a tech to admit, YES it is the mysql, NO it won’t ever be less slow.

    Lying to knowledgeable customers is even more silly than lying to ignorant customers. Thank you, DH, for inspiring me to learn about mysql & how hosting environments could (should!) be run.

  27. anon Says:

    Josh, the biggest disaster of Dreamhost is the incredibly decrease in reliability recently. My websites are down YET AGAIN after the planned outage over the weekend (which was wayyy too long) and YEt AGAIN after the unplanned outage that occurred when you were working on another cluster a few weeks back. Let’s not forget the FAT FINGER that caused your central router to be hosed in the past few weeks as well.

    You need to get your reliability improved. We’ll put up with billing snafus if your service is excellent, but lately, it’s been the pits. Get it together.

  28. Curtis Says:

    Well, my site is down again along with the 30 blog that u host with it.

  29. anon Says:

    PS I wanted to add Josh, that what people aren’t seeing here at DH is any leadership or proactiveness in terms of fixing problems *before* they happen. You guys are constantly putting out fires. You need to start taking action BEFORE things get critical. I’d like to see you talk to us about your plans to make your service more reliable. Maybe you need a consultant or something. Because seriously, the way you are handling things does not appear to be working.

    Just trying to be helpful.

  30. anon Says:

    Well I think you’ve been a great company, and because I don’t live paycheck to paycheck, your small billing error would of caused me no harm. When I first read about your billing error I didn’t even go look to see if I had been charged, as I knew you’d make it right before I would even notice if I had been charged.

    Keep up the good work, and don’t forget some customers will never be happy.

  31. Tom Baker Says:

    I am continually amazed at the sense of entitlement some customers have.

    The customers are buying a service. The details of the arrangement are in the TOS. If the customer doesn’t like the service, then go elsewhere.

    People are talking with such emotion, as if they were betrayed by a best friend. Dreamhost never agreed to be your best friend, they simply agreed to provide a service. And yet people continue to impose implied expectations, which is frankly neither fair nor logical.

    I run two businesses from my cheap-ass shared hosting here, http://www.steradiantech.com and http://www.omegapedal.com . One is a high-dollar B2B company that I own, the other is a smaller but high traffic business to consumer website. And you know what? If my website is down today, the customer will come back tomorrow and buy, because they want my product.

    I just don’t see how DH has a tough enough skin to let these personal attacks not ruin their lives. Because, boy, to read these posts, they are the scum of the earth who promised a cure for cancer to you individually, then said “April Fools!”

    And no, I will not buy any arguments about lost money. If your car fried it’s alternator, you’d be out 3 years of hosting money. And you know what? You couldn’t have personally planned for either outage, car or hosting. Do you think the car repair shop would give a rat’s ass about your complaining?

  32. T1 Says:

    Is it safe to say that you terminated the accounts of everyone that stole the $170,000 in accidental refunds?

  33. John Knowles Says:

    Hasn’t anyone thats moaning about problems stopped to consider that EVERY hosting company has “FIRES” and put them out WHEN they happen. The difference is josh comes on here and says ooops I/We screwed up its my fault and we are sorting it.

    Other companies post up an error report (if your lucky) saying a technical error has caused a problem and it will be fixed soon.

    I have been with DH for over a year now and I have had 99.9%+ uptime I know of 2 outages one if not both was scheduled.

    DH has OVER 600,000 domains yet when a blog is posted about 3-6 people moan. Not bad going if you ask me.

  34. anon Says:

    John, you apparently have not been reading the posts over at status.dreamhost.com. Blingy and spunky customers have been dealing with intermittent service and outages for four days now. This is not just about people moaning. I’ve been on DH for a few years now and I’ve never seen the service this bad. Yes, I am beginning to lose confidence in DH and so are many other customers.

    I’m not sure why people who are simply other customers for DH get so defensive when people have legitimate complaints. I am very, very concerned that DH is only putting out fires and they have lost control over their technical capacity lately.

    I do want to hear more from Josh about plans DH has to proactively deal with their issues. This is not about whining. This is about wanting to have my confidence in DH return.

    DH is a business and as such they absolutely should be concerned when customers complain. Businesses that are not concerned about customer complaints go out of business eventually…unless they are monopolies, and DH is clearly not a monopoly.

  35. jcisio Says:

    Agree with #34. I’ve been with DH for 3 years and I’ve never seen such situation like the last few weeks. Maybe something happened after “the worst disaster” 2 months ago.

  36. mike Says:

    John,

    I believe that dreamhost’s “the worst disaster” happened on Saturday 23rd, when you decided to make the whole spunky cluster (this is over 100 servers with approximately 50 thousands of users) TO BRING THEM ALL DOWN for 12 hours WITHOUT even a slightest notice by e-mail – only a short notice was published on status.dreamhost.com…

    Moreover today all those sites were down again for at least 6 hours – the whole spunky cluster again. Not sure why this has happened but for sure it was not planned.

    Unfortunately I am one of those thousands of users of the unlucky spunky cluster.

    Shame on you guys. I resign today and will keep you in my memory as a nightmare.

  37. nron Says:

    I have been a loyal customer with dreamhost for 5 years, and I have to say that something has changed in the last 2 months with your service.

    My sites are down (blingy has been down for over 10 days, including email!), there has been no updates via status or responses to filed assistance.

    I really admire your commitment to an open attitude, but your recent inability to communicate with your customers has resulted in a mass exodus. If you wish to continue this business, I believe that you should inform all customers immediately as to the status of your service.

    If you were so destroyed by this fat finger, and are considering shutting down, you owe the community you serve information as well.

    I am really shocked at how much your service has deteriorated in the past 1 month, and I can only hope that my patience doesn’t run out. What’s going on??

  38. veda Says:

    I feel sorry for you. My website is down for now but I totally understand this things happen.

    Don’t feel sorry for this. It’s just life and it happens to everybody.

    I was on many hosters before and dreamhost team is one of the most kindness.

    Many thanks.

  39. John Knowles Says:

    I have no problems with legitimate complaints but some people come on who left DH months<years ago and have a go.
    I just wanted to make a point that most hosts don’t even inform everyone of problems so noone knows who else has a problem. your website goes down you think its just your site and get annoyed with the company over it. With DH they inform you there is a problem say they are working on it and keep you up to date.
    I am only defensive because I have had ALOT worse service for ALOT more money.
    I’m not here to argue just to make a point about how the clarity of Dreamhost is rare and welcome policy and not given by many other companies.
    I would also be annoyed if my site was down but problems happen anyone who has ever used a computer knows that

  40. Mark W will probably be banned again Says:

    You gotta be kidding. You actually DID take several people’s websites out when they were suspended for “non-payment”, or have you conveniently forgot that because it didn’t fit the arc of your spiffy blog post?

    Good Christ, Josh. You pretend to be transparent about your practices, and most people swallow it, while meanwhile the way that you’re transparent is quite a bit different.

    It’s pretty rough when a shameful, inexcusable error costs you money, I know. Who could have foreseen that care and responsibility should be exercised with people’s credit and bank information? Shock!

  41. Mark W will probably be banned again Says:

    Not only that, but I guess most people were getting a far better deal than me if 1,000 accounts only costs you $120,000 for a year. I was paying twice that. I certainly was never switched over to a lower pricing scheme when you offered discounted rates year after year. My price was always “Grandfathered” at the high rate I started with. For my first couple years, I was paying close to $50/month for what is now a $10/month account even when the price dropped again and again.

    I can’t remember which blog post it was where you mentioned that you (paraphrased) ‘let customers know when a cheaper hosting plan would suit their needs’, but I was sure never one of those customers.

  42. Purple Says:

    To be honest I would have rather been involved in the big financial over charging mess that occured than to be currently involved in the disaster that is the Blingy Cluster.
    DH came highly recommended to me. I only hope that I get to see it!

  43. Tom Baker Says:

    That you #37. Your concerns are laid out in a manner that dreamhost should be able to respond to. If they don’t, I’d be heading out the door.
    Pretty much all the posts over on the status page seem to be nothing more than a bar fight. I can see why that is “Not an Official Way to Contact Dreamhost”.

    It’d be interesting to see how many companies actually provide a hosted, branded place for all their customers to rip on them and say the worst possible things they can come up with.

  44. Glenn Says:

    As I’m sure you already know from this situation, a little bit of communication goes a long way. The current situation with Bling and Spanky is causing problems for your customers because the communications in not there like it was during the billing snafu.

    The dreamhostsstatus.com site is suppose to contain the status information on your systems, but with the current situations it is not being updated. When their websites and e-mail are down for hours/days and there is no updated status, the customers are going to get mad. And when you give a status, tell them what is going on, not just some canned “we’re working on it” response.

    Many of your customers are very bright people. If a hard-drive is being replaced, tell them! That at least will give them an idea of the time that will be required. Also, providing some facility to redirect requests to servers that are down to give their user’s a better error message (rather than “connection timeout”) would go a long way with the users that actually are trying to get on your customer’s websites.

    The big issue now is the lack of communication, which is giving DreamHost a bad image. The impression is that the company cannot handle the number of customers it has. Put on your “leader hat” and provide the leadership that is needed right now.

    For the record, like many of your customers, I have been in the computer industry for more than two decades. I have to deal with customers on a daily basis. Communications is key with them. Communicate with your customers now about the Bling/Spanky cluster issue like you did with the billing issue. Maybe you can keep the number of lost customers DOWN TO only 1000.

  45. Cerri Says:

    I have to agree 100% with Glenn a little communication would go a long way with me. I happen to be one of the unlucky on blingy whose site is going up and down like a see saw. If there were some time frame or an end in sight notice it’d be really appreciated. Even if you could say something like “this will take another 48 hours” at least I could put up a notice for my visitors and tell them that the issue can be fixed. Some of us have very frequented websites and have been getting nasty emails about our sites being down but telling them “it’s a problem but I don’t know what it’s going to be fixed” makes us look just as bad! I’ve had 3 great months of service but this past week has had me tearing my hair out angry that I can’t take my money and run.

    I’d love for you guys to prove me wrong since the 3 people who recommended me to Dreamhost have had great service for three YEARS. I know things happen, but this is days now…and no update for over 12 hours…?

  46. Gareth Says:

    #42 – right on.
    Like most of the more recent posts, I’d contend your worst snafu is currently in progress. Not only did you feel fit to move servers with a large outage, most of the servers have been cactus for days. And all we hear is “we’re working on it”. It may not be “mission critical” hosting, but just about any fault like this becomes urgent enough if it is left to fester for days on end.
    I left my previous host because of their lack of communication regarding a (in that case a multi-week) outage. It’s starting to look like it maybe occurring again….

  47. hostdemon.net Says:

    You would think a large web hosting company like DreamHost would refrain from shedding anymore light on this billing situation with a blog post like this.

    Good customer service and decent servers doesn’t matter much when you make big finical burdens for clients.

    This only makes DreamHost seem like they don’t. After all the massive overselling, lack of knowledge to run a billing system correctly, and customer service cockiness do you think clients wouldn’t question canceling?

  48. Jeff @ DreamHost Says:

    > You would think a large web hosting company like
    > DreamHost would refrain from shedding anymore
    > light on this billing situation with a blog post
    > like this.

    Yeah, you’d think so. In our industry, covering up for stupid mistakes is the norm. Most would simply pretend it never even happened.

    Trust me, if I ever see that stupid Homer Simpson picture again it will be far too soon. It’s all too tempting to never mention our not-so-little mistake again.

    As you can imagine, though, there was a lot of – erm – interest in the matter among those customers impacted. I think this post can serve as a valuable cautionary tale to other companies that such mistakes can have very real bottom-line ramifications.

    If other companies were half as forthright about their screw-ups as we are about ours, things would be a far different. They would have more incentive to make changes and improvements, lest they get a well-deserved black eye.

    Then again, who knows? Maybe we should do the usual thing and hire an expensive PR firm to “manage our message” and handle damage control. Maybe we should keep these things to ourselves, wrapping them in a thick layer of BS that any customer of average intelligence will see right through. Maybe we should limit our blog posts to promotion of our services and dry essays about the industry (yawn). Maybe our co-founder, upon making a huge billing error, should just schedule another round of golf and let the rank and file take the blame.

    I can understand still being angry about what happened, but I’m not sure I can understand why anyone would prefer we just paper over it and pretend it didn’t happen.

    And it’s not as if we’ve limited ourselves to talking about it – we also implemented and outlined four specific changes that should help prevent it from ever happening again.

    Say what you will about our company or our offerings – that’s open for debate (a debate we’ll gladly host in our own weblog, mind you), but we’re not the type of company that will make a huge mistake and then just sweep it under the rug.

    - Jeff @ DreamHost Web Hosting

  49. McGregor Says:

    I just started with Dreamhost. My e-mail has been down since Monday. WTF? I’m trying to launch a business and I have no e-mail. The folks who designed my website said go with Dreamhost because we’ve had problems with Go Daddy. I guess I just sit and wait? Poor business practices!!!

  50. Dave Says:

    Thanks for such a witty blog update… only if you took the time you spent on this and put it into your product.

    “What we’ve always tried to focus on more (even though it might not seem like it at times!) is our hosting system’s stability, performance, and features.” is a joke if you ask me.

    I love dreamhost and all the great features you guys have but not when you do something like the spunky move.

    ps. fix it!

  51. phoenix Says:

    Ah well, that’s 1000 losers that, while the revenue is gone, would you really have wanted those people as customers? Really? Them and those pesky people who use blog posts as cries for tech support help-oops!

    That all being said, more resources for the paying ones! ;)

  52. Rob M Says:

    I find it staggering that people still come on here to whine about Josh’s glib tone and heavy use of irony.

    Are they new to this blog or something?

    Change your bookmarks to status.dreamhost.com and go away, you po-faced whiners.

  53. Martin Says:

    After two months it’s still not funny so get your head out of your fucking ass.

  54. Nick Says:

    What I’d like to know is if any heads rolled for this.

    I’d also like to know, WHY weren’t the checks in place implemented sooner? It seems that all of DreamHost’s problems are things that could have been anticipated sooner.

    I’m on Lunarpages, and was tempted to switch to DreamHost. But Lunarpages basic plan offers 1500 GB (three times DreamHost’s), more auto-installs, and just about everything else DreamHost offers, it meets or tops. And I have yet to experience downtime that wasn’t due to them changing something for me (like servers).

  55. Ralphie Says:

    What, we’re supposed to feel sorry for you now Josh? Fuck off asshole – lick my balls you faggot.

  56. lprefugees Says:

    @#54 On March 28, 2008, in a Comment by Nick: “What I’d like to know is if any heads rolled for this… I’m on Lunarpages…”

    What a hoot that is! Hey Nick, did any “heads roll” over at Lunar Pages over that little stunt when they misappropriated all you guys’ error pages?

    So you’d fire someone for a mistake, and just go on with business as usual when a company deliberately screws their customers and then handles it deceitfully? Nice!

    http://lprefugees.com

  57. T1 Says:

    I guess we could believe that all other lunarpages reviews out there are fake, but Nick, who shows up on a competitor’s blog to advertise is a “real customer.”

    Everything you need to know about lunarpages is right here http://spacytracie.com/dreamhost-and-lunarpages/391/

    For some crazy reason, if lunarpages made the same billing error, I’d picture them on a beach somewhere in Mexico laughing and counting the money.

    I wouldn’t care if they offered 100% uptime for $1/year… I’d trust a crackhead with my credit card info before them.

  58. RighteousIndignation Says:

    This is so sad … I came here today to see what Josh would do for April Fools’ Day, but it seems maybe the fools who bitch about his style have discouraged him from evidencing any humor at all.

    This is so sad … that so many people take themselves so seriously, are so handicapped by their lack of a sense of humor, and are just so generally miserable in their existence that their whining actually starts to infect the mentally healthy.

    This is so sad … but, yeah, I already said that. :/

    So to all you outing little mental defectives – meh.

  59. db Says:

    The loss of customers and cash is easily calculated. Whats harder to figure out is how many people are not signing up due to the “fat finger”.

  60. Bob Says:

    > Well, it was a little over two months ago that
    > we had what I think is pretty safe to call the
    > worst disaster in DreamHost history.

    The worst is yet to come. I think that the worst disaster in DreamHost history is gonna be this: http://www.dreamhoststatus.com/2008/03/27/filer-problems-with-blingy-cluster/

    It’s 2 weeks now and people are closing accounts by the minute…

  61. nightmare host whiner Says:

    Wow ! thats great … I hope it happens again… love to see you making losses… and thanks for fucking up my websites for half a month.

  62. Huy Says:

    Hey guys,

    Wow! Sorry for the big loss. That is crazy! I myself didn’t get affected by this that bad, as I guess I was never really charged for it, and never “refunded” either. :P

    But anyhow, keep up the great offers. I look forward to continued business with you all. Besides, who else sends out those funny newsletters (though, admittedly, I don’t read them much), and write up the blogs like the average Joe… showing you’re human, and not just “professional” (personas), trying to extract money from us… though I’m sure that being like that, you’re doing just that.

    I’m sure you’ll recover just fine… heck, I’m not worried about you guys at all financially speaking. I better worry about my own bank account.. NO CUSHION at all. :P LOL.

    Take care.

  63. Graham Says:

    What a mistake! You will get over it, right?

  64. Graham Says:

    What a mistake! You will get over it, right..

  65. T1 Says:

    Wow ! thats great … I hope it happens again… love to see you making losses… and thanks for fucking up my websites for half a month.

    That bitterness is hurting you more than it hurts them.

  66. Sean Serritella Says:

    Josh, I can’t take this anymore. I’m competing with other baseball sites and because of YOU and your shit company, I’m going to lose. My website, NEVER works. Almost never and I can’t take it anymore. For now on, on my site, after every post, I’m thinking about making an announcement about how much dream host sucks. I’m also thinking about starting my own website like some other guy did about how bad you guys blow. Josh, how does it feel to be incompetent? Right now, some one is sending their posts to digg and other sites but I can’t because I’ve been trying to make an update, just 1 little update to my site, FOR ONE WHOLE MOTHER FUCKING HOUR!!!! I CAN’T TAKE THIS ANY MORE YOU JERK OFF!!!!!

  67. Sean Serritella Says:

    WATCH WHAT I WRITE ABOUT YOU ON MY BLOG JOSH, YOU JERK OFF.

  68. Sean Serritella Says:

    Want to hear something stupid I did? I paid dream Host 3 years in advance. Talk about dumb. Josh, if you’re so honest, would you refund my money back and let me take my blog elsewhere? You know that your service sucks and that no one likes your hosting company. You know I would be better off with another company because you’re such a fucking idiot. Josh = Jerk Off.

  69. Sean Serritella Says:

    And when you respond back Josh, you will be called by me, “The piece of shit,” because that’s what you are Josh, a no good mother fucking failure and and a big piece of shit. I fucking HATE you Josh, you piece of shit.

  70. T1 Says:

    Oh no! An angry 10 year old with a blog!!!

    WATCH WHAT I WRITE ABOUT YOU ON MY BLOG JOSH, YOU JERK OFF.

    Better type fast if you want to finish the post before your tears short out the keyboard.

  71. VOO Says:

    WATCH WHAT I WRITE ABOUT YOU ON MY BLOG JOSH, YOU JERK OFF.

    Wow… what a useless to humanity website. Obviously your shitty site is being punished by The Lord for promoting people to think about useless shit like sports. Thank you SO FUCKIN MUCH for perpetuating humanity’s devolution. Why don’t you get yourself some peyote or something and expand your mind beyond who can hit a ball with a stick the best?

    Yes… VOO has returned after all of this time. Update: The Lord regrets to inform you that he has decided not to host His websites on your servers at this time, but he gives you blessings and is grateful for the role that you play in smiting such shitty sites as Sean’s.

  72. The Captain Says:

    It’s funny….In the next week I’ll finally be able to cancel my account with DH, a direct result of the 2006 power outage. Yes, it’s taken a few years but moving about 300 domains has been a slow tedious process. But it’s been more than a year since I’ve used DH for anything but DNS.

  73. DHater Says:

    DH asshats are about as useful as tits on a bull.

  74. robina Says:

    I believe the WORST Dreamhost disaster to ever happen to ME was when you DIDN’T include ME in your accidental repayment issue. I’m in debt, Josh. Throw some of that $170k my way next time. I promise not to complain about it. :P

    Welcome to the internet, everyone. It’s as reliable as a monkey shoveling snow. As one who has had her domain hosted with 4 other services, I still maintain that DH is the best. Keep the communication up, and I’ll stick around.

  75. E. Dewey Says:

    What about for those of us who already left due to your mistake? ARe you going to refund us for the last month? My php admin and ftp were down for over a month before you did anything about it. And even then you didn’t fix it. SO I moved on.

  76. JasonA Says:

    Thank you for the transparency. Nothing like a crisis to get people focused.

    Jeff, thanks for posting the link, (four comments up), to the update, (which explains steps taken to prevent it from reocurring).

  77. Anthony Says:

    People make mistakes, this type of problem could happen to any company. Actually I think that the whole incident shows how transparent DreamHost is. They made a mistake and fixed it as soon as possible, imho going above and beyond what they needed to do. With other companies you would have an uphill struggle trying to get your bank charges back!

    I wouldn’t hesitate to recommend DreamHost to anyone. :)

    I actually only can fault them on one thing, the fact that new customers (with a voucher code) can get 3 IPs for life for FREE, but long term customers have to pay $47 for per IP address per year!!! Now that is annoying! :(

  78. Losing Business Says:

    Our email and website went down on Friday. AmEx is all fired up about fraudulent charges. Most of our clients communicate with us solely by email. So, we’ll just call Dreamhost and get it resolved. So, the voicemail box is full, can’t get through by email and sending a fax got no response. It’s nice to know that you can still joke around on your blog while people are losing money and tearing their hair out.

  79. Islami Kiz Isimleri Islami Erkek Isimleri Says:

    You can use my laser pointer if you want one day.

  80. kareem Says:

    this is wonderful topic …. i will put acopy of this topic on
    my site here
    http://www.as7ap4you.com

Powered by WordPress. Pool theme by Borja Fernandez, modified by DreamHost.
Like WordPress? Consider attending WordCamp LA.
Entries and comments feeds. ^Top^