Good Reminiscing Friday

March 21, 2008 on 6:09 pm | In Foobars, Insider View, Updates by Josh Jones |

Those were the days!

Well, it was a little over two months ago that we had what I think is pretty safe to call the worst disaster in DreamHost history.

In retrospect to me, it’s kind of funny that the worst disaster didn’t turn out to be due to a security breach, a power outage, a loss of data, or actually anything related to our actual hosting service. I guess it shouldn’t be a surprise that people care a lot more about their bank accounts than they do their websites.

I have realized that billing is the one issue where how important we feel it is is completely at odds with how important you guys feel it is.

What I’m trying to say is, we’ve always been ultra-flexible and lax about how people pay, when people pay, or even about giving credits, discounts, or refunds. We figure, whatever, pay us when you’re ready, we’re not sending anybody to collections or ruining anybody’s credit over some measly bandwidth bill.

If everybody had just been paying by check!

What we’ve always tried to focus on more (even though it might not seem like it at times!) is our hosting system’s stability, performance, and features.

I guess I’ve always figured that any billing-related error can be easily undone (worst case scenario, it costs us a little money); there is no lasting harm done to the customer. Whereas having a website or email problem could potentially cause permanent damage to somebody’s business or personal life or something?

Well then, let’s go back and see just how little money a worst case scenario actually costs, shall we?

Credits and refunds to cover people’s bank fees: $52,000.

Sigh, if only everybody kept a big cushion of cash in their account! The main damage that can be caused by a billing snafu is for people who get their account overdrawn, and because of that aren’t able to make a critical purchase, or have a check bounce, causing hassles and incurring bank fees. We offered to pay people any amount their bank charged them for going negative, and in the end that total looks like it came to about $52,000.

Discover how much money I lost DreamHost!

Accidental refunds: $170,000.

The worst part of this whole process (for us) turned out to be just after the accidental billing, ironically when we were trying to make things right!

If you recall, our system was not actually charging about 75% of the time we thought it did.. and so we refunded thousands of people who were never charged (but, 75% of the refunds didn’t work either). Well, out of all that, and after two months, there are still about 600 accounts who were credited a total of $170,000 in excess of what we charged them that we haven’t been able to get back from them or their bank.

It is slightly annoying when the same guy who complains to the high heavens when he thought he’d been over-charged $9,000 by accident conveniently disappears when we realize that actually, he’s been over-refunded $9,000 by accident.

Extra credit card fees: $82,000.

Another slightly annoying thing is that credit card processors don’t credit you back any fees when you refund a transaction. Overall, the extra credit card processing we did resulted in extra fees of about $350,000! Fortunately, after a whole lot of groveling and explaining the situation (and waiting two months), we finally got all but $82,000 of that back from First Data, American Express, and Discover Card.

Apparently our snafu didn't screw up Visa's IPO too badly.

Extra support messages: 20,000.

As you may have surmised, people wrote to us about this thing. About 20,000 times… and it would have been tens of thousands more if we hadn’t put up an “emergency block” against new messages for a little while in there.

How much this extra support actually cost (in terms of your wased time, tech support overtime pay, and other questions taking longer to answer to) is hard to say, but normally we only get about 45,000 messages in a whole month!

Accounts canceled: 1000.

It’s also kind of hard to say how many people actually closed their account because of the incident, but in January we did have about 1,000 more accounts closed than average. Assuming each of those accounts would have stayed for maybe another year, that’s another $120,000 down the Intertubes. It’s crazy… from all our power problems back in 2006, we hardly lost any accounts at all.

mastercard.jpg

Goodwill lost: Priceless.

Yeah, it turns out this whole blog post is nothing more than another clichéd MasterCard commercial parody.

P.S. I guess it’s nice to know, less than two hours away from our biggest data center move ever, that we’ll cause a tiny fraction of the disruption to our customers that one unexpected fat finger did!

P.P.S. Thanks RIM, for scheduling a blackberry outage exactly at the same time. It makes us look better. And, maybe some of our Happy Customers will blame their lack of email tonight on you!

78 Comments »

RSS feed for comments on this post.

Pages: [1] 2 » Show All

  1. 1

    Is it bad that I actually feel better knowing this little mistake cost you guys lots of money? :)

    Comment by Punny Money — March 21, 2008 #

  2. 2

    Holy cow that fiasco cost you guys a lot of money. 1000 customers is a lot– that IS crazy. If I did my numbers right, this whole thing ended up costing you nearly half a million dollars. Good thing you guys are a privately held company and not publicly traded– your investors would have had your heads on a stick!

    Comment by BW — March 21, 2008 #

  3. 3

    I hope your supreme overlords go easy on you, Josh — I for one enjoy these posts and don’t want to see you go.

    Comment by Andrew — March 21, 2008 #

  4. 4

    Probably would have cost less to get liposuction on that finger before the error. Hell, I hear doctors are giving away free anuses these days.

    These sound like the most obvious costs but what about incidentals (e.g., all the people who are not going to renew when their contract is up, all the people who would have signed up but didn’t because they heard about the problem, etc.)?

    Comment by Ian Clifton — March 21, 2008 #

  5. 5

    Holy smokes…
    that is indeed a lot of money.
    Contrary to what BW guesstimated in his comment, I’d say that it’s much more than a half mil.

    Let’s assume that, in average, each of the 1000 customers would have stayed (without the billing issue) for 3 years. Let’s also assume that 800 of them are “normal” shared accounts (120 bucks a year) and the remaining 200 are VPS accounts @300Mhz (30 bucks a month, or 360 bucks a year)

    So, this comes to a yearly loss of 168k

    Now let’s assume a discount rate of 12% and the income loss comes to a present value of 404k (the net present value of lost sales)

    Adding this to the other figures in the post, total net loss comes to almost 710,000 bucks. A LOT OF CASH

    Yes, it was a huge mishap. Yes, we all hope (and kinda know) it won’t happen again. But yes… most of us know better than running away from DH. You guys provide an AMAZING service for just a few bucks. Keep it up!!!
    So what if you lost almost a three quarters of a mil in the process. Money comes and goes… you guys can endure this.

    Go DH!

    Comment by Viking — March 21, 2008 #

  6. 6

    I still find it a bit sad that you consider the root problems due to a “fat finger” rather than poor financial controls. And it makes me even sadder that you haven’t written anything to explain how you will avoid these kinds of situations in the future.

    You wrote, “The worst part of this whole process (for us) turned out to be just after the accidental billing, ironically when we were trying to make things right!” I would think that the worst part was actually the loss of goodwill and returning customers. Yes, $170,000 is a lot of money. But it was a one-time potentially-recoverable fee. Personally, I would be much unhappier about the loss of $120,000 in recurring annual subscription fees. Or that customers will stop referring friends and family to your site. Or long-term customers who pay yearly in advance may decide to cancel their account at the end of their term. (Like I may decide to do.)

    You also wrote, “What I’m trying to say is, we’ve always been ultra-flexible and lax about how people pay[...]” That’s great–it’s your prerogative and a nice way to retain customers. How would you feel if your suppliers overcharged you, didn’t return your email messages, and generally acted snarky about the entire incident? There’s a big difference between Accounts Receivable and Accounts Payable. Given that we have a business relationship, I’m less concerned about you making money than I am about me not losing money. I doubt that you truly feel differently.

    So… Serious question: After $304K of one time fees and an estimated recurring loss of $10K/month, what steps have you taken to make sure this can’t happen again?

    On the bright side, you guys get to claim most of this lost money as business expenses. Lower tax liability is always nice, right?

    Comment by Sam Greenfield — March 21, 2008 #

  7. 7

    I still find it a bit sad that you consider the root problems due to a “fat finger” rather than poor financial controls. And it makes me even sadder that you haven’t written anything to explain how you will avoid these kinds of situations in the future.

    You wrote, “The worst part of this whole process (for us) turned out to be just after the accidental billing, ironically when we were trying to make things right!” I would think that the worst part was actually the loss of goodwill and returning customers. Yes, $170,000 is a lot of money. But it was a one-time potentially-recoverable fee. Personally, I would be much unhappier about the loss of $120,000 in recurring annual subscription fees. Or that customers will stop referring friends and family to your site. Or long-term customers who pay yearly in advance may decide to cancel their account at the end of their term. (Like I may decide to do.)

    You also wrote, “What I’m trying to say is, we’ve always been ultra-flexible and lax about how people pay[...]” That’s great–it’s your prerogative and a nice way to retain customers. How would you feel if your suppliers overcharged you, didn’t return your email messages, and generally acted snarky about the entire incident? There’s a big difference between Accounts Receivable and Accounts Payable. Given that we have a business relationship, I’m less concerned about you making money than I am about me not losing money. I doubt that you truly feel differently.

    So… Serious question: After $304K of one time fees and an estimated recurring loss of $10K/month, what steps have you taken to make sure this can’t happen again?

    Comment by Sam Greenfield — March 21, 2008 #

  8. 8

    Why is this one worser than June through August of 2006? Was it because DH suffered more financial losses, or was it because of something else? In honesty, from a client’s point of view, June through August of 2006 was probably significantly worser than this one :X

    Comment by Andy Huang — March 21, 2008 #

  9. 9

    @Sam (”And it makes me even sadder that you haven’t written anything to explain how you will avoid these kinds of situations in the future.”)

    Actually, didn’t they?
    http://blog.dreamhost.com/2008/01/16/the-aftermath/

    Comment by Jeff T — March 21, 2008 #

  10. 10

    Bad karma on them for keeping that money. You guys did what you could to fix a situation that was shitty, yes, but could have been a hell of a lot worse.

    SHIT HAPPENS. MISTAKES HAPPEN. BOO FUCKING HOO.

    I’m so sick of people whining. Let me whine about the whiners.

    Comment by Nix — March 22, 2008 #

  11. 11

    It is slightly annoying when the same guy who complains to the high heavens when he thought he’d been over-charged $9,000 by accident conveniently disappears when we realize that actually, he’s been over-refunded $9,000 by accident.

    Is this an exaggerated example or is there really a dude out there who’s gained $9K from this?

    Comment by Matt A — March 22, 2008 #

  12. 12

    There really is! :(

    Comment by Josh Jones — March 22, 2008 #

  13. 13

    On the up side, there are now less people on my server :-)

    Comment by ryan — March 22, 2008 #

  14. 14

    Its fine fellas. We all make mistakes and you guys did something about it in return instead of saying sorry, no refunds. People give you a lot of crap about it but their anger is just a psychological thing, probally something from their childhood. Forgive and Forget is my standard. I was refunded and I am still happy with dreamhost :-) Thats why I am now up to 10 domains :-D -Shawn

    Comment by Shawn — March 22, 2008 #

  15. 15

    I know you guys can swing the money that was lost and move on, but I find it pretty disturbing that while you were trying to make an ACCIDENT right, and even paying back more than you were required to… that you had customers trying to rob you blind.

    That $9,000 loudmouth is really pathetic, though.

    Is it bad that I actually feel better knowing this little mistake cost you guys lots of money? :)

    I think what makes you bad isn’t so much that that’s how you think, but that you aren’t even grown up enough to keep it in your head.

    Comment by T1 — March 22, 2008 #

  16. 16

    @ #6 :

    Josh did mention the more longterm cost of lost customer growth and respect when he said…

    “Goodwill lost: Priceless.”

    The post is really only about summing up the near-term financial loss. The long-term losses, financially and otherwise, are very difficult to determine and potentially much more significant.

    Comment by Dallas Kashuba — March 22, 2008 #

  17. 17

    Huh, well yeah I lost some money due to the fact the euro is higher than the dollar right now. This whole thing did cost me a pizza and a coca cola! So I was thinking about complaining, but I didn’t know where to go. To Dreamhost or my Bank? I’d rather blame the Bank than Dreamhost, but I was too lazy to go there and draw a waterpistol and eh… eh… sh…! sh…! I’m just too peaceful for doing so.

    Comment by Al — March 22, 2008 #

  18. 18

    The post really just summed up how much it costs to shoot yourself in the foot. I hope DreamHost has an awesome set of bulletproof shoes now.

    Comment by sdayman — March 22, 2008 #

  19. 19

    You can use my laser pointer if you want one day.

    Comment by jimnter — March 22, 2008 #

  20. 20

    It’s a good job it was Josh’s mistake and not one of the techs/accounts staff otherwise they would be walking out the door with a P45 (if you have P45’s in the US) or a not walking and in an ambulance.

    You have the blokes address hunt him down with some big blokes in a blacked out van and get them to “inform” him how it upset you that he tried to con you.

    Comment by John Knowles — March 23, 2008 #

  21. 21

    What Josh didn’t mention is that to recover those costs, he’ll be without pay this year (also known as being “fired”) and that the interns will be replacing him. They feel the intern’s can’t do any worse. ;)

    Comment by Grey Hodge — March 24, 2008 #

  22. 22

    Well, if it helps any — this incident actually made me glad to be a client with Dreamhost. Instead of blaming it on something beyond your control (software malfunction), you were completely honest and very upfront about the entire situation and worked very quickly to fix things on your end. You can’t help that the banking and credit card industries move like molasses. The brutal honesty policy you guys have here is what keeps me a customer, so that’s $120/year you guys can count on :)

    Comment by Melissa — March 24, 2008 #

  23. 23

    The thing I remember most about the whole screwup is that they explained pretty much immediately what went wrong. I’m so sick of corporate bullshit that every time DH actually claims responsibility I’m blown away.

    In fact, that’s pretty much my entire opinion of DH. I have incredible respect for them because they offer quality service at good prices and actually take responsibility for everything that goes wrong. Heck, they even gave a name and picture for the one who did it this time.

    When is the last time you saw a major company admit they were wrong and have the employee most directly responsible personally apologize? DH is the only place I’ve seen it.

    Comment by John — March 24, 2008 #

  24. 24

    Thanks for the great information. *stumbles*

    Comment by Matt Butts — March 24, 2008 #

  25. 25

    I am proud to be one of the 1,000 customers you lost due to your horrendous error. In my case, that is the reason my account is closed, and why I moved my domains and hosting. What happened was unacceptable, to me, the customer. And, you are still referring the problem to a “fat finger”, which reinforces why I am glad not to be a DH customer any longer.

    Am I sorry you lost the money you did. Not really, because it was your mistake and mistakes need to be rectified and corrected, even if it cost you a monetary outlay. If your post is a way of whining about the money you lost and the accounts you lost, then you still don’t get it, don’t get the seriousness and entirety of the situation, and the problems you caused the majority of your customers.

    And, for those of you who love DH and stayed, you will have one less to contend with on your server! Me. Good for me!

    Comment by Lor — March 24, 2008 #

  26. 26

    I actually think the “worst” disaster in DH history is the generally awful mysql.

    I don’t understand why a company offers something (and even promotes it!) that is basically terrible.

    I have “copy pasted” the Content Management System & all content onto other servers, from other companies, that charge less, and amazingly? Seen page generation drop to under 1 second.

    Yet, on DH, it routinely hits 10+.

    You don’t take any pride in a basic component of your service. That’s why people left. Too many screw-ups & the billing was the last thing.

    I wish I’d left back then :(. Now I’ve paid another 2 years. What a dummy.

    It’s just hard to accept that 9 dollar a month hosting outperforms 15 dollar a month hosting: every day of the week! And after months of being told “it’s your web server let us move you no it’s not the mysql” I finally got a tech to admit, YES it is the mysql, NO it won’t ever be less slow.

    Lying to knowledgeable customers is even more silly than lying to ignorant customers. Thank you, DH, for inspiring me to learn about mysql & how hosting environments could (should!) be run.

    Comment by david streever — March 25, 2008 #

  27. 27

    Josh, the biggest disaster of Dreamhost is the incredibly decrease in reliability recently. My websites are down YET AGAIN after the planned outage over the weekend (which was wayyy too long) and YEt AGAIN after the unplanned outage that occurred when you were working on another cluster a few weeks back. Let’s not forget the FAT FINGER that caused your central router to be hosed in the past few weeks as well.

    You need to get your reliability improved. We’ll put up with billing snafus if your service is excellent, but lately, it’s been the pits. Get it together.

    Comment by anon — March 25, 2008 #

  28. 28

    Well, my site is down again along with the 30 blog that u host with it.

    Comment by Curtis — March 25, 2008 #

  29. 29

    PS I wanted to add Josh, that what people aren’t seeing here at DH is any leadership or proactiveness in terms of fixing problems *before* they happen. You guys are constantly putting out fires. You need to start taking action BEFORE things get critical. I’d like to see you talk to us about your plans to make your service more reliable. Maybe you need a consultant or something. Because seriously, the way you are handling things does not appear to be working.

    Just trying to be helpful.

    Comment by anon — March 25, 2008 #

  30. 30

    Well I think you’ve been a great company, and because I don’t live paycheck to paycheck, your small billing error would of caused me no harm. When I first read about your billing error I didn’t even go look to see if I had been charged, as I knew you’d make it right before I would even notice if I had been charged.

    Keep up the good work, and don’t forget some customers will never be happy.

    Comment by anon — March 25, 2008 #

  31. 31

    I am continually amazed at the sense of entitlement some customers have.

    The customers are buying a service. The details of the arrangement are in the TOS. If the customer doesn’t like the service, then go elsewhere.

    People are talking with such emotion, as if they were betrayed by a best friend. Dreamhost never agreed to be your best friend, they simply agreed to provide a service. And yet people continue to impose implied expectations, which is frankly neither fair nor logical.

    I run two businesses from my cheap-ass shared hosting here, http://www.steradiantech.com and http://www.omegapedal.com . One is a high-dollar B2B company that I own, the other is a smaller but high traffic business to consumer website. And you know what? If my website is down today, the customer will come back tomorrow and buy, because they want my product.

    I just don’t see how DH has a tough enough skin to let these personal attacks not ruin their lives. Because, boy, to read these posts, they are the scum of the earth who promised a cure for cancer to you individually, then said “April Fools!”

    And no, I will not buy any arguments about lost money. If your car fried it’s alternator, you’d be out 3 years of hosting money. And you know what? You couldn’t have personally planned for either outage, car or hosting. Do you think the car repair shop would give a rat’s ass about your complaining?

    Comment by Tom Baker — March 25, 2008 #

  32. 32

    Is it safe to say that you terminated the accounts of everyone that stole the $170,000 in accidental refunds?

    Comment by T1 — March 25, 2008 #

  33. 33

    Hasn’t anyone thats moaning about problems stopped to consider that EVERY hosting company has “FIRES” and put them out WHEN they happen. The difference is josh comes on here and says ooops I/We screwed up its my fault and we are sorting it.

    Other companies post up an error report (if your lucky) saying a technical error has caused a problem and it will be fixed soon.

    I have been with DH for over a year now and I have had 99.9%+ uptime I know of 2 outages one if not both was scheduled.

    DH has OVER 600,000 domains yet when a blog is posted about 3-6 people moan. Not bad going if you ask me.

    Comment by John Knowles — March 26, 2008 #

  34. 34

    John, you apparently have not been reading the posts over at status.dreamhost.com. Blingy and spunky customers have been dealing with intermittent service and outages for four days now. This is not just about people moaning. I’ve been on DH for a few years now and I’ve never seen the service this bad. Yes, I am beginning to lose confidence in DH and so are many other customers.

    I’m not sure why people who are simply other customers for DH get so defensive when people have legitimate complaints. I am very, very concerned that DH is only putting out fires and they have lost control over their technical capacity lately.

    I do want to hear more from Josh about plans DH has to proactively deal with their issues. This is not about whining. This is about wanting to have my confidence in DH return.

    DH is a business and as such they absolutely should be concerned when customers complain. Businesses that are not concerned about customer complaints go out of business eventually…unless they are monopolies, and DH is clearly not a monopoly.

    Comment by anon — March 26, 2008 #

  35. 35

    Agree with #34. I’ve been with DH for 3 years and I’ve never seen such situation like the last few weeks. Maybe something happened after “the worst disaster” 2 months ago.

    Comment by jcisio — March 26, 2008 #

  36. 36

    John,

    I believe that dreamhost’s “the worst disaster” happened on Saturday 23rd, when you decided to make the whole spunky cluster (this is over 100 servers with approximately 50 thousands of users) TO BRING THEM ALL DOWN for 12 hours WITHOUT even a slightest notice by e-mail - only a short notice was published on status.dreamhost.com…

    Moreover today all those sites were down again for at least 6 hours - the whole spunky cluster again. Not sure why this has happened but for sure it was not planned.

    Unfortunately I am one of those thousands of users of the unlucky spunky cluster.

    Shame on you guys. I resign today and will keep you in my memory as a nightmare.

    Comment by mike — March 26, 2008 #

  37. 37

    I have been a loyal customer with dreamhost for 5 years, and I have to say that something has changed in the last 2 months with your service.

    My sites are down (blingy has been down for over 10 days, including email!), there has been no updates via status or responses to filed assistance.

    I really admire your commitment to an open attitude, but your recent inability to communicate with your customers has resulted in a mass exodus. If you wish to continue this business, I believe that you should inform all customers immediately as to the status of your service.

    If you were so destroyed by this fat finger, and are considering shutting down, you owe the community you serve information as well.

    I am really shocked at how much your service has deteriorated in the past 1 month, and I can only hope that my patience doesn’t run out. What’s going on??

    Comment by nron — March 26, 2008 #

  38. 38

    I feel sorry for you. My website is down for now but I totally understand this things happen.

    Don’t feel sorry for this. It’s just life and it happens to everybody.

    I was on many hosters before and dreamhost team is one of the most kindness.

    Many thanks.

    Comment by veda — March 26, 2008 #

  39. 39

    I have no problems with legitimate complaints but some people come on who left DH months<years ago and have a go.
    I just wanted to make a point that most hosts don’t even inform everyone of problems so noone knows who else has a problem. your website goes down you think its just your site and get annoyed with the company over it. With DH they inform you there is a problem say they are working on it and keep you up to date.
    I am only defensive because I have had ALOT worse service for ALOT more money.
    I’m not here to argue just to make a point about how the clarity of Dreamhost is rare and welcome policy and not given by many other companies.
    I would also be annoyed if my site was down but problems happen anyone who has ever used a computer knows that

    Comment by John Knowles — March 26, 2008 #

  40. 40

    You gotta be kidding. You actually DID take several people’s websites out when they were suspended for “non-payment”, or have you conveniently forgot that because it didn’t fit the arc of your spiffy blog post?

    Good Christ, Josh. You pretend to be transparent about your practices, and most people swallow it, while meanwhile the way that you’re transparent is quite a bit different.

    It’s pretty rough when a shameful, inexcusable error costs you money, I know. Who could have foreseen that care and responsibility should be exercised with people’s credit and bank information? Shock!

    Comment by Mark W will probably be banned again — March 26, 2008 #

  41. 41

    Not only that, but I guess most people were getting a far better deal than me if 1,000 accounts only costs you $120,000 for a year. I was paying twice that. I certainly was never switched over to a lower pricing scheme when you offered discounted rates year after year. My price was always “Grandfathered” at the high rate I started with. For my first couple years, I was paying close to $50/month for what is now a $10/month account even when the price dropped again and again.

    I can’t remember which blog post it was where you mentioned that you (paraphrased) ‘let customers know when a cheaper hosting plan would suit their needs’, but I was sure never one of those customers.

    Comment by Mark W will probably be banned again — March 26, 2008 #

  42. 42

    To be honest I would have rather been involved in the big financial over charging mess that occured than to be currently involved in the disaster that is the Blingy Cluster.
    DH came highly recommended to me. I only hope that I get to see it!

    Comment by Purple — March 26, 2008 #

  43. 43

    That you #37. Your concerns are laid out in a manner that dreamhost should be able to respond to. If they don’t, I’d be heading out the door.
    Pretty much all the posts over on the status page seem to be nothing more than a bar fight. I can see why that is “Not an Official Way to Contact Dreamhost”.

    It’d be interesting to see how many companies actually provide a hosted, branded place for all their customers to rip on them and say the worst possible things they can come up with.

    Comment by Tom Baker — March 26, 2008 #

  44. 44

    As I’m sure you already know from this situation, a little bit of communication goes a long way. The current situation with Bling and Spanky is causing problems for your customers because the communications in not there like it was during the billing snafu.

    The dreamhostsstatus.com site is suppose to contain the status information on your systems, but with the current situations it is not being updated. When their websites and e-mail are down for hours/days and there is no updated status, the customers are going to get mad. And when you give a status, tell them what is going on, not just some canned “we’re working on it” response.

    Many of your customers are very bright people. If a hard-drive is being replaced, tell them! That at least will give them an idea of the time that will be required. Also, providing some facility to redirect requests to servers that are down to give their user’s a better error message (rather than “connection timeout”) would go a long way with the users that actually are trying to get on your customer’s websites.

    The big issue now is the lack of communication, which is giving DreamHost a bad image. The impression is that the company cannot handle the number of customers it has. Put on your “leader hat” and provide the leadership that is needed right now.

    For the record, like many of your customers, I have been in the computer industry for more than two decades. I have to deal with customers on a daily basis. Communications is key with them. Communicate with your customers now about the Bling/Spanky cluster issue like you did with the billing issue. Maybe you can keep the number of lost customers DOWN TO only 1000.

    Comment by Glenn — March 27, 2008 #

  45. 45

    I have to agree 100% with Glenn a little communication would go a long way with me. I happen to be one of the unlucky on blingy whose site is going up and down like a see saw. If there were some time frame or an end in sight notice it’d be really appreciated. Even if you could say something like “this will take another 48 hours” at least I could put up a notice for my visitors and tell them that the issue can be fixed. Some of us have very frequented websites and have been getting nasty emails about our sites being down but telling them “it’s a problem but I don’t know what it’s going to be fixed” makes us look just as bad! I’ve had 3 great months of service but this past week has had me tearing my hair out angry that I can’t take my money and run.

    I’d love for you guys to prove me wrong since the 3 people who recommended me to Dreamhost have had great service for three YEARS. I know things happen, but this is days now…and no update for over 12 hours…?

    Comment by Cerri — March 27, 2008 #

  46. 46

    #42 - right on.
    Like most of the more recent posts, I’d contend your worst snafu is currently in progress. Not only did you feel fit to move servers with a large outage, most of the servers have been cactus for days. And all we hear is “we’re working on it”. It may not be “mission critical” hosting, but just about any fault like this becomes urgent enough if it is left to fester for days on end.
    I left my previous host because of their lack of communication regarding a (in that case a multi-week) outage. It’s starting to look like it maybe occurring again….

    Comment by Gareth — March 27, 2008 #

  47. 47

    You would think a large web hosting company like DreamHost would refrain from shedding anymore light on this billing situation with a blog post like this.

    Good customer service and decent servers doesn’t matter much when you make big finical burdens for clients.

    This only makes DreamHost seem like they don’t. After all the massive overselling, lack of knowledge to run a billing system correctly, and customer service cockiness do you think clients wouldn’t question canceling?

    Comment by hostdemon.net — March 27, 2008 #

  48. 48

    > You would think a large web hosting company like
    > DreamHost would refrain from shedding anymore
    > light on this billing situation with a blog post
    > like this.

    Yeah, you’d think so. In our industry, covering up for stupid mistakes is the norm. Most would simply pretend it never even happened.

    Trust me, if I ever see that stupid Homer Simpson picture again it will be far too soon. It’s all too tempting to never mention our not-so-little mistake again.

    As you can imagine, though, there was a lot of - erm - interest in the matter among those customers impacted. I think this post can serve as a valuable cautionary tale to other companies that such mistakes can have very real bottom-line ramifications.

    If other companies were half as forthright about their screw-ups as we are about ours, things would be a far different. They would have more incentive to make changes and improvements, lest they get a well-deserved black eye.

    Then again, who knows? Maybe we should do the usual thing and hire an expensive PR firm to “manage our message” and handle damage control. Maybe we should keep these things to ourselves, wrapping them in a thick layer of BS that any customer of average intelligence will see right through. Maybe we should limit our blog posts to promotion of our services and dry essays about the industry (yawn). Maybe our co-founder, upon making a huge billing error, should just schedule another round of golf and let the rank and file take the blame.

    I can understand still being angry about what happened, but I’m not sure I can understand why anyone would prefer we just paper over it and pretend it didn’t happen.

    And it’s not as if we’ve limited ourselves to talking about it - we also implemented and outlined four specific changes that should help prevent it from ever happening again.

    Say what you will about our company or our offerings - that’s open for debate (a debate we’ll gladly host in our own weblog, mind you), but we’re not the type of company that will make a huge mistake and then just sweep it under the rug.

    - Jeff @ DreamHost Web Hosting

    Comment by Jeff @ DreamHost — March 27, 2008 #

  49. 49

    I just started with Dreamhost. My e-mail has been down since Monday. WTF? I’m trying to launch a business and I have no e-mail. The folks who designed my website said go with Dreamhost because we’ve had problems with Go Daddy. I guess I just sit and wait? Poor business practices!!!

    Comment by McGregor — March 27, 2008 #

  50. 50

    Thanks for such a witty blog update… only if you took the time you spent on this and put it into your product.

    “What we’ve always tried to focus on more (even though it might not seem like it at times!) is our hosting system’s stability, performance, and features.” is a joke if you ask me.

    I love dreamhost and all the great features you guys have but not when you do something like the spunky move.

    ps. fix it!

    Comment by Dave — March 27, 2008 #

Pages: [1] 2 » Show All

Leave a comment

XHTML: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Comments for this post will be closed on 5 May 2009.

Powered by WordPress. Pool theme by Borja Fernandez, modified by DreamHost.
Entries and comments feeds. ^Top^