Um, Whoops.

January 15, 2008 on 9:52 am | In Foobars, Insider View, Musings by Josh Jones |

The $7,500,000 finger.

Hello.. how’s your morning going?

I hope it’s been a little better than mine.

We had a teensy eensy weensy little billing error last night… my first clue something was up when I saw this morning’s daily billing report (so far): $7,500,000.

It turns out due to my excessively fat fingers, nearly every one of our customers has been seriously over-billed in the last 12 hours.

I bet when you read this part of the last newsletter:

4. New Office!

Another important thing I’ve been doing instead of writing newsletters
is looking out the window of our NEW OFFICE:

http://blog.dreamhost.com/2007/12/21/were-so-high-right-now-you-dont-even-know

If your next web hosting bill from us is mysteriously tripled, now you
know why.

.. you thought it was a joke!

Ha, the joke is on you! I guess. Um, okay, no, not really, I’m sorry.

How on earth could something like this happen?

Let Me Explain

A couple of weeks ago, just around new years, we started beefing up some of our internal “controller” servers. These are the machines that run all of our “behind-the-scenes” services; things from adding a user to registering a domain to configuring apaches to rebilling customers.

I was on a little-bit-too-long vacation, but when I got back, I noticed our daily credit card payments seemed a tad low in the new year.

So, late last week I tried re-running the billing services for all the days back three weeks or so. I knew this was safe, because after 10 years, the one thing you DO get perfect is your billing system. Our biller is pretty bug-free and robust at this point, because we’d be broke and eating bugs if it weren’t.

In fact, it’s so robust you can just run it on any day you want, and it’s safe. It won’t double-charge people and it’ll even automatically find any missing charges and catch everything up to the day you said.

Anyway, I ran it, and things were fine.. and sure enough, it caught a lot of missed payments. I didn’t have time to look into it right then, but I made a note to myself to check up on it on Monday (yesterday) and see if things were fine or still messed up.

And a terminal case it is.

Come Monday

Monday came. I checked the reports and sure enough, things were still pretty low. So I looked at the logs for some of the biller services, and I noticed they were only failing on the machines that had been recently upgraded!

That explained why we were getting some money still (since not all the controllers have been upgraded yet), but not all of it.

Anyway, it turned out there was no 64 bit version of the PFProAPI module we use to interface to the credit card transaction server. No big deal, there’s a new module that interfaces with their new and preferred https interface, and it was only a couple of lines of code to change to get us switched over!

So anyway, I made the change, and it worked, and I even tested it, and things were fine!

But then… late last night, I realized: when I re-ran those biller services last week, they must not have fixed everybody then either! It’s just that by running it again I randomly got different people being charged on the working controllers who had been assigned an upgraded (and therefore broken) one before.

So why not just run it all one more time?

Sure, it should be no problem! So I did, manually running the biller (which is normally automatically scheduled) for 2008-01-14, 2008-01-13, 2008-01-12, 2008-01-11, 2008-01-10, 2008-01-09, 2008-01-08, 2008-01-07, 2008-01-06, 2008-01-05, 2008-01-04, 2008-01-03, 2008-01-02, and 2008-01-01.

I probably should have just stopped there. But then I thought better. I thought to myself, “When did we start upgrading these controllers anyway?”

I couldn’t remember. But, since the biller is super-safe and robust anyway, I went ahead and ran it for 2008-12-31, 2008-12-30, 2008-12-29, 2008-12-28, 2008-12-27, 2008-12-26, and 2008-12-25, just for the hell of it.

Notice Anything?

Don’t feel bad if you didn’t. I kind of missed it myself.

THOSE SHOULD HAVE BEEN 2007!!

Heh, uh.. um, er.. my bad?

So what happened?

Well, that super-robust and stable biller did what it was programmed to do, it ran as though today was December 31st, 2008!

And what did it see? Well, it saw a whole lot of accounts (essentially all of them) who for some unknown, mysterious reason hadn’t been charged at all for eleven and a half months!

So off it went, busily through the night, “fixing” everything up for “today”, December 31st, 2008.

Really, it’s sort of amazing this never happened before in the last ten years.

We have a NEW SUPPORT RECORD!

There IS a bug here.

I can imagine the half second or so of thought that sprinted through the programmer’s mind when he was adding the ability to allow you to pass in what day to run the biller as though today is:

Hmm.. well, I could see us POSSIBLY wanting to be able to bill for a future date.

Well guess what… NO! We will NEVER want to rebill as though today were a day that hasn’t happened yet! But instead, somebody along the line (Sage? Me? Somebody else?) figured, “What’s the harm in keeping it flexible?”

About $7,500,000 in harm, that’s what!

The serious part.

The end to this story is that of course, I’m very very sorry, we’re very very sorry, and I’m sure you’re very very sorry this happened. I really am. I understand the sort of problems that an unexpected large charge to your credit card (or worse yet, your debit card) can cause. If the tone of this blog post seemed a little light, I apologize I don’t mean to offend and I realize how serious an issue this is. I’ve been up since 3:50am trying to undo the damage and maybe I’m a little shell-shocked.

A new service is running right now (in parallel on all the controllers) that fixes all those future charges, re-enables your account if it was erroneously suspended, and if your credit card was automatically rebilled, refunds the payment automatically. You don’t have to contact us or your bank, and you’ll get an email when your account is finished fixing up. It’s going to take several more hours to complete. There are (or were, after this incident) a lot of you these days!

If, because of this billing mistake, you somehow incurred some fees from your bank or credit card company, please let us know after tomorrow (today we are just replying to all 10,000+ billing messages with a generic explanation) and we’ll do our best to make it right for you.

And of course, the biller no longer allows dates in the future.

The moral of this story is that “flexibility” is rarely desired in programming! The less a program will accept/the less a program will do/the less options and preferences it has, the more usable it is/the more understandable it is/the more stable it is.

Tough Love

I wouldn’t want him to compile me!

When designing a program, you’ve got to make some tough decisions .. and when you really can’t decide if this is something your users will need someday, err on the side of leaving it out.

Otherwise, your users will someday err on the side of your face.

667 Comments »

RSS feed for comments on this post.

Pages: [1] 2 3 4 5 6 7 8 9 10 1114 » Show All

  1. 1

    Thanks for the explanation. Might I suggest, if you’re reading this, Josh, that users like me whose accounts were disabled for nonpayment be re-enabled? I’ve got clients screaming at me because they can’t get their e-mail!

    Comment by jason — January 15, 2008 #

  2. 2

    just checked one of my webids and it is back to normal…something must be working.

    Comment by Me — January 15, 2008 #

  3. 3

    Quote: “THOSE SHOULD HAVE BEEN 2007!!”

    and

    “The moral of this story is that “flexibility” is rarely desired in programming!”

    Nope…

    ‘The moral of this story’ is that you guys need a proofreading or double-checking system before you do anything, since using 2008 instead of 2007 is EXACTLY the type of thing that would get caught by someone else double checking your work.

    You can be sure that once the web panel is up, I’m going to make a ’suggestion’ that you guys implement exactly this sort of thing, and lets see how many Dreamhosters vote for it.

    Also, since you link to the DH Status page, you’ve no doubt read some of the comments, a great many of which requested that you NOT reply to this in a flippant manner….which is pretty much what you’ve done here.

    Comment by PROOFREAD! — January 15, 2008 #

  4. 4

    your honesty makes me so happy, you guys rule

    Comment by Shivian Balaris — January 15, 2008 #

  5. 5

    Josh - it is time you realized when the light jokey style in which you run this business is NOT the appropriate tone. Any of us involved in running a business knows when you approach your customers with a serious tone. Your customers are taking that tone with you; Time to step up and be serious and act like your company depends on it. I think it does.

    Comment by Jeremy — January 15, 2008 #

  6. 6

    Quote: “THOSE SHOULD HAVE BEEN 2007!!”

    and

    “The moral of this story is that “flexibility” is rarely desired in programming!”

    Nope…

    ‘The moral of this story’ is that you guys need a proofreading or double-checking system before you do anything, since using 2008 instead of 2007 is EXACTLY the type of thing that would get caught by someone else double checking your work.

    You can be sure that once the web panel is up, I’m going to make a ’suggestion’ that you guys implement exactly this sort of thing, and lets see how many Dreamhosters vote for it.

    Also, since you link to the DH Status page, you’ve no doubt read some of the comments, a great many of which requested that you NOT reply to this in a flippant manner….which is pretty much what you’ve done here.

    The ‘flippant’ thing would be cool if you didn’t make simple errors on a semi regular basis, but since you DO make simple errors on a regular basis, it’s not so cool…

    Comment by PROOFREAD! — January 15, 2008 #

  7. 7

    Thanks for the jokes. I’m out of here.

    Comment by thanks for the jokes — January 15, 2008 #

  8. 8

    Sooo…. is the other moral of the story that I don’t have to pay the bills that I got this morning?

    Also, I like the light style… keep it up, forget the stick-in-the-muds.

    Comment by Jayson — January 15, 2008 #

  9. 9

    That just made me laugh my ass off.

    I know you all have a lot to do in the next month, but at least you can always look back and laugh and tell your grandchildren, “Hey, when I worked in IT, we fucked up big time.”

    The 7.5 million dollar mistake… It should be a horror movie!

    Comment by Lady — January 15, 2008 #

  10. 10

    Thank you very much for the immediate detailed explanation and the quick fix. Somehow, I don’t think any other hosting company would have been this forthcoming about an error of this magnitude.

    Comment by Tim — January 15, 2008 #

  11. 11

    That just made me laugh my ass off.

    I know you all have a lot to do in the next month, but at least you can always look back and laugh and tell your grandchildren, “Hey, when I worked in IT, we fucked up big time.”

    The 7.5 million dollar mistake… It should be a horror movie!

    No one has the right to be pissed off at you for trying to put a positive spin on the whole situation, because if they know anything about the company that’s just how everyone is.

    Thanks for keeping your sense of humor. That’s the main reason I moved over here.

    Comment by Lady — January 15, 2008 #

  12. 12

    I’m assuming this doesn’t affect customers that have paid two years out? I checked my bank activity and I have seen nothing; neither in billing nor in refund. Then again, maybe my bank just hasn’t posted the updates yet

    Comment by john lascurettes — January 15, 2008 #

  13. 13

    I’m initiating a class action law suit because you made me laugh :(

    Comment by Dan — January 15, 2008 #

  14. 14

    Thanks for the comprehensive explanation. We’re all only human and we all screw up every now and again, although this was pretty spectacular ;)

    Comment by Roy — January 15, 2008 #

  15. 15

    Please, show some genuine concern for your customers. Sure, you screwed up, but I don’t see any sort of genuine concern other than the $7,500,000 in credit card debt you accidentally set off.

    The joking tone works for some errors, but how bout some actual concern for your customers instead.

    Here’s a sample.

    “Wow. Seriously, we made a gigantic mistake. Just to make it right, we’re going to…” and then somehow create a conciliatory tone. Otherwise, it just sounds like you’re mocking us.

    Comment by Randall Bennett — January 15, 2008 #

  16. 16

    Guys, I know there are people that will bitch no matter what, but I will /always/ appreciate being informed when things go wrong. This is one of the main reasons I have been with Dreamhost for years. It’s not even about the joking style, but it’s a lot of background info that makes it all seem like a reasonable mistake. Other companies give a “billing error, fixing shortly” statement, and you wonder if it was just automatically generated by their trouble ticket system.

    Thanks. Love you guys.

    Comment by spinn — January 15, 2008 #

  17. 17

    Jokes are NOT APPROPRIATE in this situation.

    Do you realize that for some credit cards, you have just pushed them over their max, meaning some people will pay fees, overage charges, higher rates PERMANENTLY, and their credit scores could be affected? Will you be reimbursing those charges?

    I don’t carry a balance on my card so it won’t be a problem for me in that regard, but we’re house-hunting, and if a an almost $600 charge came up on the day that I was having my credit checked for mortgage purposes, it might affect what rate I am offered. Will you be reimbursing people for those potential losses?

    This isn’t a joke. IT’S FRAUD, even if you fix it, because in the time that it takes you to fix it, people are losing money. I called my credit card and they said they couldn’t see the charge I got an email about, so it was either already gone or hadn’t gone through. If there isn’t some serious restitution and firings over this, I will call them back to red flag a potential fraud situation. Start taking this seriously or you will risk the credit card companies blocking you from making charges in the future.

    Comment by Kimberly Chapman — January 15, 2008 #

  18. 18

    Oh yeah, how about that mass email alerting customers to the problem?

    Comment by thanks for the jokes — January 15, 2008 #

  19. 19

    Hey, sorry your rent bounced, but here’s a picture of Homer Simpson and some lulzy hipster prose.

    Joking around might not be the best technique when you are messing with people’s money.

    Comment by Eric Ogilvie — January 15, 2008 #

  20. 20

    Well if you guys need a place to go after fleeing from this billing crisis, you should definietly head over to Lunarpages:

    “For all DH webmasters looking for an escape, Lunarpages has created a coupon code just for you! Use code DHRefugees to receive $40 off hosting with Lunarpages and we will also buy out the remainder of your contract with DreamHost! (Up to 9 months!) If you have several months left on your contract with DH, sign up for Lunarpages and email us your billing receipt (not the triple billing ones.. lol) and we will add that extra time to your account with us!”

    Comment by Tiara — January 15, 2008 #

  21. 21

    And this is why I switched to Westhost this morning…

    http://affiliates.westhost.com/z/14/CD970/

    Any word from Visa or Mastercard as to whether or not they are going to close down your mechant account for the sheer amount you managed to charge fraudulently this morning?

    Comment by Michael Dulanson — January 15, 2008 #

  22. 22

    I have to echo the sentiment that your flippant tone isn’t appreciated, Josh. Sorry, it’s just not. There’s nothing funny about this. I appreciate trying to save face and inject some humor, but you screwed up, dude. Be a man about it.

    Your explanation doesn’t make sense, either. If the biller is SUPER ROBUST AND SAFE and you MERELY ran it for all of 2008 accidentally, why was I billed for 10 out of 12 months in 2008 TWICE? I have TWO charges for february, TWO charges for march, TWO for april, etc. Can you explain that? If you can’t, then I think you have more troubleshooting to do. I’m seriously hoping that I’m not only refunded one “set” of erroneous charges when there are TWO.

    Comment by Mark W — January 15, 2008 #

  23. 23

    Eh… Well, as an old computer-nerd, i too know that these things might happend. But, i have to agree to that the information about this HUGE blunder, isnt very pleasing.
    Why, why why, didnt you send information-email to all, as soon as the error was noticed?? Your pages are almost impossible (guess very big pressure!) to get in to. And NO information at “standard” login, the webpanel? Bad… very bad.
    Took me 30 min… to get into ANY information-pages!!
    Let this be a lesson. 10 years witout big problems dont mean you can be lazy… Keep it up, this was first major error i seen, so you DO a good job, after all. :D

    Comment by Foggy — January 15, 2008 #

  24. 24

    I just wanted to agree with Jayson and say that I appreciate the honesty. I personally like the light style too — it’s a bit refreshing in today’s world of closed-up, tight-lipped companies who don’t even say anything when these kind of things happen.

    I know you’ll never say, but I’d love to know how much this cost you guys. At a 2% credit card discount rate (wayyy conservative), that’s $150k right there. That doesn’t include the average $.25 charge per transaction, nor does it include the charges you must be paying to refund all of that money. I hope you worked something out with the bank that actually does the charging…

    Comment by Andrew — January 15, 2008 #

  25. 25

    Well I’m glad you take this seriously enough to write a funny blog post about it instead of working to refund the money with all available time/energy. I think we see more clearly where your priorities are.

    Comment by LukeP — January 15, 2008 #

  26. 26

    Another vote for the not-yet-sent mass email. I found out about the issue first from Digg, and then from my bank’s fraud department. Still not a single message from someone at DH.

    Comment by don — January 15, 2008 #

  27. 27

    Thanks for keeping us posted.

    Also, congrats on being runner up for The Editor’s Choice of The Best Use of a Corporate Blog in 2007 in the Performancing Blog Awards.

    http://performancing.com/performancing-awards/performancing-blog-awards-2007-winners

    Comment by Dee — January 15, 2008 #

  28. 28

    OH LOL WE FUCKED UP YOUR BANK ACCOUNT

    Comment by Mark — January 15, 2008 #

  29. 29

    This morning I received a mail from you in which you stated that my renew wouldn’t be possible because you weren’t be able to debt my credit card. Since I used a rechargeable credit card to make my subscription AND I remember I subscribed in november (but not which year), first of all I thought my credit card needed some recharge. So I put 200 euros on it and wrote an email telling you that I was not able to “make payement” from the control panel: if you wanted my money then take it.
    I made the same mistake as you: didn’t notice that my account was just one year old and I payed for two; didn’t notice the 2008-thing. And I trust you so deeply that I didn’t doubt for a second it was my fault.

    But now… well, fellow customers, let’s see the half-full glass: we currently own money we thought weren’t ours anymore, and it’s BIG money: 7.500.000 USD :-D

    (I really like your service and the style you run it, light and transparent)

    Comment by spider — January 15, 2008 #

  30. 30

    >There are a lot of you these days!

    Somewhat fewer by Friday I imagine.

    Seriously though, guy, please explain why there are so many DUPLICATE 2008 charges for so many people. I appreciate you explaining even as much as you did when we really, in the long run, pay so little (until today) for the service, but your explanation doesn’t add up.

    It’s also pretty uncool to be adding so many people to the SPAM filter on the dreamhoststatus.com blog. People are (understandably and justifiably) pissed.

    I love Dreamhost as a company. I think you guys are great and have the right attitude, except for today. You really dropped the ball. You need to know when the joking and frivolity end and when “oh shit we’re SO sorry and here’s how we’re gonna make it up to you” begins.

    Comment by Mark W — January 15, 2008 #

  31. 31

    That’s great, but in the meantime, my account has been suspended when I’m up to date on my billing. Yet more downtime from you guys. This past year has been horrible, especially compared to prior years where I saw practically no issues with my hosting.

    Yet I still pay you the same regardless. With all the issues and downtime we’ve seen, we should be seeing partial credit. As a senior support engineer for a software company, I know the appropriate way to handle customers, and you guys are screwing the pooch in that regard.

    I’d come work for you and help you fix it, but you’re down there in evil SoCal :P

    Comment by Kristie L. — January 15, 2008 #

  32. 32

    This is not a funny, lighthearted matter and is NOTHING TO JOKE ABOUT. Let me explain why:

    I now have hundreds of dollars in overdraft charges a result of this error. Is DH going to issue a credit to me for these third party charges that resulted from the error as well?

    I WILL BE CANCELING MY ACCOUNT AS SOON AS THIS ERROR IS CORRECTED - due to a complete lack of a professionalism in the midst of this HUGE mistake. I had to search the web for 2 hours before I was able to find the information on this error. Dreamhost being aware of the problem on their end should have been responsible enough to contact me directly and immediately letting me know the error was being fixed. They should have sent an email to every customer about this if they couldn’t determine which accounts were billed incorrectly. That would be professional…. whereas the staff making jokes, trying to make light of things with funny pictures, and other nonsense to your newsletters (where I actually found them enjoyable). WHEN YOU MAKE AN ALMOST 8 MILLION DOLLAR ERROR YOU SHOULD DO A LITTLE MORE THAN MAKE A BLOG ENTRY ENTITLED ‘WHOOPSIE’.

    There is now way DH could make up for the time and money I have lost trying to deal with this issue. Not only do I have to now contact my mortgage company and explain why dreamhost will be unable to make my mortgage payment on time this month… but now I am the one responsible for compensating my own clients regarding this matter, along being unable to regain the time and money spent this morning when trying to deal with the mess this whole situation created.

    I will do everything in my power to ensure than some additional form of compensation is given to me as a result of the damage it has done to my credit, my wallet, and my schedule. I DEFINITELY FEEL A CLASS ACTION LAW SUIT IS IN ORDER and I plan to contact my lawyer about this ASAP.

    Comment by CLASS ACTION LAW SUIT — January 15, 2008 #

  33. 33

    As my former host told me when they lost all of the data for over 300 users over two weeks, “It just isn’t possible to send an email to everyone.”

    And it’s true. In order to email everyone and let them know what is up, they have to check their emails, a lot. Some people still don’t know what is going on in the first place, because they haven’t looked in the inbox.

    That’s why DH has the status domain. It’s up all the time in order to let everyone know what is going on and that’s why they suggest that you bookmark it, just in case you need to check.

    I would suggest that anyone look there if anything seems buggy. If there’s nothing, then check the support. If there is nothing on any of those, then you can complain and whine and go on.

    Just understand that this is a big issue for DH and that they are more stressed about it than any of you. Look at the facts: If they owe even half of the people billed $20 for overcharge fees, then that all comes from the company.

    The people that work here are *people.* They have jobs and bills too, so it’s not as though they don’t understand how important it is to everyone.

    Comment by Lady — January 15, 2008 #

  34. 34

    Wow, what some people will congratulate you for…I do not appreciate what others are calling “honesty.” You had no choice but to admit to this mistake. It wasn’t your *choice* to be honest; your choice was in how you handled it, and considering the damage done to people’s bank accounts and the downtime on their sites for being “delinquent,” there is no way that pretending this is cute or funny is appropriate. It’s also not a good use of your time to find cartoons to post while my bank account still reflects that you haven’t reversed the DH charge and my site is still down and I have yet to receive an email acknowledging this to go with the email I received this morning where you double-billed me for a year.

    Comment by Deir — January 15, 2008 #

  35. 35

    You smug fucks.

    I am literally shaking my head in disgust at the unprofessional way you’re handling this. The overwhelming sentiment in the comments on your status.dreamhost.com posting is that you DON’T RESPOND TO THIS MATTER IN YOUR USUAL, CUTESY,”OOPS WE MADE A MISTAKE…BUT OH, ARENT WE SO CLEVER WITH ALL THESE FUNNY IMAGES AND SARCAST-IRONIC WRITING STYLE?”

    Suffice it to say, you’ve cost a lot of people a lot of money, in addition to jeopardizing thier credit ratings and account statuses. Joking around about this incident just proves to your customers that you guys are not a serious, professional hosting company.

    If you had simply posted a no-nonsense, “we fucked up, here’s how and why we fucked up, here’s how we’re gonna make it up to you, and here’s our formal, public promise that it will never happen again” blog post, you would have instantly regained my trust, respect and confidence that while you guys are a bunch of fun lovin’, non-uptight corporate types, you still know that you have professional responsibilities, and are mature enough to know when it’s clearly not the time to be joking around with your customers.

    The fact that you responded this way means that I am seriously going to consider pulling all my accounts with Dreamhost. I can’t trust my money to a bunch of fucking adolescents who don’t understand the concept of professionalism and respect for it’s customers.

    Westhost, here I come!

    Comment by bookem_dan-o — January 15, 2008 #

  36. 36

    Heheh, “whoops” indeed. =) Didn’t take my bank account into the red, happily enough. No harm, no foul.

    Comment by Adam Backstrom — January 15, 2008 #

  37. 37

    Wow, Ms. Chapman really needs to calm down. It’s also clear that she’s just plain wrong.

    “Do you realize that for some credit cards, you have just pushed them over their max, meaning some people will pay fees, overage charges, higher rates PERMANENTLY, and their credit scores could be affected? Will you be reimbursing those charges?”

    See, when you hit your credit limit, charges get declined. That’s how credit cards work, and that’s exactly what happened to me when Dreamhost tried to bill me for two years today. They were only able to charge me for $5 instead of $238.80, and it’s already been refunded. Problem solved. As for “permanently” higher rates… if your credit card company won’t remove a default rate after six months of good payment history and staying under the limit, you need to find a different CC company.

    “…we’re house-hunting, and if a an almost $600 charge came up on the day that I was having my credit checked for mortgage purposes, it might affect what rate I am offered.”

    Actually, it wouldn’t, since credit card companies don’t report balances to the credit bureaus on a daily basis. But thanks for playing.

    “I called my credit card and they said they couldn’t see the charge I got an email about, so it was either already gone or hadn’t gone through. If there isn’t some serious restitution and firings over this, I will call them back to red flag a potential fraud situation.”

    DH is fixing it and said there was no need to call your CC company/bank, so you wasted your time there. Good luck with claiming “fraud” when DH has already corrected the situation on their own. It’s pretty clear nobody’s getting fired. I also see that you have made no mention of leaving DH over this, so I presume you’re just fine with the way this has been resolved.

    Comment by Tim — January 15, 2008 #

  38. 38

    Will you double our disk space and bandwidth for the mistake? :)

    Comment by Charles — January 15, 2008 #

  39. 39

    TO “Lady” ^^
    “Cant email 300 people” and “People have to check email every min”? Thats just SO wrong…
    Im in communities with 20000+ members.. and THEY can sen correction-emails right away. If a big webhosting-company cant, thats just bizarr..
    Offcorse people wont be so angry, IF they had seen a email right away… Thats just what makes so many angry.

    Comment by Foggy — January 15, 2008 #

  40. 40

    I like Charles’ idea!

    :)

    Comment by Lady — January 15, 2008 #

  41. 41

    My dear sweeties at Dreamhost. I’ve been with you since 1998 and have enjoyed your enjoyable newsletters that made me laugh when no one else could do that, however, I now take exception with how you have made a joke out of this billing issue. This time you have gone too far. No, I will not be leaving you, but you MUST know that regarding an issue of this importance, it is NOT THE TIME TO BE JOKING AROUND. I’m a senior citizen…listen to your elders and just be good kids, ok??

    Comment by Helen Chambers — January 15, 2008 #

  42. 42

    Additionally, what’s with these complaints about not being sent an email regarding the problem? I received one:

    “Ack. Through a COMPLETE bumbling on our part, we’ve accidentally attempted to charge you for the ENTIRE year of 2008 (and probably 2009!) ALREADY (it was all due to a fat finger)!

    We’re really really realllly embarassed about this, but you have nothing to worry about. Please ignore any confusing billing messages you may have received recently; we’ve already removed all those bum future charges on your account (#XXXXXX) and fixed everything up.

    Thank you very very much for your patience with this.. we PROMISE this won’t happen again. There’s no need to reply to this message unless of course you have any other questions at all!

    Sincerely,
    The Foolish DreamHost Billing Team!”

    Comment by Tim — January 15, 2008 #

  43. 43

    Charges that go over the limit on your card aren’t always declined, it depends on the company, your account history, and your specific terms of use. My card has been overdrawn before even though it should have been declined.

    In this case, though, they hit me in the checking account. Thank God that my mortgage check cleared anyway, since I only keep about enough to cover outstanding checks and gas and groceries in there.

    There’s still nothing to explain why some people got billed TWICE for 2008 (or almost twice, some months were billed only once, most twice). That’s not explainable by a simple erroneous single run for the wrong year. Something else happened. I’d feel a lot more confident that this is going to be fixed if they’d explain why their explanation doesn’t match up to WHAT ACTUALLY HAPPENED.

    Comment by Mark W — January 15, 2008 #

  44. 44

    I’m going to agree with some of the people on here–the flippant attitude probably wasn’t the best choice for a blog post about something this serious.

    I also would have appreciated a mass email saying “Hey, there’s been a mistake, we’re in the process of fixing it, visit the status page for updates”. I knew about the status page, but plenty of people don’t.

    I won’t be canceling my account or moving anything–I like DH and I’m willing to give you guys the benefit of the doubt. You worked with me to move my site here even after my previous host didn’t want to release my domain name back to me, and I’ve had nothing but good service since signing up. But if there’s anything you guys learn from this, it’s that your attitude towards informing your customers about HUGE stuff like this needs to be a little more professional.

    Comment by Liz — January 15, 2008 #

  45. 45

    For such a vast problem, I don’t think the jokey tone really serves you well. I usually don’t mind it, but in this case, it is a bad idea.

    Comment by e — January 15, 2008 #

  46. 46

    My company has one website hosted with dreamhost. An inconsequential website where if it goes down, no big deal. All other sites are hosted with proper, professional web hosts. I think now you can see why.

    For those who enjoy all problems (and excuses) being so transparent, well I’ll tell you what I enjoy: my websites working without constant problems and exuses to those problems. I enjoy when my other sites are up and running the promised uptime, with no problems. No constant jokey blog posts about “oopsie, we f-ed up again!”

    I would say you get what you pay for, but with the amount of hosting competition, there are many less expensive yet much better hosting companies out there. But if you enjoy the abuse…stay with Dreamhost. You two deserve each other.

    Comment by LG — January 15, 2008 #

  47. 47

    Thank you for making me finally get off my butt and looking for new hosting provider for my websites.

    This could have been handled in a professional manner, but instead, you insisted on turning it into a big joke.

    Anyway, guess what? It’s not funny.

    Comment by Robin — January 15, 2008 #

  48. 48

    Thanks for getting this taken care of as quickly as you’ve been able to. Please remind the people crying “fraud” that it’s NOT fraud, as they gave you their CC numbers, and you were technically billing them for future services.

    Comment by Anna — January 15, 2008 #

  49. 49

    Actually, my credit card will allow me to go over the limit and not decline what they deem to be “reasonable” charges (which can be well over $100 past my credit limit, I’ve found).

    And I have to imagine most people have debit cards set up for this instead. I do. And they deny things even less.

    It’s very possible that these things weren’t denied whatsoever. And in fact, let’s say this went through and only brought me $5 negative? Let’s say then that I have several other pending charges that hadn’t cleared yet. Now they’ve cleared and the bank thinks I’ve tried to take money from nothing 5 times. They’ll charge me 5 overdraft fees.

    So even once Dreamhost’s charge disappears (and I’d be really damn surprised if anyone’s bank does this instantaneously to begin with) they’re likely not just going to get rid of the overdraft fees. You ARE going to have to call and explain the situation. So calling the bank is hardly a stupid thing to do to begin with, you’re setting up the story for this to be less of an issue to clear up later on.

    God forbid people get upset and freaked out about what has amounted to almost $8 million dollars in extra charges.

    Comment by Tony — January 15, 2008 #

  50. 50

    Tim, I will be anticipating my email on this as well.

    Comment by Helen Chambers — January 15, 2008 #

Pages: [1] 2 3 4 5 6 7 8 9 10 1114 » Show All

Leave a comment

XHTML: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Comments for this post will be closed on 17 April 2009.

Powered by WordPress. Pool theme by Borja Fernandez, modified by DreamHost.
Entries and comments feeds. ^Top^