Um, Whoops.
January 15, 2008 on 9:52 am | In Foobars, Insider View, Musings by Josh Jones |
Hello.. how’s your morning going?
I hope it’s been a little better than mine.
We had a teensy eensy weensy little billing error last night… my first clue something was up when I saw this morning’s daily billing report (so far): $7,500,000.
It turns out due to my excessively fat fingers, nearly every one of our customers has been seriously over-billed in the last 12 hours.
I bet when you read this part of the last newsletter:
4. New Office!
Another important thing I’ve been doing instead of writing newsletters
is looking out the window of our NEW OFFICE:http://blog.dreamhost.com/2007/12/21/were-so-high-right-now-you-dont-even-know
If your next web hosting bill from us is mysteriously tripled, now you
know why.
.. you thought it was a joke!
Ha, the joke is on you! I guess. Um, okay, no, not really, I’m sorry.
How on earth could something like this happen?
Let Me Explain
A couple of weeks ago, just around new years, we started beefing up some of our internal “controller” servers. These are the machines that run all of our “behind-the-scenes” services; things from adding a user to registering a domain to configuring apaches to rebilling customers.
I was on a little-bit-too-long vacation, but when I got back, I noticed our daily credit card payments seemed a tad low in the new year.
So, late last week I tried re-running the billing services for all the days back three weeks or so. I knew this was safe, because after 10 years, the one thing you DO get perfect is your billing system. Our biller is pretty bug-free and robust at this point, because we’d be broke and eating bugs if it weren’t.
In fact, it’s so robust you can just run it on any day you want, and it’s safe. It won’t double-charge people and it’ll even automatically find any missing charges and catch everything up to the day you said.
Anyway, I ran it, and things were fine.. and sure enough, it caught a lot of missed payments. I didn’t have time to look into it right then, but I made a note to myself to check up on it on Monday (yesterday) and see if things were fine or still messed up.
Come Monday
Monday came. I checked the reports and sure enough, things were still pretty low. So I looked at the logs for some of the biller services, and I noticed they were only failing on the machines that had been recently upgraded!
That explained why we were getting some money still (since not all the controllers have been upgraded yet), but not all of it.
Anyway, it turned out there was no 64 bit version of the PFProAPI module we use to interface to the credit card transaction server. No big deal, there’s a new module that interfaces with their new and preferred https interface, and it was only a couple of lines of code to change to get us switched over!
So anyway, I made the change, and it worked, and I even tested it, and things were fine!
But then… late last night, I realized: when I re-ran those biller services last week, they must not have fixed everybody then either! It’s just that by running it again I randomly got different people being charged on the working controllers who had been assigned an upgraded (and therefore broken) one before.
So why not just run it all one more time?
Sure, it should be no problem! So I did, manually running the biller (which is normally automatically scheduled) for 2008-01-14, 2008-01-13, 2008-01-12, 2008-01-11, 2008-01-10, 2008-01-09, 2008-01-08, 2008-01-07, 2008-01-06, 2008-01-05, 2008-01-04, 2008-01-03, 2008-01-02, and 2008-01-01.
I probably should have just stopped there. But then I thought better. I thought to myself, “When did we start upgrading these controllers anyway?”
I couldn’t remember. But, since the biller is super-safe and robust anyway, I went ahead and ran it for 2008-12-31, 2008-12-30, 2008-12-29, 2008-12-28, 2008-12-27, 2008-12-26, and 2008-12-25, just for the hell of it.
Notice Anything?
Don’t feel bad if you didn’t. I kind of missed it myself.
THOSE SHOULD HAVE BEEN 2007!!
Heh, uh.. um, er.. my bad?
So what happened?
Well, that super-robust and stable biller did what it was programmed to do, it ran as though today was December 31st, 2008!
And what did it see? Well, it saw a whole lot of accounts (essentially all of them) who for some unknown, mysterious reason hadn’t been charged at all for eleven and a half months!
So off it went, busily through the night, “fixing” everything up for “today”, December 31st, 2008.
Really, it’s sort of amazing this never happened before in the last ten years.
There IS a bug here.
I can imagine the half second or so of thought that sprinted through the programmer’s mind when he was adding the ability to allow you to pass in what day to run the biller as though today is:
Hmm.. well, I could see us POSSIBLY wanting to be able to bill for a future date.
Well guess what… NO! We will NEVER want to rebill as though today were a day that hasn’t happened yet! But instead, somebody along the line (Sage? Me? Somebody else?) figured, “What’s the harm in keeping it flexible?”
About $7,500,000 in harm, that’s what!
The serious part.
The end to this story is that of course, I’m very very sorry, we’re very very sorry, and I’m sure you’re very very sorry this happened. I really am. I understand the sort of problems that an unexpected large charge to your credit card (or worse yet, your debit card) can cause. If the tone of this blog post seemed a little light, I apologize I don’t mean to offend and I realize how serious an issue this is. I’ve been up since 3:50am trying to undo the damage and maybe I’m a little shell-shocked.
A new service is running right now (in parallel on all the controllers) that fixes all those future charges, re-enables your account if it was erroneously suspended, and if your credit card was automatically rebilled, refunds the payment automatically. You don’t have to contact us or your bank, and you’ll get an email when your account is finished fixing up. It’s going to take several more hours to complete. There are (or were, after this incident) a lot of you these days!
If, because of this billing mistake, you somehow incurred some fees from your bank or credit card company, please let us know after tomorrow (today we are just replying to all 10,000+ billing messages with a generic explanation) and we’ll do our best to make it right for you.
And of course, the biller no longer allows dates in the future.
The moral of this story is that “flexibility” is rarely desired in programming! The less a program will accept/the less a program will do/the less options and preferences it has, the more usable it is/the more understandable it is/the more stable it is.
Tough Love
When designing a program, you’ve got to make some tough decisions .. and when you really can’t decide if this is something your users will need someday, err on the side of leaving it out.
Otherwise, your users will someday err on the side of your face.
January 15th, 2008 at 10:08 am
Thanks for the explanation. Might I suggest, if you’re reading this, Josh, that users like me whose accounts were disabled for nonpayment be re-enabled? I’ve got clients screaming at me because they can’t get their e-mail!
January 15th, 2008 at 10:10 am
just checked one of my webids and it is back to normal…something must be working.
January 15th, 2008 at 10:11 am
Quote: “THOSE SHOULD HAVE BEEN 2007!!”
and
“The moral of this story is that “flexibility” is rarely desired in programming!”
Nope…
‘The moral of this story’ is that you guys need a proofreading or double-checking system before you do anything, since using 2008 instead of 2007 is EXACTLY the type of thing that would get caught by someone else double checking your work.
You can be sure that once the web panel is up, I’m going to make a ’suggestion’ that you guys implement exactly this sort of thing, and lets see how many Dreamhosters vote for it.
Also, since you link to the DH Status page, you’ve no doubt read some of the comments, a great many of which requested that you NOT reply to this in a flippant manner….which is pretty much what you’ve done here.
January 15th, 2008 at 10:12 am
your honesty makes me so happy, you guys rule
January 15th, 2008 at 10:13 am
Josh - it is time you realized when the light jokey style in which you run this business is NOT the appropriate tone. Any of us involved in running a business knows when you approach your customers with a serious tone. Your customers are taking that tone with you; Time to step up and be serious and act like your company depends on it. I think it does.
January 15th, 2008 at 10:18 am
Quote: “THOSE SHOULD HAVE BEEN 2007!!”
and
“The moral of this story is that “flexibility” is rarely desired in programming!”
Nope…
‘The moral of this story’ is that you guys need a proofreading or double-checking system before you do anything, since using 2008 instead of 2007 is EXACTLY the type of thing that would get caught by someone else double checking your work.
You can be sure that once the web panel is up, I’m going to make a ’suggestion’ that you guys implement exactly this sort of thing, and lets see how many Dreamhosters vote for it.
Also, since you link to the DH Status page, you’ve no doubt read some of the comments, a great many of which requested that you NOT reply to this in a flippant manner….which is pretty much what you’ve done here.
The ‘flippant’ thing would be cool if you didn’t make simple errors on a semi regular basis, but since you DO make simple errors on a regular basis, it’s not so cool…
January 15th, 2008 at 10:21 am
Thanks for the jokes. I’m out of here.
January 15th, 2008 at 10:22 am
Sooo…. is the other moral of the story that I don’t have to pay the bills that I got this morning?
Also, I like the light style… keep it up, forget the stick-in-the-muds.
January 15th, 2008 at 10:23 am
That just made me laugh my ass off.
I know you all have a lot to do in the next month, but at least you can always look back and laugh and tell your grandchildren, “Hey, when I worked in IT, we fucked up big time.”
The 7.5 million dollar mistake… It should be a horror movie!
January 15th, 2008 at 10:24 am
Thank you very much for the immediate detailed explanation and the quick fix. Somehow, I don’t think any other hosting company would have been this forthcoming about an error of this magnitude.
January 15th, 2008 at 10:24 am
That just made me laugh my ass off.
I know you all have a lot to do in the next month, but at least you can always look back and laugh and tell your grandchildren, “Hey, when I worked in IT, we fucked up big time.”
The 7.5 million dollar mistake… It should be a horror movie!
No one has the right to be pissed off at you for trying to put a positive spin on the whole situation, because if they know anything about the company that’s just how everyone is.
Thanks for keeping your sense of humor. That’s the main reason I moved over here.
January 15th, 2008 at 10:26 am
I’m assuming this doesn’t affect customers that have paid two years out? I checked my bank activity and I have seen nothing; neither in billing nor in refund. Then again, maybe my bank just hasn’t posted the updates yet
January 15th, 2008 at 10:29 am
I’m initiating a class action law suit because you made me laugh :(
January 15th, 2008 at 10:29 am
Thanks for the comprehensive explanation. We’re all only human and we all screw up every now and again, although this was pretty spectacular ;)
January 15th, 2008 at 10:29 am
Please, show some genuine concern for your customers. Sure, you screwed up, but I don’t see any sort of genuine concern other than the $7,500,000 in credit card debt you accidentally set off.
The joking tone works for some errors, but how bout some actual concern for your customers instead.
Here’s a sample.
“Wow. Seriously, we made a gigantic mistake. Just to make it right, we’re going to…” and then somehow create a conciliatory tone. Otherwise, it just sounds like you’re mocking us.
January 15th, 2008 at 10:30 am
Guys, I know there are people that will bitch no matter what, but I will /always/ appreciate being informed when things go wrong. This is one of the main reasons I have been with Dreamhost for years. It’s not even about the joking style, but it’s a lot of background info that makes it all seem like a reasonable mistake. Other companies give a “billing error, fixing shortly” statement, and you wonder if it was just automatically generated by their trouble ticket system.
Thanks. Love you guys.
January 15th, 2008 at 10:30 am
Jokes are NOT APPROPRIATE in this situation.
Do you realize that for some credit cards, you have just pushed them over their max, meaning some people will pay fees, overage charges, higher rates PERMANENTLY, and their credit scores could be affected? Will you be reimbursing those charges?
I don’t carry a balance on my card so it won’t be a problem for me in that regard, but we’re house-hunting, and if a an almost $600 charge came up on the day that I was having my credit checked for mortgage purposes, it might affect what rate I am offered. Will you be reimbursing people for those potential losses?
This isn’t a joke. IT’S FRAUD, even if you fix it, because in the time that it takes you to fix it, people are losing money. I called my credit card and they said they couldn’t see the charge I got an email about, so it was either already gone or hadn’t gone through. If there isn’t some serious restitution and firings over this, I will call them back to red flag a potential fraud situation. Start taking this seriously or you will risk the credit card companies blocking you from making charges in the future.
January 15th, 2008 at 10:30 am
Oh yeah, how about that mass email alerting customers to the problem?
January 15th, 2008 at 10:30 am
Hey, sorry your rent bounced, but here’s a picture of Homer Simpson and some lulzy hipster prose.
Joking around might not be the best technique when you are messing with people’s money.
January 15th, 2008 at 10:31 am
Well if you guys need a place to go after fleeing from this billing crisis, you should definietly head over to Lunarpages:
“For all DH webmasters looking for an escape, Lunarpages has created a coupon code just for you! Use code DHRefugees to receive $40 off hosting with Lunarpages and we will also buy out the remainder of your contract with DreamHost! (Up to 9 months!) If you have several months left on your contract with DH, sign up for Lunarpages and email us your billing receipt (not the triple billing ones.. lol) and we will add that extra time to your account with us!”
January 15th, 2008 at 10:32 am
And this is why I switched to Westhost this morning…
http://affiliates.westhost.com/z/14/CD970/
Any word from Visa or Mastercard as to whether or not they are going to close down your mechant account for the sheer amount you managed to charge fraudulently this morning?
January 15th, 2008 at 10:32 am
I have to echo the sentiment that your flippant tone isn’t appreciated, Josh. Sorry, it’s just not. There’s nothing funny about this. I appreciate trying to save face and inject some humor, but you screwed up, dude. Be a man about it.
Your explanation doesn’t make sense, either. If the biller is SUPER ROBUST AND SAFE and you MERELY ran it for all of 2008 accidentally, why was I billed for 10 out of 12 months in 2008 TWICE? I have TWO charges for february, TWO charges for march, TWO for april, etc. Can you explain that? If you can’t, then I think you have more troubleshooting to do. I’m seriously hoping that I’m not only refunded one “set” of erroneous charges when there are TWO.
January 15th, 2008 at 10:33 am
Eh… Well, as an old computer-nerd, i too know that these things might happend. But, i have to agree to that the information about this HUGE blunder, isnt very pleasing.
Why, why why, didnt you send information-email to all, as soon as the error was noticed?? Your pages are almost impossible (guess very big pressure!) to get in to. And NO information at “standard” login, the webpanel? Bad… very bad.
Took me 30 min… to get into ANY information-pages!!
Let this be a lesson. 10 years witout big problems dont mean you can be lazy… Keep it up, this was first major error i seen, so you DO a good job, after all. :D
January 15th, 2008 at 10:35 am
I just wanted to agree with Jayson and say that I appreciate the honesty. I personally like the light style too — it’s a bit refreshing in today’s world of closed-up, tight-lipped companies who don’t even say anything when these kind of things happen.
I know you’ll never say, but I’d love to know how much this cost you guys. At a 2% credit card discount rate (wayyy conservative), that’s $150k right there. That doesn’t include the average $.25 charge per transaction, nor does it include the charges you must be paying to refund all of that money. I hope you worked something out with the bank that actually does the charging…
January 15th, 2008 at 10:37 am
Well I’m glad you take this seriously enough to write a funny blog post about it instead of working to refund the money with all available time/energy. I think we see more clearly where your priorities are.
January 15th, 2008 at 10:37 am
Another vote for the not-yet-sent mass email. I found out about the issue first from Digg, and then from my bank’s fraud department. Still not a single message from someone at DH.
January 15th, 2008 at 10:37 am
Thanks for keeping us posted.
Also, congrats on being runner up for The Editor’s Choice of The Best Use of a Corporate Blog in 2007 in the Performancing Blog Awards.
http://performancing.com/performancing-awards/performancing-blog-awards-2007-winners
January 15th, 2008 at 10:37 am
OH LOL WE FUCKED UP YOUR BANK ACCOUNT
January 15th, 2008 at 10:38 am
This morning I received a mail from you in which you stated that my renew wouldn’t be possible because you weren’t be able to debt my credit card. Since I used a rechargeable credit card to make my subscription AND I remember I subscribed in november (but not which year), first of all I thought my credit card needed some recharge. So I put 200 euros on it and wrote an email telling you that I was not able to “make payement” from the control panel: if you wanted my money then take it.
I made the same mistake as you: didn’t notice that my account was just one year old and I payed for two; didn’t notice the 2008-thing. And I trust you so deeply that I didn’t doubt for a second it was my fault.
But now… well, fellow customers, let’s see the half-full glass: we currently own money we thought weren’t ours anymore, and it’s BIG money: 7.500.000 USD :-D
(I really like your service and the style you run it, light and transparent)
January 15th, 2008 at 10:39 am
>There are a lot of you these days!
Somewhat fewer by Friday I imagine.
Seriously though, guy, please explain why there are so many DUPLICATE 2008 charges for so many people. I appreciate you explaining even as much as you did when we really, in the long run, pay so little (until today) for the service, but your explanation doesn’t add up.
It’s also pretty uncool to be adding so many people to the SPAM filter on the dreamhoststatus.com blog. People are (understandably and justifiably) pissed.
I love Dreamhost as a company. I think you guys are great and have the right attitude, except for today. You really dropped the ball. You need to know when the joking and frivolity end and when “oh shit we’re SO sorry and here’s how we’re gonna make it up to you” begins.
January 15th, 2008 at 10:40 am
That’s great, but in the meantime, my account has been suspended when I’m up to date on my billing. Yet more downtime from you guys. This past year has been horrible, especially compared to prior years where I saw practically no issues with my hosting.
Yet I still pay you the same regardless. With all the issues and downtime we’ve seen, we should be seeing partial credit. As a senior support engineer for a software company, I know the appropriate way to handle customers, and you guys are screwing the pooch in that regard.
I’d come work for you and help you fix it, but you’re down there in evil SoCal :P
January 15th, 2008 at 10:40 am
This is not a funny, lighthearted matter and is NOTHING TO JOKE ABOUT. Let me explain why:
I now have hundreds of dollars in overdraft charges a result of this error. Is DH going to issue a credit to me for these third party charges that resulted from the error as well?
I WILL BE CANCELING MY ACCOUNT AS SOON AS THIS ERROR IS CORRECTED - due to a complete lack of a professionalism in the midst of this HUGE mistake. I had to search the web for 2 hours before I was able to find the information on this error. Dreamhost being aware of the problem on their end should have been responsible enough to contact me directly and immediately letting me know the error was being fixed. They should have sent an email to every customer about this if they couldn’t determine which accounts were billed incorrectly. That would be professional…. whereas the staff making jokes, trying to make light of things with funny pictures, and other nonsense to your newsletters (where I actually found them enjoyable). WHEN YOU MAKE AN ALMOST 8 MILLION DOLLAR ERROR YOU SHOULD DO A LITTLE MORE THAN MAKE A BLOG ENTRY ENTITLED ‘WHOOPSIE’.
There is now way DH could make up for the time and money I have lost trying to deal with this issue. Not only do I have to now contact my mortgage company and explain why dreamhost will be unable to make my mortgage payment on time this month… but now I am the one responsible for compensating my own clients regarding this matter, along being unable to regain the time and money spent this morning when trying to deal with the mess this whole situation created.
I will do everything in my power to ensure than some additional form of compensation is given to me as a result of the damage it has done to my credit, my wallet, and my schedule. I DEFINITELY FEEL A CLASS ACTION LAW SUIT IS IN ORDER and I plan to contact my lawyer about this ASAP.
January 15th, 2008 at 10:41 am
As my former host told me when they lost all of the data for over 300 users over two weeks, “It just isn’t possible to send an email to everyone.”
And it’s true. In order to email everyone and let them know what is up, they have to check their emails, a lot. Some people still don’t know what is going on in the first place, because they haven’t looked in the inbox.
That’s why DH has the status domain. It’s up all the time in order to let everyone know what is going on and that’s why they suggest that you bookmark it, just in case you need to check.
I would suggest that anyone look there if anything seems buggy. If there’s nothing, then check the support. If there is nothing on any of those, then you can complain and whine and go on.
Just understand that this is a big issue for DH and that they are more stressed about it than any of you. Look at the facts: If they owe even half of the people billed $20 for overcharge fees, then that all comes from the company.
The people that work here are *people.* They have jobs and bills too, so it’s not as though they don’t understand how important it is to everyone.
January 15th, 2008 at 10:42 am
Wow, what some people will congratulate you for…I do not appreciate what others are calling “honesty.” You had no choice but to admit to this mistake. It wasn’t your *choice* to be honest; your choice was in how you handled it, and considering the damage done to people’s bank accounts and the downtime on their sites for being “delinquent,” there is no way that pretending this is cute or funny is appropriate. It’s also not a good use of your time to find cartoons to post while my bank account still reflects that you haven’t reversed the DH charge and my site is still down and I have yet to receive an email acknowledging this to go with the email I received this morning where you double-billed me for a year.
January 15th, 2008 at 10:43 am
You smug fucks.
I am literally shaking my head in disgust at the unprofessional way you’re handling this. The overwhelming sentiment in the comments on your status.dreamhost.com posting is that you DON’T RESPOND TO THIS MATTER IN YOUR USUAL, CUTESY,”OOPS WE MADE A MISTAKE…BUT OH, ARENT WE SO CLEVER WITH ALL THESE FUNNY IMAGES AND SARCAST-IRONIC WRITING STYLE?”
Suffice it to say, you’ve cost a lot of people a lot of money, in addition to jeopardizing thier credit ratings and account statuses. Joking around about this incident just proves to your customers that you guys are not a serious, professional hosting company.
If you had simply posted a no-nonsense, “we fucked up, here’s how and why we fucked up, here’s how we’re gonna make it up to you, and here’s our formal, public promise that it will never happen again” blog post, you would have instantly regained my trust, respect and confidence that while you guys are a bunch of fun lovin’, non-uptight corporate types, you still know that you have professional responsibilities, and are mature enough to know when it’s clearly not the time to be joking around with your customers.
The fact that you responded this way means that I am seriously going to consider pulling all my accounts with Dreamhost. I can’t trust my money to a bunch of fucking adolescents who don’t understand the concept of professionalism and respect for it’s customers.
Westhost, here I come!
January 15th, 2008 at 10:44 am
Heheh, “whoops” indeed. =) Didn’t take my bank account into the red, happily enough. No harm, no foul.
January 15th, 2008 at 10:44 am
Wow, Ms. Chapman really needs to calm down. It’s also clear that she’s just plain wrong.
“Do you realize that for some credit cards, you have just pushed them over their max, meaning some people will pay fees, overage charges, higher rates PERMANENTLY, and their credit scores could be affected? Will you be reimbursing those charges?”
See, when you hit your credit limit, charges get declined. That’s how credit cards work, and that’s exactly what happened to me when Dreamhost tried to bill me for two years today. They were only able to charge me for $5 instead of $238.80, and it’s already been refunded. Problem solved. As for “permanently” higher rates… if your credit card company won’t remove a default rate after six months of good payment history and staying under the limit, you need to find a different CC company.
“…we’re house-hunting, and if a an almost $600 charge came up on the day that I was having my credit checked for mortgage purposes, it might affect what rate I am offered.”
Actually, it wouldn’t, since credit card companies don’t report balances to the credit bureaus on a daily basis. But thanks for playing.
“I called my credit card and they said they couldn’t see the charge I got an email about, so it was either already gone or hadn’t gone through. If there isn’t some serious restitution and firings over this, I will call them back to red flag a potential fraud situation.”
DH is fixing it and said there was no need to call your CC company/bank, so you wasted your time there. Good luck with claiming “fraud” when DH has already corrected the situation on their own. It’s pretty clear nobody’s getting fired. I also see that you have made no mention of leaving DH over this, so I presume you’re just fine with the way this has been resolved.
January 15th, 2008 at 10:45 am
Will you double our disk space and bandwidth for the mistake? :)
January 15th, 2008 at 10:49 am
TO “Lady” ^^
“Cant email 300 people” and “People have to check email every min”? Thats just SO wrong…
Im in communities with 20000+ members.. and THEY can sen correction-emails right away. If a big webhosting-company cant, thats just bizarr..
Offcorse people wont be so angry, IF they had seen a email right away… Thats just what makes so many angry.
January 15th, 2008 at 10:49 am
I like Charles’ idea!
:)
January 15th, 2008 at 10:49 am
My dear sweeties at Dreamhost. I’ve been with you since 1998 and have enjoyed your enjoyable newsletters that made me laugh when no one else could do that, however, I now take exception with how you have made a joke out of this billing issue. This time you have gone too far. No, I will not be leaving you, but you MUST know that regarding an issue of this importance, it is NOT THE TIME TO BE JOKING AROUND. I’m a senior citizen…listen to your elders and just be good kids, ok??
January 15th, 2008 at 10:49 am
Additionally, what’s with these complaints about not being sent an email regarding the problem? I received one:
“Ack. Through a COMPLETE bumbling on our part, we’ve accidentally attempted to charge you for the ENTIRE year of 2008 (and probably 2009!) ALREADY (it was all due to a fat finger)!
We’re really really realllly embarassed about this, but you have nothing to worry about. Please ignore any confusing billing messages you may have received recently; we’ve already removed all those bum future charges on your account (#XXXXXX) and fixed everything up.
Thank you very very much for your patience with this.. we PROMISE this won’t happen again. There’s no need to reply to this message unless of course you have any other questions at all!
Sincerely,
The Foolish DreamHost Billing Team!”
January 15th, 2008 at 10:50 am
Charges that go over the limit on your card aren’t always declined, it depends on the company, your account history, and your specific terms of use. My card has been overdrawn before even though it should have been declined.
In this case, though, they hit me in the checking account. Thank God that my mortgage check cleared anyway, since I only keep about enough to cover outstanding checks and gas and groceries in there.
There’s still nothing to explain why some people got billed TWICE for 2008 (or almost twice, some months were billed only once, most twice). That’s not explainable by a simple erroneous single run for the wrong year. Something else happened. I’d feel a lot more confident that this is going to be fixed if they’d explain why their explanation doesn’t match up to WHAT ACTUALLY HAPPENED.
January 15th, 2008 at 10:50 am
I’m going to agree with some of the people on here–the flippant attitude probably wasn’t the best choice for a blog post about something this serious.
I also would have appreciated a mass email saying “Hey, there’s been a mistake, we’re in the process of fixing it, visit the status page for updates”. I knew about the status page, but plenty of people don’t.
I won’t be canceling my account or moving anything–I like DH and I’m willing to give you guys the benefit of the doubt. You worked with me to move my site here even after my previous host didn’t want to release my domain name back to me, and I’ve had nothing but good service since signing up. But if there’s anything you guys learn from this, it’s that your attitude towards informing your customers about HUGE stuff like this needs to be a little more professional.
January 15th, 2008 at 10:52 am
For such a vast problem, I don’t think the jokey tone really serves you well. I usually don’t mind it, but in this case, it is a bad idea.
January 15th, 2008 at 10:52 am
My company has one website hosted with dreamhost. An inconsequential website where if it goes down, no big deal. All other sites are hosted with proper, professional web hosts. I think now you can see why.
For those who enjoy all problems (and excuses) being so transparent, well I’ll tell you what I enjoy: my websites working without constant problems and exuses to those problems. I enjoy when my other sites are up and running the promised uptime, with no problems. No constant jokey blog posts about “oopsie, we f-ed up again!”
I would say you get what you pay for, but with the amount of hosting competition, there are many less expensive yet much better hosting companies out there. But if you enjoy the abuse…stay with Dreamhost. You two deserve each other.
January 15th, 2008 at 10:53 am
Thank you for making me finally get off my butt and looking for new hosting provider for my websites.
This could have been handled in a professional manner, but instead, you insisted on turning it into a big joke.
Anyway, guess what? It’s not funny.
January 15th, 2008 at 10:54 am
Thanks for getting this taken care of as quickly as you’ve been able to. Please remind the people crying “fraud” that it’s NOT fraud, as they gave you their CC numbers, and you were technically billing them for future services.
January 15th, 2008 at 10:54 am
Actually, my credit card will allow me to go over the limit and not decline what they deem to be “reasonable” charges (which can be well over $100 past my credit limit, I’ve found).
And I have to imagine most people have debit cards set up for this instead. I do. And they deny things even less.
It’s very possible that these things weren’t denied whatsoever. And in fact, let’s say this went through and only brought me $5 negative? Let’s say then that I have several other pending charges that hadn’t cleared yet. Now they’ve cleared and the bank thinks I’ve tried to take money from nothing 5 times. They’ll charge me 5 overdraft fees.
So even once Dreamhost’s charge disappears (and I’d be really damn surprised if anyone’s bank does this instantaneously to begin with) they’re likely not just going to get rid of the overdraft fees. You ARE going to have to call and explain the situation. So calling the bank is hardly a stupid thing to do to begin with, you’re setting up the story for this to be less of an issue to clear up later on.
God forbid people get upset and freaked out about what has amounted to almost $8 million dollars in extra charges.
January 15th, 2008 at 10:55 am
Tim, I will be anticipating my email on this as well.
January 15th, 2008 at 10:56 am
Thanks for the jokes Josh… =).
January 15th, 2008 at 10:56 am
I had moved two clients over to lunar pages in august…what a mistake that was!!! their service totally sucks…i have one VPS account, and one business hosting account. the VPS is the most royal pain i have ever encountered. I can’t run all of my services over there, so opted to renew my account here when renewal came up.
I am not leaving DH again. I know this will be a costly mistake for dreamhost but i have every confidence that they will pay all charges incurred.
Seems the script is already working and peoples charges are being reversed.
January 15th, 2008 at 10:56 am
I am glad the situation is being taken care of. Thank you Josh and dreamhost and lets try to prevent this from occuring again. -Shawn
January 15th, 2008 at 10:57 am
Joke around all you want.
If people think that being ultra serious is going to make this any better then they are retarded.
I do think you are shitting your pants right now about all the customers you are losing, but I will be staying I think.
January 15th, 2008 at 10:58 am
All these people throwing around the world Fraud: I don’t think that word means what you think it means. Fraud, usually, involves something called intent, which was definetly not present in this situation. It was an ACCIDENT, folks, and DH didn’t charge everyone for a year or two on purpose - it was not their INTENT to overcharge their customers $7,500,000. Besides, they were working through the night to figure out wtf was up. There is a lot of people that would have waited until it was time to start work.
On the other hand, my account is still suspended for nonpayment. I only owe $10, and the payment has been made through PayPal. PayPal’s holding it, and would appreciate it if I can get it up and going. I can’t do any work until it is.
Thanks,
Amanda
January 15th, 2008 at 10:59 am
OH NOES TEH JOKES R NOT FUNNEHS!!
You know, I got my normal billing notice in December and was pretty short on cash so I figured I’d pay it this month. Then this happens and I was thinking “Wow, this is pretty short notice! But that’s cool, I’ll just go ahead and send in my mon….” *email 3 and 4 arrive* “Whoa…. I can’t pay THAT much!” xD
The greatest thing is - I’m not even using DreamHost’s services anymore - I have a Dedicated Server and I’m loving it. But what I love even more, and the only reason why I continue to pay for my DreamHost account, is Dreamhost’s sense of humor and friendliness with its customers. My dedicated host, while not nearly as awesome, still is very friendly and forthcoming with responses and results, but not at all like DreamHost.
I personally found Josh’s explanation to be FAR less “funny” than his normal blog posts. He also explained things in a great amount of detail that I would not expect from any other company I’ve ever worked for or with.
To Summarize - thank you for being so awesome DreamHost!! The little mistakes aside. ;)
January 15th, 2008 at 11:00 am
Last month I accidentally deleted a whole blog site. I know your pain. Thanks for responding so quickly.
January 15th, 2008 at 11:01 am
Well, I’ll be leaving. And I don’t have autobill and I paid via PayPal so I didn’t even get charged.
I am leaving, even with six months left on my plan, because I do not like the light tone. This is not transparent, this is moronic. This is high school.
When you deal with people’s money, you take it seriously. There’s a few shades of gray inbetween IBM and the Simpsons. For example:
“Dear Valued Dreamhost Customer:
We made a major mistake last night. We’ll be writing up a detailed post on exactly what happened, and posting it to our blog (located at: blahblahblah) if you would like to know more.
Everyone was billed in error yesterday, some of you multiple times. We are currently in the process of refunding all of the money taken from people we shouldn’t have taken it from. If you feel you need to call your bank and credit card company, of course you have every right to do so. I can assure you that we are dealing with each and every overage as quickly as possible, and that you don’t need to make that call, but we would never presume to tell you how to handle your personal (or company) finances.
There’s no excuse for the error except human error. We realize that we have put people into difficult financial and professional situations, and we deeply regret that our actions have caused any inconvenience whatsoever.
Some of you will already see refunds. If you have gotten a refund, and there are other extenuating circumstances (overdraft fees, exchange rates), please contact us at this special address we’ve set up to deal with individual issues from the billing snafu: blah@blah.com.
You should see your refund by DATE, TIME. If you do not have your refund by DATE, TIME, please contact the billing address above and we’ll get right on it.
Everyone here at Dreamhost will be working around the clock to make this right for everyone, down to the last customer.
Also, we will be compensating every last customer for the inconvenience caused by this issue. We haven’t had time to discuss what form that would take yet, and we hope you understand when I say that we need to focus on other issues first. But we will be providing every customer with some type of credit or other gesture to apologize for our error.
Finally, I’d like to thank you for being a Dreamhost customer. There’s no shortage of hosting choices and you choose to give us your money every month. We wouldn’t be here without your patronage and we understand that.
Sincerely,
Dreamhost Poobah
January 15th, 2008 at 11:01 am
Bravo guys.
It’s an unfortunate accident, and it happens. But at least you guys kept us somewhat informed. And at least you’re taking responsibility for it!
For those that complain about it: suck it up. You’re not paying for Digg.com’s hosting, or some other media conglomerate. You’re paying a great price for a small firm in CA to do it, with 10 years of hosting experience, and a limited number of years in management.
Mistakes happen. But at least DH is willing to suck it up and correct it. And they didn’t have to tell us exactly what went wrong. They could’ve just given a corporate answer of “there was an error and blah blah blah”.
At least their mistake was owned up to, unlike ExistHost: http://www.dustinmeadows.com/existhost.html
I’m sticking with you guys. I know a number of others will too.
January 15th, 2008 at 11:04 am
your explanation doesn’t cut it; you should be fired.
January 15th, 2008 at 11:04 am
The $7.5 million question is: Why does your “bug-free” billing system not do sanity checking on the date it’s passed?
January 15th, 2008 at 11:09 am
Guys, seriously, get a grip. I’m pissed. I’m super pissed. I think this explanation is crap (the charges I was billed, along with many others, don’t jibe with his explanation of one extra run) but the FBI? Fraud? You’re pissing up a rope.
The flippant, joking attitude, the half-assed explanation, and the overall joviality about the whole thing are all WAY inexcusable, but fraud? Federal charges? Class action suits? You’re all nuts.
January 15th, 2008 at 11:09 am
I think you should double our bandwidth allowance as a sorry. I am currently using 803824908239403275932058329053247329057329480935757328057203720349823094823095723094823904823904832490238492384023948903248239048329408324908324902384932048095709128501295832059324752093482309572309523849320597235098234-9023570239589023483948394839848349839849384209483.2 TB a second on my blog about pantyliners.
January 15th, 2008 at 11:11 am
Once again, people using the terms “stealing”, “fraud”, or “theft” are missing one key factor of all of those - intent. You can’t go to the FBI because someone accidentally hit the wrong thing when processing billing. This was an accident, and Dreamhost is actually rectifying the situation by issuing refunds and possibly covering people’s bank fees. If it were “theft” or “stealing” or “fraud”, they wouldn’t be fixing their error.
It’s perfectly reasonable to be upset and to even feel like the explanation is inadequate. It’s batsh*t insane to be screaming about calling the police because SOME1 ON TEH INTARWEBS STOLEZ MAH MUNEY! Relax. Take a Valium, have a cup of chamomile tea - whatever. Act like a rational person and not a nutjob. Voice your opinions calmly and like an adult, and for the love of whatever god you believe in, stop invoking the T.L.A.s (Three Letter Acronyms).
January 15th, 2008 at 11:11 am
Wow, I’m glad I dropped you guys months ago. Even though you DON’T host my sites anymore, the way which you handled this is pretty insulting.
January 15th, 2008 at 11:12 am
Josh, how the hell DARE you joke and make light of this situation? You have just caused millions of dollars in damage, and you think this is funny?? Your butt should have been fired, immediately.
I can’t believe Dreamhost is this callous about the damage they - and you, specifically, Josh - have caused.
Once my account is cleared from your mistake, I will be leaving - and I hope many others will leave as well. You do not deserve to remain in business.
I would have been willing to give you the benefit of the doubt, allowing that people make mistakes, but reading your blog convinces me - you guys just don’t get it. This is like laughing at a car crash.
January 15th, 2008 at 11:13 am
Our website and email has been down for over a week due to a billing issue and we can get no response from you guys. This is crazy!
January 15th, 2008 at 11:14 am
[quote]This is you STEALING money and totally screwing many of us to the tune of $7,500,000 dollars.
[/quote]
What money are they stealing? Do you think DH benefits from an overdraft charge caused by YOUR bank? Or a charge for YOUR credit card company?
Yes, it’s a big mistake. Yes, they should refund or reimburse all charges caused by their mistake. Yes, this blog post is stupid in tone and, in addition, has a huge copyright violation by using a copyrighted and trademarked image.
But there’s no fraud here. And no theft.
January 15th, 2008 at 11:15 am
X Employee,
I wish I was getting paid for saying this stuff!
Can I start charging for being nice now?
January 15th, 2008 at 11:16 am
The worst part is the first part:
>>”Hello.. how’s your morning going?
I hope it’s been a little better than mine.”
No, it’s been WORSE. You have $7.5 million more that you should.
I’m overdrawn (I was charged more than $500!!!).
What a smug MF.
I’m backin’ up and movin’ out…
January 15th, 2008 at 11:18 am
Why is it my job to continually tell my clients who have accounts with you what is going on? Today they all get an email saying their being charged - and they don’t know why. YOU STILL HAVEN’T SENT OUT AN EMAIL TO THOSE SAME PEOPLE TELLING THEM YOU GOOFED - NOR HAVE YOU OFFERED ANY INCENTIVE FOR THEM TO STAY WITH YOU.
Not everyone checks your blog to find out if they emails they received were in error.
January 15th, 2008 at 11:20 am
Considering how many companies stonewall, lie, and self-congratulate in their “we screwed up” announcements, I thought this was perfectly fine. Amusing even. I’m a programmer. I’ve made mistakes. Granted, this was a pretty colossal mistake, but still, it’s a learning experience, and if they fire the guy, then that experience is lost.
Regarding the humor, I think a lot of you forget that you’re reading a blog. If you want the official company statement/apology, that would be found at status.dreamhost.com, not on the blog.
January 15th, 2008 at 11:20 am
- Got billing email. Said “WTF?”
- Checked my Dreamhost Panel’s Billing section. No help. Said “OMG WTF?”
- Thought “Wait, Dreamhost is actually good at COMMUNICATING with its users about problems like this. I bet the Blog has something to say!”
- Read this post. Felt better.
Finding information here, tongue-in-cheek or not, is infinitely better than being left in the dark. I like a touch of humor, but certainly understand that I may be in the minority here. Regardless, I trust that you’ll take care of everyone.
As a fellow Harvey Mudd alum, I’ll just treat this as a really elaborate prank. Just remember that pranks must be reversed within 24 hours of notification to do so. So, uh, good luck :)
January 15th, 2008 at 11:21 am
I’ll add my 2 cents to the voices of people asking for a more professional tone to this particular situation… might be time to hire a PR company to help Josh “write” his blog posts?
January 15th, 2008 at 11:21 am
I have gotten every email that they have sent out.
If people don’t use the email addresses they have registered, there is no way that they can magically be contacted.
If they are your clients, don’t you probably have their current emails? Instead of yelling at a blog that won’t give you a response, please go ahead and direct them in the direction dreamhoststatus.com so they understand.
Afterall, that is the reasonable thing to do.
January 15th, 2008 at 11:21 am
I would like to hear some indication that the double-billing I and others have observed is being addressed, since it’s hard to see how that fits in with the supposed otherwise-bug-free nature of the thing.
I don’t care about the jokes, and I’m not upset about the temporary charges. But since the people who’ve already mentioned this issue above also complained about the jokeyness, maybe you missed it.
January 15th, 2008 at 11:22 am
Thanks, you just put me overdrawn for the month.
£80 I now have to find.
January 15th, 2008 at 11:22 am
What an incredibly unprofessional and flippant explanation. Yeah, hah hah funny until YOU have $400 unauthorized dollars come out of YOUR account, sending your ass into overdraft, getting you hit with additional charges from your bank that, I suspect, will take another several days to weeks to get returned, if ever.
Not. Funny. And this response is flat out insulting to me, someone who’s been a loyal customer for over five years. Mistakes happen. Rude, unprofessional excuses don’t have to.
January 15th, 2008 at 11:26 am
Well, at least we get confirmation that they are going to compensate people for all the overdraft charges they caused.
It still pisses me off that something so careless can happen. And I really don’t appreciate the joking attitude toward all of this. This isn’t a cute situation. The fun ends when you’re dealing with people’s finances. Not everybody has enough money to cover an erroneous $300 charge.
January 15th, 2008 at 11:29 am
Wow, a cute “whoopsy daisy” blog post, complete with pictures from google images? Didn’t see THAT coming at ALL.
Are you a real company staffed by grownups or a bunch of fucking kindergarteners? Jesus.
Also, bye. Taking my fucking hosting elsewhere, you incompetent pricks.
January 15th, 2008 at 11:30 am
I enjoyed reading the responses, partially because I work for another hosting firm that didn’t make this mistake, and partially because I love seeing people cry. “OH NO, YOU MADE ME OVERDRAWN” followed by “I’M CALLING MY LAWYER. CLASS ACTION OMG” is hilarious. Does your lawyer work for free? Can I have his name, I’d like to file a bunch of silly lawsuits for free, too.
HMC Prank rule in effect. 24 hour clock to reverse prank starts … Now.
January 15th, 2008 at 11:30 am
Normally, this kind of thing would piss me off badly; however, I’ve dealt with Dreamhost’s customer service a few times in the past and have always come away satisfied. This is an example of what I really like about Dreamhost: regular REAL people providing a service. If some other company to whom I normally pay a bill — Comcast or the electric company, for example — made this kind of mistake, they would certainly delay and obfuscate to cover their tails. Dreamhost, however, grows a pair and admits the mistake — in a funny blog entry, nonetheless.
I suppose the people who have clients who can’t access sites and email may feel more inspired to anger than me because my account is for hosting my site and some sites for some friends’ bands. Also, my service wasn’t interrupted. Yeah, I probably got a huge fee from my bank because of this, but I know Dreamhost will take care of it. Like Josh requested, I’ll wait until later in the week to ask about it.
Thanks for owning up to it Dreamhost!
January 15th, 2008 at 11:31 am
This whole fiasco I’m sure underscores for your customers (some of whom are certain to be former customers) that you need a phone number. The we’re-dotcomers-and-don’t-need-phones approach to customer service is infuriating horseshit.
And the light, joket tone is OK most of the time, but when you have tied up 7.5 million of your customers funds, perhaps causing some of them to default on paying other bills, wrack up bank fees, spend how ever much time on the phone or via email sorting out what you find to be a chuckle-worthy oopsie, it’s really in poor taste. You suck if you don’t pen a heartfelt apology right now.
January 15th, 2008 at 11:31 am
This is rediculous.
Oops! I made a booboo. 7.5 million dollars is a shit-ton of money.
and Kristie L., you can shut your stupid fucking face. Accidental here isnt shit. If you accidentally killed someone you’re still screwed and you’re apparently not one who got fucked over.
January 15th, 2008 at 11:31 am
Cutriss, is right people. You are reading a blog. Not the offical site. Yes, they were wrong. But you know something. This is why I don’t let companies charge my credit card every month automatically. I didn’t have to go through what you all did. I can sit back and laugh my brains out at you all having panic attacks while I sit back in my nice cool logically sound brain and find out exactly why my hosting bill was so high. This is why you don’t let companies automatically charge your credit cards. I hope you all learned a lesson. DH sends out a billing invoice every month telling you that you need to pay your hosting fees. That is when I pay them. I don’t have to worry, I have no problems what so ever. This morning when I say the email I just opened my account, seen the error and laughed. Then came here to read what they had to say and then read all your comments. So I really do hope you all learned not to let Companies automatically charge your credit cards. This is what could happen.
January 15th, 2008 at 11:33 am
Leaving has it exactly correct. Had the response from the Dreamhost team, and Josh in particular, been as professional and concilliatory as the imaginary letter Leaving penned, then far fewer people would be outraged. Yes, mistakes happen. But how a company chooses to deal with those mistakes is what separates the winners from the losers. Advising people NOT to call their banks is pretty thin beer, folks. Sounds like your next hire should be for a Corporate Communications officer or similar - because you guys have lost far more than what this will cost you in immediate financial impact - you’ve lost the goodwill of a hell of a number of customers, and that’s not something you’ll regain without some serious “shaping up”.
January 15th, 2008 at 11:35 am
P.S. have fun reading the accounts of your dumb-ass post in the major media outlets.
January 15th, 2008 at 11:35 am
This conversation is funnier than the original blog. I would agree that maybe the blog post was trying a little too much to be “funny” though a bit of light heartedness is still good. As for credit card usage, you use it, you must also realize the risk just like those who accept them take a risk. Finally, your bank or credit card holder probably had a security breech and you have never been informed of it… One just needs to be prepared.
BTW Josh and friends, keep up the good work. You have the best service for the best price and saves my companies thousands each year for what they want to do. However, a little more discretion in some of these blogs and a lot less “filler”.
January 15th, 2008 at 11:38 am
Is it the same fat finger that deals with Rails issues?
January 15th, 2008 at 11:39 am
Many folks have the autobilling coming directly from their accounts - as soon as the bill hits, the money is gone - real money that was probably there to pay other bills, OR an instant overdraft of their bank account or overlimit on their credit card account.
This is more serious to the customers than a laugh it off, gee isn’t that funny, ha ha blog post will cover.
In many cases you may have actually damaged the customer and caused serious problems.
Josh, a professional response to this would help you a lot, as would a proper public plan on how to handle the fallout.
At the very least you probably need to be ready to cover overdraft charges or overlimit charges is a customer is able to present you with simple proof that your billing error caused it.
Secondly, it would be a good idea for your to have your lawyers whip up a standard letter for your customers to use in explaining that YOU caused the problems with their accounts, not poor planning on the customer’s part.
Lastly, you need to reconsider the flippant way you tend to blow off problems, especially ones this seriously affecting your customers.
(Now I need to head for my account control panel and turn OFF auto billing…)
Tom
January 15th, 2008 at 11:39 am
Are you going to leave DreamHost? Yes/No…
Vote on this topic here: http://nachostatic.com/polls/dh080115.html
January 15th, 2008 at 11:39 am
Hey Josh,
congratulations!
You’ve managed to piss off nearly all of your customers, not just by overcharging them, but by handling it like a fool.
Oh, but your explanation was SO FUNNY! Right?
Think of how many customers and money this is going to cost you. Now, THAT’S FUNNY!
January 15th, 2008 at 11:41 am
The only way this would be fraud is if the money wasn’t refunded…
I think you prbably did cost people alot of money in overdraft fees due to an unexepected charge (you charged me $166). It’s amazing how quickly the bank can really jack you when the overdraft fees start piling up. I just wrote $1200 in checks to pay bills yesterday and the charge really had me freaked out this morning… I had to borrow money to make sure it was covered. I won’t be getting any overdraft charges now… but I’ve never even heard of any company making this kind of mistake before and the fact that it happened is just a huge RED FLAG… if you can make a mistake like this… could you make a mistake and lose all of our credit card information to some loser?
Go on and joke about it some more… if you don’t see some sort of legal action over this I will be surprised.
January 15th, 2008 at 11:42 am
“and Kristie L., you can shut your stupid fucking face. Accidental here isnt shit. If you accidentally killed someone you’re still screwed and you’re apparently not one who got fucked over.”
Oh, please. A billing error isn’t even in the same damned ballpark as someone dying. You’re seriously blowing this out of proportion. A billing error can be corrected, for one. Once someone’s dead, it’s not reversible. And that’s the point Dreamhost is making - they ARE fixing the problem.
And for the record: yes, I DID wake up this morning to a scary email regarding hundreds of dollars I supposedly owed Dreamhost. Of course, I also don’t put things on auto-pay for just this reason - errors happen. It hasn’t even been 24 hours and they’re already on top of rectifying the situation. You’re being unreasonable, and screaming and acting like a child isn’t going to make the process go any faster.
January 15th, 2008 at 11:42 am
Ouch. Just ouch!
I’ve checked my Google Checkout page and all seems right. I also deleted my Credit Card info, just in case…
January 15th, 2008 at 11:44 am
I must agree with quite a few previous comments here (even if I won’t yell out “fraud” or “call the FBI”); There is a time for jokes, and there is a time for being serious. This is a time for option 2.
I can surely understand the comedic values here, but good comedy has always been about… timing.
January 15th, 2008 at 11:50 am
Dreamhost managed to charge me a mere $763.20 - which apparently got refunded a mere ten hours later. I need to double-check my credit card statements to make doubly sure there are no strange currency conversion problems.
Apart from a confusing email with a disconcertingly large number in it, it’s not caused me any hassle. But I was lucky - and I’m still off to find another web-host. One with fewer awkward downtimes, general reliability problems, shutting-my-database-down-without-warning incidents, severe billing issues, you name it…
Sorry.
The Homer Simpson didn’t help either.
January 15th, 2008 at 11:50 am
Nice to see that Josh takes the accidental billing of $7,500,000 so seriously. Customers–;
January 15th, 2008 at 11:52 am
I certainly appreciate the style in which these posts are written, especially as it counterbalances the seriousness of the situation. This is exactly what we need.
I know many, many companies who respond to a mistake like this by denying it’s existence and waiting for people to contact them about the huge charge on their statements.
The fact remains that Dreamhost made a mistake, have told us about it and are doing everything they can to put things right.
Congratulations Dreamhost for letting us know what happened and what is going on in an entertaining way.
I know that I will be staying.
(P.S. I DON’T work for them)
January 15th, 2008 at 11:52 am
Kristie L.:
By the way, your site is down. Ha.
January 15th, 2008 at 11:54 am
Yeah, this “Look at us, we’re not corporate stiffs” bullshit is uncalled for in a situation like this.
I want to know what they are going to do about this, and more importantly, what they are going to do to keep me as a customer. “I’m sorry” just isnt going to cut it this time. They better be prepared to cough up free hosting and disk/bandwidth upgrades *WITHOUT BEING ASKED*. And if they want to complain about losing money for free hosting, they can offset it with Josh’s salary, since I’m sure people are already asking for his head, and it needs to be offered to calm down the masses ready to leave.
January 15th, 2008 at 11:56 am
Wow - I didn’t know that there were so many 12 year olds using Dreamhost.
Yeah, there was a huge mistake. I was charged, too. They are fixing it. Yes, humor isn’t appropriate.
For all of you whining about lawsuits, FBI, robberies, etc. Grow up and get a life. This is going to cost Dreamhost big time in the long run. Lost customers, CC fees, reputation, etc. They are going to take care of your overdraft fees. Your credit rating will not suffer.
Everybody fucks up. I know you have. Get over yourself.
January 15th, 2008 at 11:57 am
Oh, Dreamhost. The love/hate relationship we share is really getting poked and prodded today. I’m glad things are getting straightened out, but damn, this is a big dropping of the ball. I’m not going to add anything that hasn’t been said already, so I’ll try to keept he digression down, but I just wanted to get my voice out there as well. I’ve made some big screw-ups in my career (none to the tune of $7.5M mind you, but they probably felt just as stressful) so I have some kind of idea what you guys are going through. I do empathize. But, being the one who’s on the receiving side of this raping, yeah I’m a little irked, but all is not lost. Please, if you learn nothing else from this or in 2008 as a whole, learn to be more god damned careful!
January 15th, 2008 at 12:01 pm
Get over it, people. You’re not curing cancer with your website hosted for $10/month. You may have incurred some inconvenience but DH is doing what it can to remedy the situation. It’s not the end of the world.
January 15th, 2008 at 12:02 pm
David…
“Kristie L.:
By the way, your site is down. Ha.”
Wow. What incredible powers of observation. This is because due to the billing glitch, my account is flagged as suspended, something which many other people are also experiencing. I also mentioned my account suspension in my first comment on this very blog entry.
January 15th, 2008 at 12:02 pm
I am not a Dreamhost Employee. I am a consumer. The total of billing I got from them is $1,642.85.
I am also one who wasn’t horribly affected. I do feel badly for those of you who were. I will say, “I know that Dreamhost will fully make this right by everyone.”
I don’t consider it fraud. CNN won’t either! Dreamhost shows transparency when it comes to their mistakes. They allow us to openly bitch about what is going on and don’t delete posts that are negative. Not saying that is any reason to make you feel better. But I believe DH to be an honest, open company who will make this good to their consumers, both who are staying and those who are leaving.
To those moaning about the moaners, money is not lucrative to everyone and some people live day to day. This most assuredly has left some people in tears today. Give them a little sympathy and let them cry.
Good luck to all. I, again, will be staying, as I believe in Dreamhost.
PS. The humor doesn’t bother me either. This is the unofficial blog for dreamhost. The official one is http://dreamhoststatus.com I would find it offensive if they were as lighthearted there, but they apologize profusely and have stated they will make this right to their users.
January 15th, 2008 at 12:04 pm
I really shouldn’t be sticking my nose into this, but I seriously did consider DH for a while before settling on Hm. And lemme tell you, I’m glad I did.
Plus, what kind of attitude is this for a “professional” business?
“We had a teensy eensy weensy little billing error last night…”
Uh… lol?
January 15th, 2008 at 12:04 pm
To the people bitching and moaning about the way in which Josh presented the problem… let it go. There was a mistake made, large as it may be, it’s still a mistake. There was nothing careless about it, the proverbial shit tends to happen. When I design an ad for a customer and don’t type the phone number correctly, they get a credit - I didn’t do it on purpose, as I’m sure Josh didn’t purposely overbill 7.5M$. Our only flaw is being cursed with fallibility ;]
What’s truly unfortunate is that the majority of these comments that I’ve read (and I’ve read most) are whining about the professionalism, or lack there of, in which Josh handled the situation. C’mon, seriously? I understand the stress of overdraft and other fees following such a mistake, and in fact I’m hoping a few things don’t ‘clear’ so I can avoid all of that. But people are really angry for how he addressed it? You all need to take a close look at yourselves in the mirror, pissed as you may be, and ask if you HONESTLY would have been more pleased with a vague acknowledgement to this fiasco.
“We’ve encountered a serious problem with our billing in which many of you have been charged without your authorization. Please be patient while we remedy this problem.”
Would that have made you feel better? Seriously? DreamHost said the problem is being fixed. They also told individuals who acrued bank an/or credit card penalties to contact them for a resolution tomorrow. If you can’t be happy with someone not only detailing how the billing system works and then pointing out the flaw that led to this ordeal, then you’re probably not happy with anything.
Sincerely,
Tired-of-people-bitching
January 15th, 2008 at 12:04 pm
Um… Mike, some of us have the higher-end accounts with Dreamhost with 20+ websites hosted on them. It’s not just the personal website market that’s been hit. I have a business account with Dreamhost and I’m paying more than $10/month. So we got hit pretty hard by this “oops.”
January 15th, 2008 at 12:07 pm
Barbara:
The header on the page looks like it says “official dreamhost blog” to me.
January 15th, 2008 at 12:07 pm
Yeah, this was a major screw up.
But Dreamhost is doing the right thing.
A lot of you screamers, however, really screwed up…
But first - this wasn’t fraud. This wasn’t a scam. This isn’t being taken lightly by Dreamhost.
Fraud: Fraud requires intent. This isn’t fraud, it was a mistake. They’re reversing the charges. Get over it.
Taken lightly: This isn’t the “official status page”, it’s a blog. Read the official status page - it’s really and sincerely apologetic. Get over it, again.
Dreamhost has already admitted its error. They have, de facto, therefore admitted responsibility for any charges stemming from this over billing.
So get your paperwork together that shows proof that their over charge is responsible for your overdrafts and submit it to Dreamhost. If they fail to reimburse you, then scream bloody murder.
Most important lesson that all of you screaming about overdraft charges should take home from this:
DO NOT EVER, EVER, EVER LINK YOUR VISA CHECKCARD, DEBIT CARD, OR ANY OTHER TYPE OF CARD WHICH LINKS TO YOUR PERSONAL BANK ACCOUNT TO AN AUTOMATIC PAYMENT SYSTEM!
Now count your blessings that Dreamhost is doing the right thing in response to their screwup and go get a credit card that’s JUST a credit card and not a bank card and play smart in the future.
January 15th, 2008 at 12:09 pm
Tony Says:
Am I the only one who thinks the FBI needs to look into this matter?
Yes, yes you are.
January 15th, 2008 at 12:09 pm
Unbelieveable. Even after getting 700+ pissed off comments on dreamhoststatus.com and improperly acquiring $7.5 MILLION, you can’t take this shit seriously. Sure, it hasn’t happened in 10 years, WHICH SHOULD BE ALL THE MORE REASON TO TAKE THIS SCREW-UP A LITTLE MORE SERIOUSLY. I was waiting to see if this blog post MIGHT be a serious, mature, contrite explanation before making a decision to stay or leave, because if erroneously billing your loyal customers by $7.5 million in one day isn’t cause for a straight face, I don’t know WHAT is, and based on this further slap in your customers’ faces, my clients and I are out of here and looking forward to seeing this utterly irresponsible, untrustworthy joke of a hosting service collapse in grand fashion.
January 15th, 2008 at 12:12 pm
Well said, “not a customer” ;]
January 15th, 2008 at 12:13 pm
Well, we’re only human. Fortunately. And humans make mistakes. Also fortunately. Because they learn from it.
I really aprreciate that you guys at dreamhost are that open and transparent to let us know what happened in detail. Can’t tell this about many other companies.
Of course customers’ money is nothing to fool around with, but this issue shurely can be reversed.
Still in favour of your service,
Christian
January 15th, 2008 at 12:14 pm
@ Robin - Sorry for the clerical error. Are you also curing cancer with your mega website(s)?
I also have 10+ sites hosted with DH and the world will go on if people can’t access them for a short period of time. I feel badly for people whose accounts were overdrawn as a result of DH’s egregious error but they are doing what they can to rectify the situation whether people here type their empty threats in ALL CAPS or not.
January 15th, 2008 at 12:14 pm
As someone who left Dreamhost last year, I can’t say I’m surprised this kind of thing happened.
I’m using AN Hosting now, and I’ve found them to be far superior. If you want to check them out, here’s my affiliate link:
http://blamcast.net/anhosting
And here’s the straight URL:
http://anhosting.com/
If you like what you see, do me a favor and sign up via the affiliate link, I’d do the same for you :)
- John
January 15th, 2008 at 12:18 pm
And you’ll be unsuspending people when exactly?
Here’s the thing — I appreciate that you had to whip up a “fix me” script pretty quick, of which, step one should’ve been “get everyone back on line regardless of billing situation”.
I’m glad I didn’t have any valid payment info stashed with DH, I’m unglad that I still don’t have a functioning web site.
January 15th, 2008 at 12:20 pm
What I really want to know - has anyone received a refund yet? Whose to say this new script that’s set to fix things isn’t as screwy as the script that caused the problem in the first place? :P
January 15th, 2008 at 12:21 pm
For a guy who just overcharged $7.5M in bad debt you’re surprisingly casual. I’d be offering free kicks to the nuts for any customer that wants to drive down. :)
We’re happy with the service and we’ll stick with you, but fix asap! GO IN THE CAVE NOW! GO!
January 15th, 2008 at 12:22 pm
I have been a customer for 4 years. This is the first time they have really screwed up on billing. I am amazed at how lightly you guys are trying to make this situation seem. At over 7 million dollars, this should warrant a full investigation by the FTC. It’s not a joke, or a “oops, sorry, my bad” type of situation. Having worked with various credit vendors like First Data Merchant Services doing large clearing of transactions, I am well aware of how they work. Given that, I would suggest that you work on a more serious note with your business to ensure that once the transactions are corrected and any additional fees incurred by customers are paid, and that you make a process to ensure this never occurs again. Normally companies do this by reviewing their settlement files and sending reversals immediately if things are not found to be correct. It also appeared from the almost $500 I was billed that, you not only did it once, but twice since I show double billing for each month billed. I am glad you are working on it and resolving the issue, but the tone and methods of handling this make me question if I want to continue my service. As of now, I can live with the panel being flaky and the occasional host being down given I don’t do 24/7 business services as long as the billing is corrected. The price for what you get is still un-beatable.
January 15th, 2008 at 12:24 pm
Get serious! We are not in the mood for another comedic blog post.
A lot of people are going to be paying overdraft fees. What do you plan to do about this?
January 15th, 2008 at 12:25 pm
Josh, thanks for the explanation of what initiated the overbilling, and for working to get all of the overcharges refunded, but really what I’m most worried about is that you didn’t acknowledge the double billing at all.
In fact, you claim “[Our billing system] is so robust you can just run it on any day you want, and it’s safe. It won’t double-charge people”
Obviously you were wrong about that, as quite a few of us were billed twice for each month of 2008. I’d appreciate an update stating that you are now aware of the double billing problem and have made/will make sure it’s fixed.
January 15th, 2008 at 12:25 pm
Look guys, do you try and sue a store assistant if they give you the wrong change… no wait, don’t answer that :|
I have to comment on the fact that so many people had mortgage payments, rent to pay or w/e conveniently (or not) on this exact date, come on, are you really saying after your payments clear you don’t have any money till your next pay day. Doubt it.
Besides, DH did give you plenty of warning that you were going to be triple billed :P
Lighten up a bit already, and think about which type of customer you would prefer and be more likely to help as a CSR at any company…
kbai
January 15th, 2008 at 12:25 pm
Thanks for that link John, I’ll check it out.
January 15th, 2008 at 12:31 pm
You guys slay me!
I received notification today about the billing bungle.
Big deal, $119? I can’t complain (of course, I speak for myself only - I have the room to spare, and I’ll be reimbursed soon enough so big f’n deal I say!)
Good job on noticing quickly … though it’d be hard to miss a $7.5 million surplus billing, but hey, who’s counting!
Anyway, I hear tell that the “competition” is jumping on the bandwagon… a retaliatory attack of sorts, based on recent past. That brought a tear to my eye… from the gut wrenching laughter it sent me through. Sadly, there are some who will get suckered in, and jump off the Dreamhost ship. To each their own I suppose…
January 15th, 2008 at 12:31 pm
Guys, this isn’t the official blog of Dreamhost. Granted it was probably way to light-hearted but you need to stop blowing things out of the water.
January 15th, 2008 at 12:32 pm
Some interesting things - with all the people wanting to check out their billing status, the Dreamhost Panel is not responding at all right now.
And another thing, does this suggest that Dreamhost’s annual turnover is $7.5 million?
January 15th, 2008 at 12:34 pm
I’m fortunate in that my account happens to be a two-year and is paid up until 2009, so I didn’t get hit. I have to say that I *will* be making a point of turning off auto-billing, however. The pros just do not outweigh the cons, after seeing this mess.
For those of you having hysterics about the whole thing, I do sympathize, but I’d just like to point out that Dreamhost is at least taking responsibility for their error and trying to correct it. That’s a bigger point in their favour than many of you may realize.
Case in point: I have experienced a similar issue before, when I had just finished university and started a full-time job. I refused to allow the bank (CIBC) to automatically deduct my student loan payments because I knew that things were going to be very tight for the first year or so and I wanted to manually make the payments in order to ensure that there would be no shortfall problems. Unfortunately, after a few months during which the bank behaved themselves, they turned on auto-billing - against my express wishes and without any notice - and took a payment after I’d already made that month’s payment. Which, of course, resulted in other payments bouncing and fees for said bounces. The bank not only refused to refund the fees, their student loans division also would not refund the extra payment that they’d taken. They also would not credit it to the following month, which left me broke till the next bi-weekly pay cheque and scrambling to come up with the loan payment for the following month. If it hadn’t been for a helpful bank teller, who put a stop payment on the bank’s own withdrawal, they’d have done the same thing the next month as well (the student loan division claimed that it was too late to stop it and I’d have to contact them after it came out to confirm that I wanted auto withdrawals turned off). The bank never did admit that they were wrong, nor did they do a single thing to make up for their error. Dreamhost *has* admitted their mistake (albeit in a more flippant fashion than I’d consider appropriate) and they *are* working on correcting it. I’ll be staying and giving them the chance to prove that truly repentant and that they’ve learned from their error.
The totally unrepentant bank, on the other hand, won’t see me for my dust once the last loan payment is made in a few months.
January 15th, 2008 at 12:34 pm
Is anyone else happy that lots of people will be leaving DH and therefore the servers will have less people on them? I, for one, am.
January 15th, 2008 at 12:35 pm
Josh, the programmer is not at fault - you are. What you claim is like crashing your car at 120mph and then blaming the car manufacturer for allowing you to go that fast.
January 15th, 2008 at 12:38 pm
As a ***professional fraud investigator***, I wish everyone who believes this is fraud a hearty “Good luck!” in proving it. If they would’ve taken the $7.5MM and run off to Cabo, that’s fraud. Already working on reimbursing money? That’s doing the right thing.
Yeah, this whole thing sucks. No argument. But I am incredibly happy with the service, and I’ll be staying.
January 15th, 2008 at 12:44 pm
People leaving DreamHost? Ooh, more bandwidth for me :)
DreamHost has been a great service, and while I do understand people’s points about overdrafts, etc., they made clear that they’re going to take care of it.
DreamHost stands to lose a LOT of money just in the transaction fees, and paying off the people who did get overdraft fees… that should be punishment enough.
January 15th, 2008 at 12:47 pm
I certainly hope this all gets cleared up soon. I prolly won’t leave mostly cause I’m too lazy but I do think it’s time for a little bit more professionalism.
As one who has been an IT professional for 25+years, a Team Lead on $100 Million projects, I would tell that if I found a programmer writing critical code like that did not have a proper validation check I would not fire them. No that would be letting them off way too easy. I would tell them everything is ok, let them have a false sense of security, and then slowly but surely turn the rest of their lives into a living hell they can never escape from, no worse fate can be had.
The biggest loser in this is Dreamhost itself. The hit to your credibility will take some time to recover from. I suspect you will have some very serious explaining to do to your upstream billing provider, and I would not be suprised if you got cutoff from billing credit cards until you could prove proper controls are in place.
January 15th, 2008 at 12:47 pm
You morons crying for this to be investigated by *insert government agency here* obviously didn’t read the part where everything was explained. There’s not a whole lot to waste taxpayer money investigating when we already know what happened.
January 15th, 2008 at 12:49 pm
After working at Hostway Corp/Affinity Internet (hostway.com, affinity.com) as an application developer, i know that this kind of operation has to be staged, tested a few times before running it. If you find discrepancies, you study them and you make sure you know why before making an executive decision (keyword: executive). What in the world are you ppl doing running scripts blindly and just because the billing looked below average.
And don’t claim your billing and provisioning systems are so perfect, how about you fail safe that S#IT so it does not ever bill future dates (date
January 15th, 2008 at 12:50 pm
People…
Millions in damage?
ITS SPREAD OUT
You got charged what 200 mabey 300 dollars?
They refund. Problem Solveed.
Those of you saying THIS IS MY COMPANY IM SO ANGRY YOU PUT US IN THE RED.
300 dollars put you in the red?
Wow…sucky company.
Thanks for the laugh.
But I do still need that email.
January 15th, 2008 at 12:50 pm
This seems like a LOT more than just a simple date goof. According to the email - it seems to imply that I was being billed $342.84 per MONTH. (”Balance of $342.84 due since 12/14/08, plus $342.84 overdue since 11/14/08″) And yet, I’m on the $190.80/2 year plan. Somewhere — that math just doesn’t add up. How do you get $342.84 a month instead of $7.95? A simple date error wouldn’t make that massive of a goof. After all… my current plan expires in June. Therefore, if the billing system thought it was December 2008 or January 2009 — that’s @ 6 months overdue. That should only be an overdue bill of $47.70, if I were to have renewed with the same plan I’m on. So how do you get that I owe $685.68 instead of $47.70? That just doesn’t calculate under the guide of a “simple user error”. There seems to be something kind of bigger problem there.
Normally, I don’t mind the flippant attitude. On something as serious as this, however ($7.5 million in overcharges?!), it seems like this would be the ONE time for a nice professional explaination. Instead, the humorous tone here gives the appearance that you were too busy looking for goofy images instead of taking this seriously and working to repair the issue. I’m not doubting that there was work going on to repair it — just stating that the tone of the entry give the impression otherwise. (Same with the goofy photo in the status blog post).
I’m just incredibly relieved that the credit card I had on file had long since expired; and I wholeheartedly feel for those people who got charged.
January 15th, 2008 at 12:52 pm
“under the guide of” should be “under the guise of”. Oops.
January 15th, 2008 at 12:52 pm
People seem to focus on the fact that this was a big ($7.5M) error, as if that entire sum affected them personally.
If you’ve been overdrafted, or hit especially hard, yes, it hurts. But would you want to be vilified in the same way if you had a similar mistake? If you answered yes, you’re not thinking rationally.
Everybody talks about how Josh and DH needs to grow up, and expects their talk of being “insulted” and “pissed off” to pass? The only childish people are those that pass blame off to others. If you only had a few bucks in your checking account, why leave auto-pay enabled? This was bound to happen somewhere at sometime. We expect machines and people who render us service to be infallible, when in reality they cannot be more infallible than ourselves. If you’re going to leave DH because of this, then by all means leave, but think through your anger.
January 15th, 2008 at 12:52 pm
After working at Hostway Corp/Affinity Internet (hostway.com, affinity.com) as an application developer, i know that this kind of operation has to be staged, tested a few times before running it. If you find discrepancies, you study them and you make sure you know why before making an executive decision (keyword: executive). What in the world are you ppl doing running scripts blindly and just because the billing looked below average.
And don’t claim your billing and provisioning systems are so perfect, how about you fail safe that S#IT so it does not ever bill future dates (date IS SMALLER THAT CURDATE()) or duplicate transactions for that matter (unique: acct, date, amount). Now that is a joke to any programmer.
The only reason i hosted here is because nobody in my company hosts personal pages in our system, they are too personal. So you go to what you feel is the next best and affordable solution. Well i must say