The Final Update

January 17, 2008 on 12:52 pm | In Foobars, Updates by Josh Jones |

Okay, all the people who had still not gotten their refunds was starting to seem a little weird, so after further investigation yesterday, I think we’ve finally got things completely fixed.

It turns out, there was a glitch in our new PayflowPro.pm that resulted in only the first transaction in a single second actually going through! According to Paypal’s site, that PayflowPro.pm should be just a drop-in replacement for the old PFProAPI.pm… and it did seem to be, after changing two lines everything seemed okay.

However, there was one little difference. The new HTTPS interface requires you to pass a unique id for each transaction, and PayflowPro.pm generated that unique id as follows:

my $request_id=substr(time . $data->{TRXTYPE} . $data->{INVNUM},0,32);

The problem was, we never passed in the (optional) “INVNUM” field.. we had an invoice number, but we passed it in as the (also optional) “COMMENT1″. So, our “unique” request_id was pretty much just the current time (plus whether it was a sale or a credit)!

In my testing this didn’t fail, because I didn’t run multiple transactions in the same second. Also, they apparently still return the same old success code we test for when this happens! But when multiple biller services run in parallel on all our controllers, lots of transactions end up happening on the same second.

The Upside

It turns out of the actually closer to $9,600,000 we thought we mistakenly charged, only actually about 1/4 of them ever _actually_ hit people’s credit cards. Our system thought we charged them, and they received an email receipt, but that was where it ended. It turns out we actually billed “only” about $2,100,000 incorrectly.

The Downside

This bug still existed until late last night (around 4am).. so when we ran our super-refunder script, the same thing was happening. Only about 1/4 of the refunds successfully went through. This resulted in the following situation:

About 9/16th of our customers: weren’t actually billed OR actually refunded.
About 1/16th of our customers: were billed AND were refunded.
About 3/16th of our customers: were billed BUT WEREN’T refunded.
About 3/16th of our customers: weren’t billed BUT WERE refunded. (of course, nobody wrote in about it!)

Anyway, last night we fixed the bug (by passing our invoice in as INVNUM) and re-ran another fixer that took an actual log of successful transactions downloaded from our processor and cross-referenced everything with our system. This is what it did:

About 9/16th of our customers: marked their bill and refund as $0 amount.
About 1/16th of our customers: left everything alone.
About 3/16th of our customers: redid the refund.
About 3/16th of our customers: redid the charge.

Double checking now, there were no more of those glitches from before, so everything seems okay.

Once again, all the stuff mentioned in the last post still holds true (you may not see the correction on your statement yet, but if you call your processor they should see it coming, for REALs this time), and once again, I’m very sorry about this whole fiasco.

Sincerely,
Josh Jones

P.S. For people wondering how the “robust and stable” rebiller could have created multiple future charges for the same date… I guess I meant “robust and stable” in regards to normal use over the last ten years. It looks like in this case, when multiple instances were running in parallel on a future date, race conditions allowed some multiple charges for the same period to be created. That too should never happen again now that we don’t allow future bill dates.

427 Comments »

RSS feed for comments on this post.

Pages: [1] 2 3 4 5 6 7 8 9 » Show All

  1. 1

    What a headache. For all involved. I assume you’ll be running simulations on your new “robust and stable” biller starting now?

    Comment by Vlad — January 17, 2008 #

  2. 2

    Everyone makes mistakes, and I love you for your honestly.

    I prefer your style to a silly press release that takes me for granted and does not give me any information. I don’t understand those who say they do.

    All the best! And if you are going to change your style of writing because of a few whiners, that would be the next big Dreamhost foobar. I hope its all behind us now. Cheers!

    Comment by Aaron — January 17, 2008 #

  3. 3

    This is fun! Can we do it all again?

    Comment by Simon Jessey — January 17, 2008 #

  4. 4

    by far my favorite week to be a dreamhost customer… a few got pissy but most of us laughed about it (you don’t want those cheap bastards as customers anyways, hahaha) … I almost told MW that if there were multiple threads or multiple machines running the script it’s highly possible for the duplicate billing to occur but I didn’t want to have to bite my tongue later, glad this was cleared up… I still think the “I Survived Dreamhost 2008″ t-shirts are called for…

    Comment by Kriss — January 17, 2008 #

  5. 5

    The problem MW is youre wanting answers right away, you pounded them for two days about the duplicate billing issue… What you don’t understand is that sometimes it takes more than 2 days to fix a bug or to understand WHY a script did something in programming … I’ve had to spend weeks before analyzing code and debugging just to fix 1 stupid line of code, so I entirely understand why it took them so long to tell us about the duplicate thing. When youre dealing with multiple machines, multiple threads, thousands of customers, not a static error (notice how he said some were billed and some were not) these sorts of things can make debugging and finding a programs flaw VERY painful and time consuming. Theres literally NOTHING you can do to speed it up. If they don’t know the answer then what would you like them to say? Do you really want to hear them just say “Oh yeah we know about the duplicate billing but we don’t know why…”. No you don’t, what does that resolve?

    Comment by Kriss — January 17, 2008 #

  6. 6

    Of only it were as simple as:

    if($cock-up === true) rollback($last-action);

    Comment by Simon Jessey — January 17, 2008 #

  7. 7

    It still hasn’t gone through on my end. I hope it does by tommorrow or it looks like no beer for me this weekend :(

    Comment by Mindbender — January 17, 2008 #

  8. 8

    I’d like to mention, this fiasco has reinforced my affinity for Dreamhost. admittedly, I do not use you for mission-critical stuff, nor was I directly affected by this foobar, but observing the way you handled both the fallout and the response from your initial post gave me hope for companies.

    I personally prefer the jokey, lighthearted style; I somehow got that they were sincere, and were trying to find a lighter side of a bad situation. That is a personal preference, however, and I know many people didn’t agree. When those people made their displeasure known, we not only got a different, yet still friendly and thorough, style, but also an apology for the “mishandling” of the initial announcement.

    What this says to me is that Dreamhost truly does listen, an attribute and action that very few companies follow through with.

    While I certainly do think the error was definately a PEBCAK error, and (honestly) a touch boneheaded, I appreciate the person responsible for personally accepting responsibility.

    Thank you for your honesty, thank you for the explanation, and thank you for toning down the jokes in direct response to your customers. Good luck with your Biller.

    Comment by Allan — January 17, 2008 #

  9. 9

    MW, Mindbender: No cash for beer, no cash for webhosting, but cash to spend on an internet connection and unlimited hours of free time to rave and rant on blogs?

    Maybe time to actually get a job, you guys?

    Comment by Aaron — January 17, 2008 #

  10. 10

    “According to Paypal’s site, that PayflowPro.pm should be just a drop-in replacement…”

    wait, so all of dreamhost’s accountings and billings go through paypal?

    um.
    fail.
    this changes my entire outlook.

    Comment by thomas — January 17, 2008 #

  11. 11

    “About 3/16th of our customers: weren’t billed BUT WERE refunded. (of course, nobody wrote in about it!)”

    I got a “refund” but I used Google Checkout so it actually didn’t give me a refund because my card wasn’t on file. Nor did I get charged anything. So some of that 3/16th were people like me who just figured nothing was charged or refunded so ignore =P

    Comment by Tim — January 17, 2008 #

  12. 12

    Oh MW.. If you cancel that card, can they send a refund to it? (I don’t know, but if you haven’t it might be something to check before you do cancel/change it)

    Comment by Tim — January 17, 2008 #

  13. 13

    Just a note on Payflow Pro .. it’s _owned_ by PayPal, because they recently bought it from VeriSign, (who bought it from cybercash I believe, which is where we originally signed up) but it’s not at all related to what everybody normally thinks of when they think of “paypal”.

    It’s a regular credit-card-payment-gateway thing, not related to the ebay-email-your-friends-money thing.

    Comment by Josh Jones — January 17, 2008 #

  14. 14

    I appreciate the tone and detail of this post a lot. I’m still watching my accounts like a crazy stalker though.

    Comment by c — January 17, 2008 #

  15. 15

    I sympathize with you Josh, that sounds like the kind of mistake I could make. I hope everything clears up quickly and you can get back to your lighthearted style, and I hope the people who are financially ruined because of this will seek help from a financial planner.

    Comment by kestasjk — January 17, 2008 #

  16. 16

    thomas, payflow (which was once verisign) was bought by ebay/paypal a couple years ago. Its a tried-and-true full card billing system thats been around a long time. Its kind of a pain in the ass system though - but at least someone is maintaining it (I swear, verisign just sat on it).

    Comment by OnyxRaven — January 17, 2008 #

  17. 17

    Your system is still fucked up, you double charged my bank card AGAIN this morning, sending me into the red.

    Since there are overdraft fees, you can look forward to a pleasant and swift chargeback from my bank. Let this be a valuable (and soon to be EXPENSIVE) lesson.

    Comment by Jeff — January 17, 2008 #

  18. 18

    I appreciated your professional entry as well as your lighthearted one. I’ve been a Dreamhost customer since 2004 and you’re going to have to do a lot more than overcharge me $750 to get rid of me. Just kidding. I’m so glad that I have Dreamhost charging to an empty Discover card!

    Mistakes happen, you’re going to pay dearly for this one, and I think you’ve learned a lot from it. Thanks for the update.

    Comment by Bill Brown — January 17, 2008 #

  19. 19

    Honestly,

    This incident has only proven that you are an honest company.
    Most other companies will cover up and pretend nothing happened.

    You blog posts are just as they should be, informative and fun to read.

    Regards.
    Brian.

    Comment by Brian — January 17, 2008 #

  20. 20

    Kriss had a great idea. I’d definitely go for an “I Survived DreamHost 2008″ shirt. :) Hey, why not? I recently got my 2007 keychain. :)

    Thanks again for keeping us updated. I remain an appreciative (and happy, of course) DreamHost customer.

    Comment by Flatvurm — January 17, 2008 #

  21. 21

    Yes - kudos for the honesty and up-front description of what happened [again] (OK, without the pics it isn’t as fun to read, but it is a lot easier to DIGEST. next post you can put pics in again).

    Looks like I’m in the 9/16. No bill no refund. Shucks, I was looking forward to spending my ill-gotten $400.

    Comment by Jeremy — January 17, 2008 #

  22. 22

    Oh for the love of god, MW.

    At this point you seem to be actively creating situations in which you can be pissed off about something… You’ve whined nonstop about what a tremendous error all this is, and how direly it’s affecting your financial situation but come on. If you cancel the number that Dreamhost has on file, and that they charged, OF COURSE they won’t be able to refund your money. That’s the only way they HAVE to refund your money.

    Dreamhost screwed up in a fairly large, but not all that uncommon sort of way. They have owned up to that, and they’ll lose customers and a HUGE amount of money over it. They won’t get their astronomical credit card fees waived because they screwed up, whereas they’ve said that you’ll be compensated for YOUR fees. In the face of all that upfront dealing they’ve done with us, you’ve complained about the style in which they’ve been so upfront, you’ve complained even after they changed the style to suit you, you’ve repeatedly called them liars, in spirit if not in word, you’ve threatened legal action well beyond what’s realistic, you’ve made asinine references to some conspiracy, and overall you’ve acted in a ridiculous manner. I know you’re upset but you’re supposed to learn around the age of five that screaming after someone broke your toy train won’t hurry them along when they’re already trying to fix it.

    And I’m sure, once you’ve removed any possibility that DH could get your money back to you, then you’ll be freaking out all over this non-official message board AGAIN on the basis that they lied and didn’t give your money back.

    Honestly, if I were running Dreamhost I would be ECSTATIC that you won’t be continuing your patronage. I don’t even really like being a fellow customer with you.

    Comment by Thomas (A different one though) — January 17, 2008 #

  23. 23

    Josh,

    You’re doing a damn good job at winning me back. I was pretty pissed at the lighthearted response, but you and Team DreamHost have been clearly working around the clock to resolve this issue. And keeping us all in the loop about it, down to the technical details. I really do appreciate it.

    Comment by Steve — January 17, 2008 #

  24. 24

    Good news, everyone!

    /farnsworth

    This type of detailed backend junk is why you’ll keep the majority of your customers.

    We appreciate and get it.

    I *do* want to see some funny spin on this in the next newsletter, tho.

    k,thx (but not bye)

    Comment by Its been emotional — January 17, 2008 #

  25. 25

    On the plus side, at least the system is being fully tested and all of the problems are being brought to the foreground. Hopefully this means there will be a lot less problems in the future.

    On the negative side, it’s too bad no one thought of doing a lot of QA and testing the system in the background first before this all happened. Sadly, that’s a common problem with most accounting software though.

    Comment by Anon — January 17, 2008 #

  26. 26

    Hmm to many Thomas’s around…

    For those wondering, when a charge goes through the Credit Card networks, it can take up to a couple days for a transaction to actually go all the way through. In the meantime you have the temporary authorizations that people have been seeing. Exactly how long it takes to go through depends on several factors including the bank if I remember correctly.

    At any rate as I mentioned in previous posts, this was a major screwup. Everyone to me gets a chance, this one used up that chance. I will be staying for now but I will be being careful.

    To those claiming fraud company… Typically a fraud company does not have a many year history of doing business. There are companies that when they have financial problems will float money, but I don’t see that being the case at this point, much to expensive for the company and no hope of resolving it in a beneficial way. I could be wrong certainly, just don’t see it in this case.

    Thomas

    Comment by Thomas — January 17, 2008 #

  27. 27

    Yes, when you cancel a card, you CAN still get refunds/credits made to it. Any purchases before the cancellation will also still go through. You simply can no longer make new purchases through a canceled or blocked card as soon as the cancelation/block has been placed and will then need to use a new card. So, that shouldn’t be a concern. The time it takes to receive the refund is both up to the merchant (DH in this case) and how quickly they do it, or how long it takes to dispute it or do a charge-back if they don’t (can be up to 60 days or more for a charge back, if the merchant disputes it — and I don’t assume DH would dispute your charge-back).

    Comment by Word — January 17, 2008 #

  28. 28

    Whoa….

    um..MW?

    Cancelled account that charge was made to = nowhere to send refund.
    Nowhere to send refund = no refund in MW’s pocket in a timely fashion when it DOES happen.

    No refund in MW’s pocket = Two more days+ of MW screaming because he didn’t get his refund and blaming DH for it.

    I don’t get the impression that you’re making this easy for anybody, least of all yourself.

    As for the rest of this? Well, all I can say is that I’m on Day Two. I’ve already stated what happens at Day Seven. I look forward to not having to go through that.

    Comment by Dany — January 17, 2008 #

  29. 29

    OK, Word clarified a bit. But still, not making it that much easier.

    Comment by Dany — January 17, 2008 #

  30. 30

    Thanks for keeping us up to date; I really appreciate it. I seem to be one of the customers who didn’t get charged or refunded, and I saw the billing issue entry on the status blog before I checked my email, so this hasn’t affected me at all, but I do appreciate the openness and candor with which the problem has been addressed.

    MW: Please, stop complaining about this issue. Dreamhost made a mistake (a big one, but still, an easy mistake to make), and is working on it as quickly as possible. I deeply appreciate the open, honest way they’ve dealt with the issue. I can understand how the joking blog entry could upset you if you’ve been burnt by this problem (I found it funny, and in good taste, as he was apologetic at the same time as he was joking, but I can see how someone who’d had serious issues wouldn’t take it that way), but by now he’s apologized for the tone, explained why some of the refunds didn’t go through immediately, and done everything he can to fix the problem and pay back everyone who was inconvenienced by this issue.

    The way Dreamhost has dealt with this is far better than how most corporations would have. There are so many cases when companies try to hush things up, only releasing the barest minimum information possible, cover their asses by only promising refunds on the next month’s bill, and so on. I would be very upset to see Dreamhost stop this policy of being open and honest about their problems because of some people who got upset when the problems weren’t fixed immediately or when there were some problems with the promised fixes.

    Comment by Brian — January 17, 2008 #

  31. 31

    @MW:

    WHY on earth would you have canceled the card number that the original payment to DH was drawn from? You have just effectively made it impossible to automatically refund your account.

    Your action in canceling the card makes no sense whatsoever.

    In my own case, the refund has not yet appeared in my account info at my bank, but then again neither has the payment I made on that account two days ago. Banks are often slow about updating credit card charges and payment. I will be patient.

    Thanks, Josh!

    Comment by hogarth — January 17, 2008 #

  32. 32

    I just read Word’s post. Apparently payment can still be refunded to a canceled card. Good.

    Comment by hogarth — January 17, 2008 #

  33. 33

    I got charged $1000+. No problems for me. It went on my AMEX, and it’ll get refunded by Dreamhost. In fact when I noticed the charge, I just sent a quick email to support and asked them to fix it. Then I went on with my work and didn’t think about it. Later, when I read Support’s email and Josh’s blog entry, I enjoyed a good laugh. I appreciate that he’s at least willing to explain it truthfully.

    I think this experience shows the value of using credit cards wisely to pay for these types of bills. Most adults have this option available if they manage their credit well. This way any errors made can get fixed before the money actually leaves your checking account. Thus no overdraft fees, no tears, and just a good, hearty laugh that actually helped to alleviate some of the stress in my life. Thanks Dreamhost!

    Comment by Mike — January 17, 2008 #

  34. 34

    MW:

    You do know you could have called your credit card company and said, “Do not approve any further debits from Dreamhost, but please *do* approve andy credits”. Then, once you receive your refund, cancel the card.

    Canceling the card really wasn’t too bright.

    Comment by Nick — January 17, 2008 #

  35. 35

    @MW

    “Thomas (A different one though)” nailed it.

    It feels creepy sharing the same filespace as you. Souls of Glass my @ss.

    Comment by androo — January 17, 2008 #

  36. 36

    @MW… follow my advice for your future benefit. Use a credit card wisely and you avoid all this hassle. Otherwise, you should have used the option that Dreamhost has always given you to put a charge limit on your account.

    Comment by Mike — January 17, 2008 #

  37. 37

    Muumuu Homer says, “Be more funny!”

    MW, don’t you have better things to do?

    Comment by Adam Backstrom — January 17, 2008 #

  38. 38

    DAMN THOSE RACE CONDITIONS! THE REVOLUTION WILL NOT BE TELEVISED!

    Comment by Zimmy — January 17, 2008 #

  39. 39

    still no refund. really amazing how bad at this ya’ll are. seriously.

    Comment by chadvavra — January 17, 2008 #

  40. 40

    Your site is showing that you issued a refund to me, however my bank is 100% absolutely sure you haven’t. It’s 8:22pm/CST, and NOTHING has been refunded. I think it’s time for a class-action lawsuit.

    Comment by Anonymous — January 17, 2008 #

  41. 41

    @Thomas (A different one though) Says, on January 17th, 2008 at 3:38 pm - “Oh for the love of god, MW.”

    Well that just about sums it up very nicely! Thanks for saying what so many are undoubtedly thinking.

    As for MW - Good luck, man! From the way it appears you handle “crisis” in your life, if you live to be anywhere nearly as long as I have, you are going to need all the “luck” you can get!

    Hang in there!

    Comment by rlparker — January 17, 2008 #

  42. 42

    The best show in all of this is watching MW get all pissed off and wasting all his time on here complaining about it.

    Don’t get me wrong - I know what it’s like to live hand to mouth, and losing $360 in that position would suck. But it IS highly stupid to cancel the card when Dreamhost has said they are refunding the money. And griping here won’t make it any better.

    Refunds always take a few days at least (I’ve rarely, if ever, had one hit my card immediately). Canceling a card only makes it that much harder to get a refund that was ALREADY in process most likely.

    The best thing to do is relax, call up any companies you owe bills to, explain the situation, and then go and work on bringing yourself new work and new money in. Sitting here griping won’t fix it, and calling up the attorney general is a waste of energy that could have been spent earning replacement money.

    Lucky for me, the credit card I had on file with Dreamhost was expired. All of my hosting bills have been paid by referrals this year so they have had no need to charge me in a while and the card is very old. So it is easy for me to find this whole thing a bit amusing….

    …and I still love Dreamhost. At least they are honest when they screw up. They are HUMAN, not some faceless corporation. Thank you, Dreamhost.

    Comment by anon — January 17, 2008 #

  43. 43

    I am continuing to get billed on my card. As of today, January 17, 2008, my credit card has another charge of $120 on it, without any money being refunded whatsoever. The email DH sent me on January 15, 2008 (a very light email seemingly acting not overly concerned), two days ago, stated I would get a refund. I have not been refunded, but charged again. This is unacceptable in every sense of the word.

    It is unethical and deceitful.

    Comment by Lor — January 17, 2008 #

  44. 44

    @MW wrote: “So long, Dreamhost. If I do get my money back, I’ll let my state’s AG know that the refund made it, at least.”

    This was some time ago. Since then, you’ve posted a lot more about the same thing. This makes your previous statement somewhat fraudulent given that you clearly gave the impression you were outta here.

    I’d contact my AG and make some class action something or other, but I’m too lazy and we don’t sue people for silly things like that in my country. In fact, I don’t even know if I have an AG.

    Dude, chill.

    I’d be pissed too if I was you, but freaking out here is only amusing the blog readers. At your expense. Notice how the tone has changed in the last 12 hours or so? People are starting to get over this. I really hope you could too. I want you to get your refund now as much as you do.

    PS: I believe you when you say you’ll miss DH. You’ll be sorry when the control panel at your next host is full of ass.

    Comment by Its been emotional — January 17, 2008 #

  45. 45

    I don’t have my refund yet, and my bank isn’t showing a reversed transaction in processing yet. I know that this stuff takes time so I’ve just put in a ticket now.

    Comment by JohnnyRnR — January 17, 2008 #

  46. 46

    @MW: about the trout

    Wow. Thanks. I’ll take trout over goldfish any day.

    I really, actually, honestly am empathic towards your grief over this. I thought your ranting about the multiple billings to be somewhat moot given the billing clusterfuck to begin with. I figured from the start that the wildly different experiences people were having was from a rampaging billing robot dumping massive quantities of transactions to DH’s billing agent in what may have been akin to a DDOS attack. Even the people reporting issues this late is likely due to the delay commonly seen with credit card transaction postings.

    My account was paid for by VISA. When I heard people had stuff bouncing because of this, there was a big disconnect because the concept of using a bank card/account to pay for an online service seemed totally alien to me. In fact, I didn’t know that was possible! Before that point, I thought people were making stuff up. Now I know that’s not the case.

    I like DH for their corporate personality. I see this as a huge cock-up (love the British), and a freak accident, but certainly not a willful and fraudulent action.

    I respect your reasons for splitting. But, as you said yourself, you’ll miss the cool stuff.

    Oh, and if you do split, as you say you will, it would be hilarious if you show up now and then and make cameos on the blog comments. Since this fiasco, I know I’ll be reading the comments a lot more than I ever did before.

    Cheers (and I still also really hope you get your refund soon)

    Comment by Its been emotional — January 17, 2008 #

  47. 47

    The trouble ticket system seems to be crashing. I’m just seeing a swirling pattern when I try to submit my ticket.

    Is there an email address out there I can use?

    Comment by JohnnyRnR — January 17, 2008 #

  48. 48

    @Johnny: AFAIK, “billing@dreamhost.com” is a good one to use, assuming that billing is the issue you’re wanting to talk to them about.

    This was the email that was sent in the original (and incorrect) billing notices that were sent out at the start of this party.

    I just tried to get into the panel and it’s working and fast for me, so it mat be a hiccup somewhere in the intertubes for you.

    Comment by Its been emotional — January 17, 2008 #

  49. 49

    I still have not been refunded the overcharge fee yet and my bank is starting to put overdraft fees on my account. I’ve sent 4 e-mails now and no one has responded to me, I would like to know what’s going on, thank you.

    Comment by Desmond Lim — January 17, 2008 #

  50. 50

    MW’s situation is funny. Couldn’t have happened to a bigger jerk!

    Comment by MW is a loser — January 17, 2008 #

Pages: [1] 2 3 4 5 6 7 8 9 » Show All

Leave a comment

XHTML: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Comments for this post will be closed on 20 March 2009.

Powered by WordPress. Pool theme by Borja Fernandez, modified by DreamHost.
Entries and comments feeds. ^Top^