Five Fun Facts For Friday

This week, I learned another five things I did not know before:

Monday: Although charging a credit card is instantaneous, refunding really does take 3-4-5-6-or-more business days to process.

Tuesday: You can erroneously credit an expired credit card. The money does leave your merchant account.

Wednesday: You can credit a canceled credit card. The money does leave your merchant account.

Thursday: You can credit a debit card tied to a checking account that has been closed for months. The money does leave your merchant account.

Friday: If you charge somebody with an international credit card and then refund their money, by the time the money gets back on, the dollar will have weakened!

Lucky you to learn these things the fun fun-facts way!

The Final Update

Okay, all the people who had still not gotten their refunds was starting to seem a little weird, so after further investigation yesterday, I think we’ve finally got things completely fixed.

It turns out, there was a glitch in our new PayflowPro.pm that resulted in only the first transaction in a single second actually going through! According to Paypal’s site, that PayflowPro.pm should be just a drop-in replacement for the old PFProAPI.pm… and it did seem to be, after changing two lines everything seemed okay.

However, there was one little difference. The new HTTPS interface requires you to pass a unique id for each transaction, and PayflowPro.pm generated that unique id as follows:

my $request_id=substr(time . $data->{TRXTYPE} . $data->{INVNUM},0,32);

The problem was, we never passed in the (optional) “INVNUM” field.. we had an invoice number, but we passed it in as the (also optional) “COMMENT1″. So, our “unique” request_id was pretty much just the current time (plus whether it was a sale or a credit)!

In my testing this didn’t fail, because I didn’t run multiple transactions in the same second. Also, they apparently still return the same old success code we test for when this happens! But when multiple biller services run in parallel on all our controllers, lots of transactions end up happening on the same second.

The Upside

It turns out of the actually closer to $9,600,000 we thought we mistakenly charged, only actually about 1/4 of them ever _actually_ hit people’s credit cards. Our system thought we charged them, and they received an email receipt, but that was where it ended. It turns out we actually billed “only” about $2,100,000 incorrectly.

The Downside

This bug still existed until late last night (around 4am).. so when we ran our super-refunder script, the same thing was happening. Only about 1/4 of the refunds successfully went through. This resulted in the following situation:

About 9/16th of our customers: weren’t actually billed OR actually refunded.
About 1/16th of our customers: were billed AND were refunded.
About 3/16th of our customers: were billed BUT WEREN’T refunded.
About 3/16th of our customers: weren’t billed BUT WERE refunded. (of course, nobody wrote in about it!)

Anyway, last night we fixed the bug (by passing our invoice in as INVNUM) and re-ran another fixer that took an actual log of successful transactions downloaded from our processor and cross-referenced everything with our system. This is what it did:

About 9/16th of our customers: marked their bill and refund as $0 amount.
About 1/16th of our customers: left everything alone.
About 3/16th of our customers: redid the refund.
About 3/16th of our customers: redid the charge.

Double checking now, there were no more of those glitches from before, so everything seems okay.

Once again, all the stuff mentioned in the last post still holds true (you may not see the correction on your statement yet, but if you call your processor they should see it coming, for REALs this time), and once again, I’m very sorry about this whole fiasco.

Sincerely,
Josh Jones

P.S. For people wondering how the “robust and stable” rebiller could have created multiple future charges for the same date… I guess I meant “robust and stable” in regards to normal use over the last ten years. It looks like in this case, when multiple instances were running in parallel on a future date, race conditions allowed some multiple charges for the same period to be created. That too should never happen again now that we don’t allow future bill dates.

The Aftermath

It seems like it’s about time for a follow-up on things from yesterday.

First, I just want to apologize for the regular-style blog post about it yesterday. Hopefully this will be the (picture, bold, and italics-free) blog post many of you would have liked to have seen yesterday.

The current status: we believe to have refunded everybody who was incorrectly billed at this point. This was pretty much finished yesterday at 3pm, but there were a few stragglers who we got today. If you were charged and haven’t seen the refund show up on your credit card / bank statement yet, try calling your bank. Lots of places take a day or two or three or even four to update their statements even if the money’s already back in, but they should see it (by tomorrow for sure) if you call them.

If this/these erroneous charge(s) by us resulted in you having any sort of overdraft/bounced check/nsf fee from your financial institution, please contact our support team from the web panel. We’d just like to request that you include a copy of your statement with the necessary info showing the fees. It can be either a paper statement or a print out of your online statement, or even a screenshot of your online statement and it can be scanned and attached to your support message via our support form or faxed to us at 714-990-2600. If you fax it, please be sure to write your domain name or DreamHost account number on the fax. When we get this, we will put money on your credit card equal to the amount your bank charged you, as well as give you a DreamHost account credit for the same amount on top of that.

Another thing… if you’ve decided because of this fiasco you’d like to cancel hosting with us, we will allow you to get a full credit card refund of any unused portion of your pre-paid contract, even if you’re past our standard 97 day money-back guarantee. To do so, just close your account as normal from our web panel (”Billing > Manage Account” area). Then, after it’s done, write into support and let them know you’d like to get your remaining account credit refunded to your credit card due to the billing snafu of January 15th and we’ll be happy to comply.

Checks to Protect Your Balances

Finally, here are the precautions we’ve now added to our billing system to make sure nothing like this happen ever again:

1. Our biller service will no longer accept a date in the future.
2. This whole time, we did have an option to specify “never automatically bill me more than $X in a day” on our web panel. Of course, not too many people had this set, and why would they have to? Nevertheless, we’ve made a change now that even if you don’t have a specific daily limit set our system will not allow billing you in one day more than 50% more than the most you’ve ever authorized in the past.
3. Our rebiller does an automatic filling-in of old charges when it finds some missing. This should never actually happen anyway, but we’ve added a new check that if it ever finds itself filling in more than 3 missing charges on any account it stops immediately and notifies our financial team.
4. We’ve also added an overall check where if the total number of payments in a day are more than double the average number of payments we’ve gotten on that calendar day for the last seven months it fails and notifies our financial team.

And that’s it.. I hope this puts things more or less behind us. And remember, if you have any specific issues, our support team is always there!

And of course, my sincere apologies for all of this.

Thanks,
Josh Jones

P.S. I apologize for that joke about the triple billing in the newsletter thing too, but you have to admit, it was kind of ironic that I actually did screw up billing less than a week later.

P.P.S. Some of you have attempted to email us directly with information about unresolved issues stemming from this billing fiasco and have received autoresponders telling you you can’t email us directly. That restriction was unintentional has now been removed so please re-send us your email if you have not already contacted us through other means.

Um, Whoops.

Hello.. how’s your morning going?

I hope it’s been a little better than mine.

We had a teensy eensy weensy little billing error last night… my first clue something was up when I saw this morning’s daily billing report (so far): $7,500,000.

It turns out due to my excessively fat fingers, nearly every one of our customers has been seriously over-billed in the last 12 hours.

I bet when you read this part of the last newsletter:

4. New Office!

Another important thing I’ve been doing instead of writing newsletters
is looking out the window of our NEW OFFICE:

http://blog.dreamhost.com/2007/12/21/were-so-high-right-now-you-dont-even-know

If your next web hosting bill from us is mysteriously tripled, now you
know why.

.. you thought it was a joke!

Ha, the joke is on you! I guess. Um, okay, no, not really, I’m sorry.

How on earth could something like this happen?

Let Me Explain

A couple of weeks ago, just around new years, we started beefing up some of our internal “controller” servers. These are the machines that run all of our “behind-the-scenes” services; things from adding a user to registering a domain to configuring apaches to rebilling customers.

I was on a little-bit-too-long vacation, but when I got back, I noticed our daily credit card payments seemed a tad low in the new year.

So, late last week I tried re-running the billing services for all the days back three weeks or so. I knew this was safe, because after 10 years, the one thing you DO get perfect is your billing system. Our biller is pretty bug-free and robust at this point, because we’d be broke and eating bugs if it weren’t.

In fact, it’s so robust you can just run it on any day you want, and it’s safe. It won’t double-charge people and it’ll even automatically find any missing charges and catch everything up to the day you said.

Anyway, I ran it, and things were fine.. and sure enough, it caught a lot of missed payments. I didn’t have time to look into it right then, but I made a note to myself to check up on it on Monday (yesterday) and see if things were fine or still messed up.

Come Monday

Monday came. I checked the reports and sure enough, things were still pretty low. So I looked at the logs for some of the biller services, and I noticed they were only failing on the machines that had been recently upgraded!

That explained why we were getting some money still (since not all the controllers have been upgraded yet), but not all of it.

Anyway, it turned out there was no 64 bit version of the PFProAPI module we use to interface to the credit card transaction server. No big deal, there’s a new module that interfaces with their new and preferred https interface, and it was only a couple of lines of code to change to get us switched over!

So anyway, I made the change, and it worked, and I even tested it, and things were fine!

But then… late last night, I realized: when I re-ran those biller services last week, they must not have fixed everybody then either! It’s just that by running it again I randomly got different people being charged on the working controllers who had been assigned an upgraded (and therefore broken) one before.

So why not just run it all one more time?

Sure, it should be no problem! So I did, manually running the biller (which is normally automatically scheduled) for 2008-01-14, 2008-01-13, 2008-01-12, 2008-01-11, 2008-01-10, 2008-01-09, 2008-01-08, 2008-01-07, 2008-01-06, 2008-01-05, 2008-01-04, 2008-01-03, 2008-01-02, and 2008-01-01.

I probably should have just stopped there. But then I thought better. I thought to myself, “When did we start upgrading these controllers anyway?”

I couldn’t remember. But, since the biller is super-safe and robust anyway, I went ahead and ran it for 2008-12-31, 2008-12-30, 2008-12-29, 2008-12-28, 2008-12-27, 2008-12-26, and 2008-12-25, just for the hell of it.

Notice Anything?

Don’t feel bad if you didn’t. I kind of missed it myself.

THOSE SHOULD HAVE BEEN 2007!!

Heh, uh.. um, er.. my bad?

So what happened?

Well, that super-robust and stable biller did what it was programmed to do, it ran as though today was December 31st, 2008!

And what did it see? Well, it saw a whole lot of accounts (essentially all of them) who for some unknown, mysterious reason hadn’t been charged at all for eleven and a half months!

So off it went, busily through the night, “fixing” everything up for “today”, December 31st, 2008.

Really, it’s sort of amazing this never happened before in the last ten years.

There IS a bug here.

I can imagine the half second or so of thought that sprinted through the programmer’s mind when he was adding the ability to allow you to pass in what day to run the biller as though today is:

Hmm.. well, I could see us POSSIBLY wanting to be able to bill for a future date.

Well guess what… NO! We will NEVER want to rebill as though today were a day that hasn’t happened yet! But instead, somebody along the line (Sage? Me? Somebody else?) figured, “What’s the harm in keeping it flexible?”

About $7,500,000 in harm, that’s what!

The serious part.

The end to this story is that of course, I’m very very sorry, we’re very very sorry, and I’m sure you’re very very sorry this happened. I really am. I understand the sort of problems that an unexpected large charge to your credit card (or worse yet, your debit card) can cause. If the tone of this blog post seemed a little light, I apologize I don’t mean to offend and I realize how serious an issue this is. I’ve been up since 3:50am trying to undo the damage and maybe I’m a little shell-shocked.

A new service is running right now (in parallel on all the controllers) that fixes all those future charges, re-enables your account if it was erroneously suspended, and if your credit card was automatically rebilled, refunds the payment automatically. You don’t have to contact us or your bank, and you’ll get an email when your account is finished fixing up. It’s going to take several more hours to complete. There are (or were, after this incident) a lot of you these days!

If, because of this billing mistake, you somehow incurred some fees from your bank or credit card company, please let us know after tomorrow (today we are just replying to all 10,000+ billing messages with a generic explanation) and we’ll do our best to make it right for you.

And of course, the biller no longer allows dates in the future.

The moral of this story is that “flexibility” is rarely desired in programming! The less a program will accept/the less a program will do/the less options and preferences it has, the more usable it is/the more understandable it is/the more stable it is.

Tough Love

When designing a program, you’ve got to make some tough decisions .. and when you really can’t decide if this is something your users will need someday, err on the side of leaving it out.

Otherwise, your users will someday err on the side of your face.

Rails Is as Rails Does

I recently wrote a self-described ‘rant’ describing some of the experiences DreamHost has had working with Ruby on Rails on our platform, and with some recommendations on how the Rails community might be able to improve the situation. That post has received some excellent comments, including some by DHH, the creator of Ruby of Rails, and I’d now like to follow-up with some clarifications and further comments of my own.

My original post was intended primarily as commentary for the Rails developer community as a whole and they are, of course, free to take it as simply that. 37 Signals and DHH obviously have their own agendas, business and personal, and those agendas are largely not in line with any agendas DreamHost is working to further. I hope my comments may be taken as ‘food for thought’ by the larger Rails developer community. The project will survive just fine with or without taking my advice, and the zealous user community will likely remain zealous.

DIY

Some of the response to my post was essentially ‘do it yourself’. DHH also went so far as to recommend we not treat the Rails community as a ‘paid vendor’ and to ‘wipe the wah-wah tears’ from my chin. While that is a very valid request, I don’t believe it applies in this situation.

Ruby on Rails is a pretty small part of our overall service. We have already put in a not insignificant amount of work to support Ruby on Rails, including a pretty good amount of user training to assist people new to Rails, and have in general worked to further the cause for it in the shared hosting environment. Aside from hardware, we don’t really have very many paid vendors of any kind now. We have ourselves developed all of the custom perl-based software our business relies on (warts and all). We very much believe in the DIY mentality and it’s been a major part of our business history. Additionally, our business will not be significantly affected one way or the other by the future actions of the Rails community. It simply does not have the critical mass necessary for that.

That said, I would like to see Ruby on Rails reach toward that critical mass of users, and thereby become a larger portion of our business. For that to become a reality I believe it needs to be simpler to use on the server side of things. A larger user community is a good thing for any open-source project.

Bride of PHP?

I mentioned PHP several times in original post, but I very much do not want Rails to become ‘another PHP’. That would be silly, as one PHP is plenty! Ruby on Rails seems to have a conflicted identity at the moment. It is simultaneously compared to enterprise technologies like Java Servlets as well as more average joe programmer technologies like PHP. There are more traditional web programmers experimenting with Ruby on Rails and exploring it, and there are professional enterprise programmers investigating it. The amount of attention Ruby on Rails has achieved in such a short time is awe-inspiring.

Programmers are Lazy

The fact remains that both of those groups of web application developers, and I’d go so far as to say all web applications developers, want development environments that more or less ‘just work’. They want to focus on programming and leave the server administration alone. Rails does a great job at saving programmers on programming time (which is exactly why programmers like it), but many reports I’ve heard indicate that in many cases it trades that programming time for back-end server administration time instead. The average joe programmers out there are typically using completely managed hosting environments that they do not have much control over, and the enterprise programmers are typically using a somewhat complex and large-scale established environment that they also do not have much personal control over. Even those VPS users who do have more control over their server environment do not want to be spending their time managing that VPS. They want to be spending time writing those great applications.

If Ruby on Rails were made to be simpler to use with a wider range of hosting environments, big and small, that would ultimately benefit Rails itself far more than it would benefit DreamHost or me personally.

I would be happy to provide a free hosting account to anyone who would like to help work on these issues with us. Just contact me through our contact form and we can talk about it.

UPDATE! : DreamHost makes using Mongrel much easier! This is one way we’re working to make Ruby on Rails easier to use.

How Ruby on Rails Could Be Much Better

A rant by Zed Shaw, the man who wrote the original version of the Ruby application server Mongrel got me thinking about the experiences DreamHost has had hosting Ruby on Rails driven websites over the past couple of years. The rant itself is somewhat lengthy but is an interesting (albeit self-indulgent) read if you’re either a Ruby on Rails developer, a nerd who’s just into stuff like that, or interested in the behind-the-scenes of a highly public open source project.

I don’t have anything to add to Zed Shaw’s comments about how the Rails development team operates as I don’t have any personal knowledge of that. What I do have personal knowledge of is how difficult it can be to get a Rails application up and running and to keep it running. DreamHost has over 10 years of experience running applications in most of the most popular web programming frameworks and Rails has and continues to be one of the most frustrating.

Some History

When we first started implementing our support for Ruby on Rails, we wanted to do whatever we could to support it. It was an exciting and new web framework that was invigorating web application development. Ruby on Rails seemed to really fit in with our company philosophy and we thought our existing customer base would love it. We ended up implementing quite a lot of new features just to support Ruby on Rails, not least of all FastCGI support. Rails itself turned out to be far too slow to use without some sort of acceleration (unlike any other web programming environment we support), and FastCGI is by far the best suited to a shared web hosting environment. Unfortunately, Rails and FastCGI just don’t really get along! No matter what we did users of Ruby on Rails kept seeing regular internal server errors. The best guess I have is the FastCGI dynamic process manager was politely asking the Rails process to die and instead they were just exiting and leaving the application high and dry. That’s in layman’s terms, of course!

These issues were later mostly mitigated by a single user of Ruby on Rails who wanted it to work better on our servers, but the solution from the Rails community itself was quite honestly, stupid. They said to stop using Apache and FastCGI (a combination that successfully powers bazillions of websites, just not Ruby on Rails ones) and instead to entirely retool our servers with Lighttpd and SCGI (a FastCGI like protocol that may be technically superior, but next to nobody uses). That suggestion shows either a complete lack of understanding of how web hosting works, or an utter disregard for the real world. It’s all good and fine to recommend that users use higher end dedicated server hosting for their commercial applications but you simply cannot ignore the fact that nearly everyone will want to use lower cost shared hosting for getting started. It’s just simple economics. Additionally, people who use systems like Ruby on Rails want to spend time programming and not time setting up servers. Recommending technologies that are not widely used or supported by any major web hosting company is putting too much of a burden on your users, the people you want to keep happy! It’s a good thing we never even tried to switch our system to support Lighttpd and SCGI, too, because 6 months later the ‘in thing’ in the Rails community had shifted to Mongrel, instead. Make up your minds already!

How It Could Be Better

Zed Shaw mentions in his rant that even the Ruby on Rails application run by DHH (the guy who wrote Rails!) has regular issues and needs restarts several times a day. The Rails community is full of very smart programmers and they have implemented a great system that works awesome, when its not needing looking after. It’s one of the most fickle web application systems I’ve ever had experience with, and this is coming from someone who has experienced both the ill-fated Netscape web server as well as many iterations of PHP (PHP has many issues of its own, so don’t get too smug, PHP people).

Ruby on Rails has amazing potential, but here are some things that simply must be fixed before it will ever be as widely used as much less hyped web applications environments like PHP…

1. Ruby on Rails needs to be a helluva lot faster. With a proper accelerator it’s nicely usable but without one it’s painful. Ruby itself is a big part of the problem so this one may come down to just simplifying the management of the accelerator technologies, unfortunately. Mongrel seems like a big step in the right direction, even though it’s not Rails-specific. I hope the Rails core developers will be cooperating a lot more closely with Mongrel developers in the future.
2. Ruby on Rails needs to more or less work in ANY environment. You can’t just expect your users to set up their servers any which way. There are millions of established systems that cannot simply integrate any bleeding edge technology you think is better this week. If you continue to keep this attitude you are surely shooting yourselves in both feet.
3. You need to maintain backwards compatibility better. Admittedly this is the area where PHP has historically done very poorly, but that’s no reason to not one-up them. Also, Rails is admittedly very young as a development platform and you guys have gotten a LOT of attention very early on. Still, with big hype comes big responsibility. You need to keep the momentum going now.
4. Officially support shared hosting environments. The feeling I get from the Rails community is that Rails is being pushed as some sort of high-end application system and that makes it ok to ignore the vast majority of user web environments. You simply cannot ignore the shared hosting users. In my opinion, the one thing the PHP people did that got them to where they are today is to embrace shared hosting and work hard to make their software work well within it. That means it has to be very lightweight (it may be too late for that in Rails already!), and it has to ‘plug in’ to a wide variety of operating environments with minimal fuss and hassle. Compatibility work like that is not glamorous, exciting, or fun, but it’s gotta be done.

What Now?

DreamHost very much intends to continue fully supporting Ruby on Rails and we are working on additional support for it to be tied in with our DreamHost PS plans. Ruby on Rails does have great promise, and I think in time it may be able to meet the hype it’s got surrounding it. I just hope the community doesn’t get a big head before that happens!

The opinions presented here are entirely from the perspective of an outsider with some web experience. I know Ruby on Rails is in use on some very high traffic websites and it is certainly possible to make it work well. What I’m saying is that it really needs to be a whole lot easier. You have to be careful not to confuse user enthusiasm with easy to use. Enthusiastic users will fill in many many gaps in usability (DreamHost thrives on that fact!), but you cannot rely on that over the long-term.

UPDATE! : Further comments and clarification in a companion post.

UPDATE 2! : DreamHost makes using Mongrel much easier! This is one way we’re working to make Ruby on Rails easier to use.

A Strike on Resolutions!

Okay, I’m not exactly sure what the current state of the writers strike is, so I’m just going to make this a Happy New Years post and be done with it.

Happy 2008!

There, got that out of the way!

And now, here are a few of my personal new years resolutions:

* Eat more chocolate.
* Exercise less.
* Start smoking.
* Start really drinking.
* Save less money.
* Spend less time with family.
* Read less books.
* Be more selfish.
* Get less organized.

and of course…

* Get the January DreamHost newsletter out by my birthday.

Now, please don’t get accustomed to this, but I thought I’d go ahead and end this shorty-short post with a NEW FEATURE RESOLUTION!

Well, a new BETA feature resolution at least. Please go and test out a new webmail client at roundcube.dreamhost.com … and comment about it in this discussion forum thread … and who knows, maybe before the end of this writer’s strike we’ll have replaced squirrelmail with it!

Ha. AS IF!