Well, it was a little over two months ago that we had what I think is pretty safe to call the worst disaster in DreamHost history.

In retrospect to me, it’s kind of funny that the worst disaster didn’t turn out to be due to a security breach, a power outage, a loss of data, or actually anything related to our actual hosting service. I guess it shouldn’t be a surprise that people care a lot more about their bank accounts than they do their websites.

I have realized that billing is the one issue where how important we feel it is is completely at odds with how important you guys feel it is.

What I’m trying to say is, we’ve always been ultra-flexible and lax about how people pay, when people pay, or even about giving credits, discounts, or refunds. We figure, whatever, pay us when you’re ready, we’re not sending anybody to collections or ruining anybody’s credit over some measly bandwidth bill.

What we’ve always tried to focus on more (even though it might not seem like it at times!) is our hosting system’s stability, performance, and features.

I guess I’ve always figured that any billing-related error can be easily undone (worst case scenario, it costs us a little money); there is no lasting harm done to the customer. Whereas having a website or email problem could potentially cause permanent damage to somebody’s business or personal life or something?

Well then, let’s go back and see just how little money a worst case scenario actually costs, shall we?

Credits and refunds to cover people’s bank fees: $52,000.

Sigh, if only everybody kept a big cushion of cash in their account! The main damage that can be caused by a billing snafu is for people who get their account overdrawn, and because of that aren’t able to make a critical purchase, or have a check bounce, causing hassles and incurring bank fees. We offered to pay people any amount their bank charged them for going negative, and in the end that total looks like it came to about $52,000.

Accidental refunds: $170,000.

The worst part of this whole process (for us) turned out to be just after the accidental billing, ironically when we were trying to make things right!

If you recall, our system was not actually charging about 75% of the time we thought it did.. and so we refunded thousands of people who were never charged (but, 75% of the refunds didn’t work either). Well, out of all that, and after two months, there are still about 600 accounts who were credited a total of $170,000 in excess of what we charged them that we haven’t been able to get back from them or their bank.

It is slightly annoying when the same guy who complains to the high heavens when he thought he’d been over-charged $9,000 by accident conveniently disappears when we realize that actually, he’s been over-refunded $9,000 by accident.

Extra credit card fees: $82,000.

Another slightly annoying thing is that credit card processors don’t credit you back any fees when you refund a transaction. Overall, the extra credit card processing we did resulted in extra fees of about $350,000! Fortunately, after a whole lot of groveling and explaining the situation (and waiting two months), we finally got all but $82,000 of that back from First Data, American Express, and Discover Card.

Extra support messages: 20,000.

As you may have surmised, people wrote to us about this thing. About 20,000 times… and it would have been tens of thousands more if we hadn’t put up an “emergency block” against new messages for a little while in there.

How much this extra support actually cost (in terms of your wased time, tech support overtime pay, and other questions taking longer to answer to) is hard to say, but normally we only get about 45,000 messages in a whole month!

Accounts canceled: 1000.

It’s also kind of hard to say how many people actually closed their account because of the incident, but in January we did have about 1,000 more accounts closed than average. Assuming each of those accounts would have stayed for maybe another year, that’s another $120,000 down the Intertubes. It’s crazy… from all our power problems back in 2006, we hardly lost any accounts at all.

Goodwill lost: Priceless.

Yeah, it turns out this whole blog post is nothing more than another clichéd MasterCard commercial parody.

P.S. I guess it’s nice to know, less than two hours away from our biggest data center move ever, that we’ll cause a tiny fraction of the disruption to our customers that one unexpected fat finger did!

P.P.S. Thanks RIM, for scheduling a blackberry outage exactly at the same time. It makes us look better. And, maybe some of our Happy Customers will blame their lack of email tonight on you!

First Republic Bank just opened a new branch a block from my house. I was kinda bummed, because I was hoping the lot would become something cool like a Starbucks or a McDonald’s.

Anyway, First Republic has the ad you see above, and some other eerily similar ones, in the window. All the ads use such banal stock photography they never made much of an impression on me, despite passing them every single day.

They never made much of an impression, until yesterday that is! Which is when I passed the Bank of America less than two blocks from them and saw they had just put a new ad that was very eerily similar in their window:

That’s the same lady! Right? Am I right? Yeah, it is! Definitely. Right?

Ha, anyway, I thought that was kinda funny. I mean, geez, B of A, couldn’t you at least check the bank closest to you before picking from the stock gallery?!

The Web Hosting Angle

Now, I shouldn’t be one to bash stock photography… the whole concept is very much alive and well in the entire Web Hosting industry. Still I’ve never come across two hosts with exactly the same “employee” on their front page!

I know sub-prime lending is a mess right now, but come on, bankers!

Speaking of Web Hosting stock photography, I’ve decided to end this post with a little collection of some of my favorite Web Hosting stock photo hotties, each one linked to some actual people employed at the various companies…

MidPhase

(What, nobody wanted to cough up for the “in-focus” version?)

BlueHost/HostMonster

(Isn’t that the same laptop they have on all the desks at IKEA?)

FastHosts

(AIEE! How does she hold that paper without any finger tips?!)

Verio

(That’s some shiny floor they’ve got at Verio!)

1 and 1

(Hummuna hummuna hummuna…!)

DreamHost

(We try our best to make sure nobody ever uses the same stock art as us!)

Ha, did you think you’d gotten through those stupid strike-themed posts?

Well, apparently you are!

The writer’s strike seems to be finally coming to an end, and I don’t know whether to be happy or sad. On the one hand, it means I can finally get back to writing awesome blog posts. On the other hand, it means I can no longer get away with writing these blog posts… which is bad news because this well of creativity is tapped, my friends.

The sad truth is, I did that entire “billing mistake” thing just so I’d have easy blog fodder for another week.

A Last Hurrah

I guess I’ll just quickly wrap up a bunch of stupid things I was planning on “striking” against but never got around to. I never expected this thing to end and was pacing myself.

Hopefully this strike really is settled or you ain’t going to be seeing any new posts here until at least the first Sunday after the Ecclesiastical Full Moon date after March 20th!

A Strike on Fax Machines!

How in the hell is it 2008 and everybody still uses fax machines?

Give me some widespread e-signature standard already, world!

A Strike on Social Networks

Is it just me, or do social networks only appeal to people who 1. are single 2. have no job or 3. care about what their friends are doing?

Because I, for one, am none of those things.

A Strike on Cell Phones

Why do cell phones still keep any data locally?

When you get a new cell phone, you should just have to log into it, like you do, say, a new email client, and whammo, all your contacts/pictures/text messages/themes/preferences/ETC.. are syncronized with a (non-proprietary) server.

We need IMAP for Phones.

(I lose my cell phone once a month.)

A Strike on Global Warming

Dallas warned me not to post this, but he’s in Thailand (trying to enjoy it while it’s still above the ocean.)

There’s just three things that bother me about global warming.

1. There’s literally no way we can be even reasonably sure about what will happen. There’s just no experiment we can run on our entire planet that we can set up an adequate control for!

2. Even if the earth does get warmer, we can’t really know (again, what would the control be?) all the effects that will have on us until it actually happens. The earth’s climate has changed a lot over the billions of years it’s been around, and yet here we are, over 6 billion strong and fatter than ever!

3. Even if the Earth does warm, and even if it is bad for us, there’s again no way we can possibly verify what actually caused it, nor if there was anything we could have done to prevent it.

I mean, I’m all for clean air and water and not wasting electricity and saving the whales, but isn’t just having clean air and water and more money and whales to ride reason enough?!

And if we want to focus on literally saving the human race as we know it, maybe we should be spending more R+D on stopping near Earth objects!

We know they’re out there, we know they’ve hit Earth before, and we know it’s very bad when they do!

A Strike on Getting Old

I broke my left foot playing basketball when I was 27.

It took about a year to heal, but it’s pretty much been fine since.

Now all of the sudden, 3 years later, everytime I get up after being inactive for a half hour or more my left foot kills!

And that’s the real reason why I don’t worry about global warming in the future … I refuse to get older.

Now, please feel free to hold your own stikes in the comments, before the writers settle!

This week, I learned another five things I did not know before:

Monday: Although charging a credit card is instantaneous, refunding really does take 3-4-5-6-or-more business days to process.

Tuesday: You can erroneously credit an expired credit card. The money does leave your merchant account.

Wednesday: You can credit a canceled credit card. The money does leave your merchant account.

Thursday: You can credit a debit card tied to a checking account that has been closed for months. The money does leave your merchant account.

Friday: If you charge somebody with an international credit card and then refund their money, by the time the money gets back on, the dollar will have weakened!

Lucky you to learn these things the fun fun-facts way!

It turns out, there was a glitch in our new PayflowPro.pm that resulted in only the first transaction in a single second actually going through! According to Paypal’s site, that PayflowPro.pm should be just a drop-in replacement for the old PFProAPI.pm… and it did seem to be, after changing two lines everything seemed okay.

However, there was one little difference. The new HTTPS interface requires you to pass a unique id for each transaction, and PayflowPro.pm generated that unique id as follows:

my $request_id=substr(time . $data->{TRXTYPE} . $data->{INVNUM},0,32);

The problem was, we never passed in the (optional) “INVNUM” field.. we had an invoice number, but we passed it in as the (also optional) “COMMENT1″. So, our “unique” request_id was pretty much just the current time (plus whether it was a sale or a credit)!

In my testing this didn’t fail, because I didn’t run multiple transactions in the same second. Also, they apparently still return the same old success code we test for when this happens! But when multiple biller services run in parallel on all our controllers, lots of transactions end up happening on the same second.

The Upside

It turns out of the actually closer to $9,600,000 we thought we mistakenly charged, only actually about 1/4 of them ever _actually_ hit people’s credit cards. Our system thought we charged them, and they received an email receipt, but that was where it ended. It turns out we actually billed “only” about $2,100,000 incorrectly.

The Downside

This bug still existed until late last night (around 4am).. so when we ran our super-refunder script, the same thing was happening. Only about 1/4 of the refunds successfully went through. This resulted in the following situation:

About 9/16th of our customers: weren’t actually billed OR actually refunded.
About 1/16th of our customers: were billed AND were refunded.
About 3/16th of our customers: were billed BUT WEREN’T refunded.
About 3/16th of our customers: weren’t billed BUT WERE refunded. (of course, nobody wrote in about it!)

Anyway, last night we fixed the bug (by passing our invoice in as INVNUM) and re-ran another fixer that took an actual log of successful transactions downloaded from our processor and cross-referenced everything with our system. This is what it did:

About 9/16th of our customers: marked their bill and refund as $0 amount.
About 1/16th of our customers: left everything alone.
About 3/16th of our customers: redid the refund.
About 3/16th of our customers: redid the charge.

Double checking now, there were no more of those glitches from before, so everything seems okay.

Once again, all the stuff mentioned in the last post still holds true (you may not see the correction on your statement yet, but if you call your processor they should see it coming, for REALs this time), and once again, I’m very sorry about this whole fiasco.

Sincerely,
Josh Jones

P.S. For people wondering how the “robust and stable” rebiller could have created multiple future charges for the same date… I guess I meant “robust and stable” in regards to normal use over the last ten years. It looks like in this case, when multiple instances were running in parallel on a future date, race conditions allowed some multiple charges for the same period to be created. That too should never happen again now that we don’t allow future bill dates.

First, I just want to apologize for the regular-style blog post about it yesterday. Hopefully this will be the (picture, bold, and italics-free) blog post many of you would have liked to have seen yesterday.

The current status: we believe to have refunded everybody who was incorrectly billed at this point. This was pretty much finished yesterday at 3pm, but there were a few stragglers who we got today. If you were charged and haven’t seen the refund show up on your credit card / bank statement yet, try calling your bank. Lots of places take a day or two or three or even four to update their statements even if the money’s already back in, but they should see it (by tomorrow for sure) if you call them.

If this/these erroneous charge(s) by us resulted in you having any sort of overdraft/bounced check/nsf fee from your financial institution, please contact our support team from the web panel. We’d just like to request that you include a copy of your statement with the necessary info showing the fees. It can be either a paper statement or a print out of your online statement, or even a screenshot of your online statement and it can be scanned and attached to your support message via our support form or faxed to us at 714-990-2600. If you fax it, please be sure to write your domain name or DreamHost account number on the fax. When we get this, we will put money on your credit card equal to the amount your bank charged you, as well as give you a DreamHost account credit for the same amount on top of that.

Another thing… if you’ve decided because of this fiasco you’d like to cancel hosting with us, we will allow you to get a full credit card refund of any unused portion of your pre-paid contract, even if you’re past our standard 97 day money-back guarantee. To do so, just close your account as normal from our web panel (”Billing > Manage Account” area). Then, after it’s done, write into support and let them know you’d like to get your remaining account credit refunded to your credit card due to the billing snafu of January 15th and we’ll be happy to comply.

Checks to Protect Your Balances

Finally, here are the precautions we’ve now added to our billing system to make sure nothing like this happen ever again:

1. Our biller service will no longer accept a date in the future.
2. This whole time, we did have an option to specify “never automatically bill me more than $X in a day” on our web panel. Of course, not too many people had this set, and why would they have to? Nevertheless, we’ve made a change now that even if you don’t have a specific daily limit set our system will not allow billing you in one day more than 50% more than the most you’ve ever authorized in the past.
3. Our rebiller does an automatic filling-in of old charges when it finds some missing. This should never actually happen anyway, but we’ve added a new check that if it ever finds itself filling in more than 3 missing charges on any account it stops immediately and notifies our financial team.
4. We’ve also added an overall check where if the total number of payments in a day are more than double the average number of payments we’ve gotten on that calendar day for the last seven months it fails and notifies our financial team.

And that’s it.. I hope this puts things more or less behind us. And remember, if you have any specific issues, our support team is always there!

And of course, my sincere apologies for all of this.

Thanks,
Josh Jones

P.S. I apologize for that joke about the triple billing in the newsletter thing too, but you have to admit, it was kind of ironic that I actually did screw up billing less than a week later.

P.P.S. Some of you have attempted to email us directly with information about unresolved issues stemming from this billing fiasco and have received autoresponders telling you you can’t email us directly. That restriction was unintentional has now been removed so please re-send us your email if you have not already contacted us through other means.

Just five and a half months after hitting 500,000 and here we are.

That makes us the 14th largest web host in the world…we’d rank even higher if you don’t count domain registrars that don’t offer real hosting.

As you can see, Godaddy/WildWestDomains has been allowed to become far too powerful. We’ll need to take them down a notch.

Thank you to all our customers - old and new - who helped us get here. We couldn’t have done it without you! And we hope we continue to live up to your expectations. We know you’ll let us know if we don’t. :D

Now get out there and party tonight, but be careful. You don’t want to end up like this guy.

On the off-chance (and judging by that graph of our Level 1 queue, it seems like a pretty good off-chance) that a few of you may have noticed a little problem we had last Thursday afternoon, all the way through Friday morning, I thought I might offer something in the ways of an explanation to go along with that apology.

It’s funny how problems cascade.

It all started Wednesday around noon, when we had a sudden and mysterious network problem related to our core 2 router.

There seemed to be some sort of corruption with the ARP tables.. we eventually figured it out, and fixed it thanks to a gazillion sendARPs. Cisco support wasn’t helpful because we weren’t running the latest version of their IOS router operating system. Unfortunately, upgrading is scary stuff since it requires a short network outage, assuming everything works smoothly. We decided we’d do the upgrade Friday night.

Come Thursday at 2pm, exactly 24 hours after our previous outage was fixed, our network started to get wonky again. It seemed like it was most likely due to all the sendARPs from the previous day expiring at the same time. We were pretty much on top of this as soon as it happened though, and re-sent the sendARPs (staggered this time)!

In fact, it wasn’t actually due to an aging issue at all, but it was just an IOS bug on the core router. No big deal, we pretty much had things under control should the same problem pop up again on Friday at 2pm before the planned upgrade Friday night.

A Chain of Events

Of course, little did we know, a chain of events had already been set in motion that would ruin everybody’s Friday.. FOREVER.

You see, every hour we have a little script that runs that purges old dead entries from our active nameserver database. Really, it isn’t the end of the world for us to keep that old stale stuff around, but in the name of being good dns citizens, I guess it’d been decided a while ago to remove them quickly.

Which is fine, I guess. However, the method in which we decided what entries should be removed was a bit suspect.

We first create a hash of ALL good domains “%domids” from our hosting database. Then, we go through all domains (as “$domid”) in our nameserver database and do:

unless ($domids{$domid}) {
print “- removing stray records under non-existant domain $domid\n”;
$pdb->do(”DELETE FROM records WHERE domain_id=” . $pdb->quote($domid));
}

Which works pretty well, assuming everything is working pretty well.

Well, everything was not working pretty well on Thursday. Because of the network weirdness, the connection to the hosting database apparently didn’t work, leaving %domdids blank.

And, due to the excellent error handling and sanity checking of that script, it did not die at that point, or even so much as raise an eyebrow as it happily decided to delete every single domain in our dns database.

Now, for bad or good, it didn’t just hose the whole table at once. Instead, it just deleted one database after another, in order.. which turned out to be a rather slow process on a busy dns database. In fact, 22 hours later when we finally found it STILL RUNNING (normally it finishes in under a minute since there’s nothing to delete) it had only deleted a third of the domains in the table.. about 300,000. Hooray!

It actually would have been a lot better if it’d just hosed everything at once. It would have been much easier to detect, and rectify, immediately.

Instead, things worsened gradually. It took over two hours before we even started getting reports from customers that their sites were down. At that point, it seemed like the problem was just some sort of residual effect of the network problem, and re-generating DNS for each person who wrote in fixed it right away, and for good.

As time went on, and the problems kept coming in, we realized there was a pretty major data loss in the nameserver database, and started running some scripts to regenerate it all. Those would take a couple of hours, but when they were done everything would be better, we assumed!

It wasn’t until those regeneration scripts finished and we discovered there were still lots of missing domains that it finally dawned on us .. dns records were continuously being deleted!

And THAT is when we finally found the culprit, fixed the mess, and started trying to make sure this would never happen again!

And where was DreamHost Status for all this?

DreamHost Status was down. (See, if you just read DreamHost Status you would have known that!)

Like they’ve said befores, when it rains it pours.

We thought DreamHost Status was down because of the huge crush of people trying to access it due to the network problems. So, when we could finally get into it, we switched it to a static html page to try and lighten the load.

Lighten the load it did not!

Right about then we got a message from our remote data center in San Francisco (both ns2.dreamhost.com and dreamhoststatus.com are kept completely off our main network and in a different city exactly so they wouldn’t be affected by outages like this!)

Your server’s switchport has been de-rated to 10 Mb/s because your server began generating an out-bound storm of packets. This type of event usually indicates a compromise in security.

We have taken this action to mitigate the amount of bandwidth transfer charges incurred by your account related to this activity

Man, what timing! We did not need a DDoS attack right now.

But wait a second. Somehow that just seemed a little bit TOO Murphy-esque. And, indeed, when we probed them further, they told us:

According to my monitor, it appears you’re being DDoS attacked on your DNS service (UDP 53) specifically to IP 208.96.10.221. At 5a,
your traffic peaked our threshold for dangerous amounts of packets going through your switch port which was when your server was de-rated.

That “Distributed Denial of Service” attack was actually just honest DNS requests!

Which was super-high because ns1.dreamhost.com was returning “I don’t have any records for that domain” for a ton of domains, due to the deletion of the DNS database entries, due to the haywire script, due to the network blip, due to the IOS bug, due to us not upgrading as quickly as possible because of the network downtime involved!

The Aftermath

Well, we did the IOS upgrade and it looks like it fixed the networking problems.

We also made our crazy script do some sanity checking. But more importantly (and in just two lines of code!), we’ve now set all our internal scripts to just DIE MISERABLY if they ever get any kind of un-good data from an sql query. Clearly, ’tis better to not do something you were supposed to then to do something you were not supposed to!

We’re also going to separate good old DreamHost Status from absolutely everything else DreamHost related.. even if that means moving it to blogger or something!

We must break the cycle!

I got one!

You’ve still got 30 minutes to submit a pic!

I waited in line for THREE HOURS (fortunately I was, ahem, working the whole time) at an AT&T store, and when I was just about to get in the store they announced “We only have 4GB ones left!”

AIIIIIIIIIIIIIEEEEEEE!

Well, since I promised an 8GB model, I quickly hopped on the subway down to the SOHO Apple store (a guy had come by and said that everybody should go there, they had a TON and the line was SUPER FAST! But at that point I was sooo close..) and whammo, about 5 minutes later I was checking the authenticity of the box they’d given me..

So, let this be a lesson for you, old East-Coast-iPhone-Waiter. Just go to an Apple store around 8pm. They have a TON and it’ll take about five minutes.

I don’t feel bad waiting though. For all the hard work you guys are doing making those photoshopped images, it was the least I could do.

Now I’m in a cab on my way to a dinner BBQ. Hooray for EVDO! There will be one more post with more exciting pictures of the CHAMPTION iPhone later this weekend, I promise.

P.S. Everybody go buy one, I have apple stock.

I refuse to be scooped this time!

We finally passed half a million domains hosted, according to webhosting.info.

Wow, that’s 125,000 domains for every DreamHost-sucks site!

I guess that means it’s about time for The Unofficial DreamHost Blog to give out an iPod Shuffle (now available in springy colors!) to “Matt Skorina”, or if he isn’t contactable, “George”?

Too bad we didn’t announce this yesterday (May 14th), because then it looks like “Kevin Hatfield” would have won. Or maybe “Jessica” .. or “May” (ha, big surprise SHE guessed a day this month!). And then, whomever runs the unofficial blog would have had a fun time since it looks like none of them left any way to be contacted!

Never ones to quit while we’re ahead, let’s see if we can turn 500,000 into 500,500!

With another one of our oh-so-rare promo code moments!

Tell your for-some-reason-still-not-hosted-with-DreamHost friends.. the first 500 people who sign up for any L1 hosting plan with the promo code 555 will get 500GB of storage, 5TB of bandwidth per month, and FIVE free domain registrations for the life of their hosting.. not to mention $50 off!

Now, for those of you already customers.. please don’t get jealous. You’ve probably already got that much anyway thanks to our weekly quota growth policy! As for the free domain registations.. well, come on, who really needs more than one domain anyway?

And, remember, each new customer we get, you get one more person to brag about how you signed up “back when they didn’t suck.”