The Building Speaks

Okay, we just got a fax from the building finally:

On Monday, September 12, 2005, at approximately 12:35 p.m., the building experienced a total loss of electrical power from the DWP on their primary grid. At this time, the building generators started and began supplying adequate power to the tenants.

At approximately 12:55 p.m., the building power was partially restored by the DWP. At approximately 1:05 p.m., the building experienced another total power failure from the DWP.

During this period of 12:55 p.m. to 1:05 p.m., two of the five generators failed. The remaining three generators were unable to sustain the power requirements of the building causing the emergency electrical systems to transfer into a “load shedding mode” and the building’s UPS system to turn itself off, thus preventing permanent UPS and related equipment damage.

The Office of the Building is conducting an investigation into both generator failures and will keep you apprised of the results.

At this time, the building is functioning at full capacity with four generators to support any future outages.

On behalf of the Office of the Building, we would like to thank all of you for your cooperation during the building evactuation and your patience with building personnel during this emergency.

So.. hmm, we will in turn keep you apprised.

Topnorati!

Still working on the packet loss..

In the meantime, we thought this was amazing yesterday:

Until we checked this morning and saw:

(keep in mind the “Impeach Bush” search is actually sort of fake.. it’s just a “spam” search by crazy bloggers to make it show up there.. which really put us as the #1 search for the hour!)

Later this afternoon, Skype passed us though:

Anyway, I guess that’s cool?

Router is upset.

We’re now having some problems with our core router (most likely due to the power stuff yesterday) and are experiencing a lot of packet loss to many customers.

We’re working on it as hard as we can!

We’ll post when we have more news.. hopefully you’ll be able to get here to read it.

Some Residual Issues

As many of you have pointed out there are still some remaining residual issues following yesterday’s power outage. All of our servers are up and running and most websites are functioning normally. There are some remaining issues and we are actively addressing those as quickly as we can. Most of us worked late into the night last night and all of us will be working all day today to make sure everything is completely back to normal.

There is currently some network degradation within our data center that we are currently looking into. That is most likely accounting for slow response times you may be seeing on your websites. That will be fixed as soon as possible.

We will also be investigating the issues with our non-functioning backup generators and will take action as necessary.

At Least We Ate Lunch Beforehand

Hey everybody! So, yeah, yesterday was exciting.

I’ll give you a little timeline/rundown on what-happened-when yesterday. But to generally describe the problems:

Beyond the obvious LA DWP screwup, and the building generator screwup (more word on exactly what that was soon), there are some non-trivial problems when all of your stuff shuts down. The biggest is that when any group of computers is unexpectedly powered off, a small percentage don’t come back up. When you’re talking about a few hundred servers, a small percentage becomes a significant number!

You do also have to be somewhat careful how quickly you power things up, because a popped breaker is just about the last thing you want.

11:45 Went to lunch at sushi place on Flower St.
12:30 BLACKOUT! We’re walking back and see streetlights out, office tower generators starting up. Our building’s UPS kicks in, generator is running (we can see the smoke from our office).
12:45 My idiotic blog post.
12:46 Building evacuated, our upstream providers are down although our servers are still powered up.
1:00(ish) UPS depleted, generator fails, power is down.
1:45 Jason, our datacenter manager, bullies his way in with me while building is still evacuated.
2:00-3:30 We plan for all the fun stuff that happens when the power comes on. Upstreams come back up, although now our servers are down.
3:45 Datacenter power back on, we start powering up cabinets slowly.
4:15 Most of our public network up, firewall busted, private network down (which means our monitoring system is down, Dreamhost site down, Web Panel down. Our blog is up because it’s just on a normal shared hosting account.)
5:00 Big problems come first: File server replaced, firewall being feverishly worked on, 3 public cabinets (out of 40) still down due to switch problems.
6:00 Firewall fixed (which lets us quickly identify continuing problems via monitoring system).
6:30 All public cabinets up, individual machines/services still wonky. Web panel, etc up.
6:30-midnight: Fixing individual servers/services. Some weird Web Panel redirection loop errors fixed. Webmail login errors fixed.

I do have to say, walking around and seeing aaaaaall our servers powered down and quiet was pretty creepy. I’ve been here for a long time, and seen plenty of nasty problems, but this one was particularly freaky.

We would have liked to have something like the above being posted live, but we were literally running and yelling and typing as fast as we physically could from 3:45 to midnight.

We are, of course, already digging into the issue of why power was ever out in this supposedly-very-prepared building in the first place. We’ve had grid outages before and never noticed a blip. There are LOTS of other internet companies in this building (including a bunch of other shared hosting folks) so yeah, we’re all pretty much going crazy about how badly the building handled things. We’ve always been told there are HOURS (not half-hours) of UPS capacity and that the generators are regularly tested and well-maintained.

Also, I bought a lottery ticket on a Red Bull run. I figure karma might balance things out . Keep your fingers crossed.

All Systems Go!

As of right now (just before midnight Pacific coast time), the final little hijinks have been worked out of everything as far as we can tell.

If you’re still having problems, please contact our support team and we’ll look into it.

Sorry about the problems. More info tomorrow.. we’re having a long day right now.

Power Outage Update

Okay, this sucked. Literally 10 seconds after Nate posted that last entry, we received an announcement over our loud speakers that our building was being evacuated and we had to leave immediately! (Also, sorry about that post being temporarily taken down, Nate thought it would be better to remove it since it was so dumb, but methinks better to just add something else!)

Unfortunately, shortly thereafter we figured out (thanks to EVDO!) our sites were no longer accessible, as our three network providers Level 3, Global Crossing, and Mzima were all down. Our physical servers were okay, and probably really happy sitting in the data center (we were no longer able to access) with nothing to do.

But that didn’t last long. Shortly thereafter the entire building where our data center is located’s back-up generators (there are SUPPOSED to be four) stopped working, and all power was gone. We were able to get back into our data center then, and it was like the day after tomorrow or something. Really creepy just walking through rows of dark, quiet, dead server after dark, quiet, dead server. We’ll post some pics from that soon..

So, now, finally power is back up and servers are being restored. We lost a net app and had to replace the head unit from a hot spare. A few servers were of course wonky, and right now three bad cabinet switches are still being worked on (so many web sites ARE still down).

There will be more posts soon, and hopefully everything will be up within the next hour.

Also, just FYI for people who don’t know, we keep http://status.dreamhost.com/ in San Jose, completely off our network, just to update people in times like this when our panel (still down) and everything is caput. This is actually the first time we’ve used it!

Now to go update it…

Well, The UPS and Generators Work!

It seems that a lot of downtown LA is going through a power outage right now. A bunch of us were walking back to the office from lunch at a sushi place, and saw lots of streetlights flashing red, and a few office buildings emitting smoke (from generators).

Luckily, our generators are humming along too, and the UPS system worked, and we didn’t see a single blip (except the air conditioning is off, and it’s getting warm in here)!

Hooray for backup power systems!

UPDATE COMING SOON!!

Why We Don’t Use CSC Codes

When buying things with a credit card online, you may have noticed on a lot of websites an area to enter the CVV2 value on your credit card. Officially, they’re known as “CSC” values (Card Security Code), but Visa calls them CVV2 codes, Mastercard calls them CVC2, and Discover and Amex call them CIDs. For the fun and simplicity of all involved, I’ll just use “CSC” the rest of this post. CSC codes are supposed to help fight fraud because in theory somebody would have to physically have the card in order to know the CSC code.

You may have noticed that DreamHost doesn’t ask for CSC codes.

There are two reasons for this. I’ll tell you reason number two first.

Reason number two is CSC codes don’t do a thing to help fight fraud.

The problem is, about 99.9% of all stolen credit cards used for purchasing things (like say, Web Hosting!) online are gleaned through the use of “phishing” scams. Those spams you get that claim to be from Paypal or Ebay or Wells Fargo or Bank of America. And, the Nigerians and Vietnamese not being total buffoons, they ask for the CSC code for your credit card too! So basically, anybody signing up for stuff online with a stolen credit card is either going to have the physical card (and therefore the CSC code), or will have the CSC code (and therefore have the CSC code).

In theory, using the CSC codes will stop that oh-so-popular case of credit card fraud where somebody goes searching through a trash can for receipts with people’s credit card numbers on it. Except, in practice these days just about all stores mark out the first 12 digits of your credit card number on their receipts.

In theory, using the CSC codes will stop that even-more-so-popular case of credit card fraud where somebody “hacks” into a merchant’s database of stored credit card numbers and compromises a bajillion cards all at once. Despite this being a very infrequent event compared to phishing scams, even when this doesCSC codes don’t help at all.

Why not? Well, think about it. Why is a merchant keeping all these bajillion cards in the first place? The only good reason is to be able to automatically rebill your credit card without you re-entering it every time. And that implies that they either don’t need to use your CSC code to charge your card (which is true.. they’re optional), or else they also have to store your CSC code… so it’ll get stolen too!

Either way, if the crooks get access to that database, you’re still cooked!

And now, the real reason DreamHost doesn’t ask for your CSC code.

Credit card processing online is a convoluted world. Every time you make a payment online with a credit card, there are four separate institutions making things happen. First, the website you enter your credit card at uses an online Point-of-Sale service like VeriSign’s Payflow Pro to pass your information electronically to a Merchant Bank (also know as an Acquirer) such as Cardservice International, who in turn uses a Processor such as FDMS Nashville to handle the actual credit card network and finally deposit the money into your Bank Account at a place like Bank of America.

All of this makes it an absolute pain in the bung to get anything changed in your credit card processing. So once you’ve got a system up and working, I whole-heartedly advice you to never touch it again.

WE haven’t!

We’ve gotten Cardservice to lower our rates a few times as we’ve grown, but other than that, our credit card system has been pretty much untouched since the late 90s… a simpler time when there were no CSC codes!

We originally got our merchant account through a place called WebOrder or something like that. I can’t really remember anymore.. they were an all-in-one place which also provided a secure site for ordering and all that. They were good for us back then because they didn’t have a set-up fee or any set monthly fees, and yet they also didn’t have horrible per-transaction rates. Before them we didn’t even need a merchant account because we were using some place that handled everything and just sent you a check every month, minus 15%. How generous of them!

Unbeknownst to us at the time, WebOrder (or whatever) set us up with a merchant account at Cardservice International, who used FDMS Nashville as our processor. This was all hidden from us, and we sure didn’t care! Things were fine for a while, until they were bought by Cybercash (or maybe we just switched to them?), another Internet Payment Gateway, and we had to change our stuff to start connecting to them. This was okay though, because it was a lot more professional to use our own SSL certificate instead of linking over to “https://secure.weborder.net/” or whatever it was.

Then, a year or so later, Cybercash got bought by VeriSign (for sure), and our hidden Cardservice International account came with us. This actually didn’t change a thing, as VeriSign kept supporting the old Cybercash MDK.. we’ve been using it ever since! Of course, it means that we can’t use all the advanced features of Payflow Pro.. features such as CSC support! And that’s it!

(One semi-funny aside.. when signing up for WebOrder, I used where I was living at the time as the phone number.. my parent’s home phone. Little did I know six years later it would still be showing up on people’s credit card bills! We were never able to figure out who could change it.. VeriSign passed us to Cardservice, Cardservice to FDMS Nashville, FDMS to VeriSign.. oh well.)

But, it’s time for a change.

Recently Bank of America came to us offering to be our Merchant Account (they’re already our Bank Account)… with much lower rates than Cardservice had ever been willing to give us. So we decided to make the switch! Which is why I’m suddenly so knowledgeable about all this stuff.

You see, when trying to switch VeriSign to use our new Merchant Account, we were informed that since our account was originally set up through Cardservice International, we are unable to switch our Merchant Account away from them. Instead, we just have to sign up for an entirely new Payflow Pro account with VeriSign directly.

And, to use a new account we have to use the new Payflow Pro SDK

Which I guess means Goodbye, Cybercash.

Hello CSC codes!