Yoohoo! What the f— just happened to my database?

July 8, 2005 on 1:36 am | In Foobars by Josh Jones |

My New Filing Technique Is Unstoppable

We had a little foobar with our registrar database tonight. Unbelievably it wasn’t my fault, although in a way everything is, since it’s my fooing database!

A number of statuses were wiped out, which resulted in about 4500 customers getting an email saying their domain had been successfully transferred to us! Yay!

Unfortunately, the last good backup was two days old.

How did we fix it?

First, Sage called me and told me all this stuff he’d like me to do.
Then, Sage mostly just did it all anyway.

Sage turned off the registrar service so no more emails would go out until everything was fixed.

Then, I figured out who just got emailed the crazy email, and sent them all another email saying “PLEASE IGNORE”. This stemmed the growing flood of confused support requests.

Andrea (a.k.a. the CULPRIT) also went in and replied to everyobdy who’d already written in.

Sage went in and wrote a script to quickly update the statuses in our system from the appropriate registries.

This didn’t help for domains with statuses like “pending-registration” and “pending-transfer”. For those we looked at a second table in our database with related data.

In fact, we decided to make the most of this opportunity to change the way all our code gets statuses from the database.. so now it uses that second table for all the special statuses we don’t get from the registry itself.. something Sage had said he’d wanted to do anyway.

Just not necessarily at 1am on a school night.

Once we’d made all the code changes necessary we copied them live and tried running our registrar service again. It pretty much worked, so we turned it back on for good and hit the hay!

Except for me, who decided to post about it here.

FP!

13 Responses to “Yoohoo! What the f— just happened to my database?”

  1. Chris Says:
    August 1st, 2005 at 1:59 pm

    Sounds like my life, only with less customers actually watching.

    especially:
    “First, Sage called me and told me all this stuff he’d like me to do.
    Then, Sage mostly just did it all anyway.”

    only in my story, I have to be Sage, because that’s the way things go sometimes…

  2. Ruby Sinreich Says:
    August 2nd, 2005 at 6:25 am

    I just want to say that I’m really happy that y’all are blogging and this post is a perfect reason why.

    Keep it up!

  3. Marc B Says:
    August 2nd, 2005 at 6:25 am

    As a current DH customer, I’m not so sure I *like* knowing what’s going on behind the scenes. I prefer to think of hamsters on wheels and magic as the driving force behind, say, registrar databases. I don’t want to know that at 1am someone’s coding frantically to get stuff back up and running. That gives me the willies.

  4. Charles Says:
    August 2nd, 2005 at 8:42 am

    It’s like this… I moved a number of domains from another rather expensive and unquestionably stuffy provider upon request from a partner. The change was mainly requested due to cost savings with our planned expansion, but only a little part of it from their perception of the service provided. To keep this short (I know Too Late), I must say that I would certainly have to take an extremely long look at leaving DreamHost for any reason now, even cost (but don’t raise them or I could change my mind -  ) I was a tad concerned when I realized there was not any support provided via phone but quickly realized the other options provided were ample and effective.

    Anyhow, when I read the blog “What the fuck just happened to my database..” I knew beyond a shadow of any doubt that I was connected to a group of people and a company which I hope to confirm with time, that employs real people, that seem to handle situations as most people “truly” handle a problem. Most being a relative term I know.

    Thank You for no bull shit and the perfect reality! Please keep it up!!!!

    PS. Was going to send a copy of this to some “Suit Upstairs” but didn’t want to raise awareness of this blindly incase they still maintain that stuffy corporate attitude I for one am so far over it I could scream… (Not Here of Course)

  5. Emma Says:
    August 2nd, 2005 at 11:15 am

    I have to say, that although sometimes I might think I’d prefer not to know, in reality - I much prefer knowing what’s going on. Even if what I know is that something unfortunately happened, or nearly happened - that’s a whole lot better than sitting there with things not working (or weird, unexplained e-mails) and not having a clue.

  6. Jack Says:
    August 3rd, 2005 at 4:37 am

    A web host, not only admitting to screw ups but detaling them.

    You guys must be trying to hide something much bigger :-D

  7. Timur Tabi Says:
    August 3rd, 2005 at 8:59 am

    Just a simple request - please no foul language in the blog. I’m not personally offended by it, but I just don’t think it’s necessary. It’s also unprofessional. It also makes it harder for me to recommend DreamHost to others.

  8. Daniel Says:
    August 3rd, 2005 at 1:41 pm

    Eh… Leave the foul language. I find that seeing the personalities behind the wonderful webhosting company that I’ve come accustomed to, is refreshing.

  9. TIm Says:
    August 4th, 2005 at 7:05 am

    I read the “yoohoo” as Yahoo and so re-read the entry like 4 times trying to figure out how Yahoo screwed everything up.

  10. Richard Says:
    August 4th, 2005 at 10:08 pm

    I read the “yoohoo” as Yahoo and so re-read the entry like 4 times trying to figure out how Yahoo screwed everything up.

    Glad I’m not the only one! ;)

  11. jake Says:
    August 8th, 2005 at 5:45 pm

    any copyright on that image? if not, that will most definitely be my new mysql_connect error indicator.

  12. Remon Says:
    August 16th, 2005 at 10:14 am

    I find it a very cool that you guys have your own blog… It kinda makes it a whole lot more personal :) And the foul language doesn’t bother me at all (as long as it isn’t overdone)… The way of writing gives a more human feel to it than blogging in 100% political correct english ;)

  13. Unofficial DreamHost Blog » Blog Archive » DreamHost Year in Review Says:
    January 1st, 2006 at 1:26 am

    [...] Community This year DreamHost launched the DreamHost Community Wiki, which is now featuring more than 150 articles, and launched the official DreamHost Blog. [...]

Powered by WordPress. Pool theme by Borja Fernandez, modified by DreamHost.
Like WordPress? Consider attending WordCamp LA.
Entries and comments feeds. ^Top^