Power Outage Update
September 12, 2005 on 6:38 pm | In Foobars, Insider View, Updates by Josh Jones |
Okay, this sucked. Literally 10 seconds after Nate posted that last entry, we received an announcement over our loud speakers that our building was being evacuated and we had to leave immediately! (Also, sorry about that post being temporarily taken down, Nate thought it would be better to remove it since it was so dumb, but methinks better to just add something else!)
Unfortunately, shortly thereafter we figured out (thanks to EVDO!) our sites were no longer accessible, as our three network providers Level 3, Global Crossing, and Mzima were all down. Our physical servers were okay, and probably really happy sitting in the data center (we were no longer able to access) with nothing to do.
But that didn’t last long. Shortly thereafter the entire building where our data center is located’s back-up generators (there are SUPPOSED to be four) stopped working, and all power was gone. We were able to get back into our data center then, and it was like the day after tomorrow or something. Really creepy just walking through rows of dark, quiet, dead server after dark, quiet, dead server. We’ll post some pics from that soon..
So, now, finally power is back up and servers are being restored. We lost a net app and had to replace the head unit from a hot spare. A few servers were of course wonky, and right now three bad cabinet switches are still being worked on (so many web sites ARE still down).
There will be more posts soon, and hopefully everything will be up within the next hour.
Also, just FYI for people who don’t know, we keep http://status.dreamhost.com/ in San Jose, completely off our network, just to update people in times like this when our panel (still down) and everything is caput. This is actually the first time we’ve used it!
Now to go update it…
168 Responses to “Power Outage Update”
Categories:
Archives:
- June 2009 (3)
- May 2009 (3)
- April 2009 (4)
- March 2009 (2)
- February 2009 (3)
- January 2009 (3)
- December 2008 (3)
- November 2008 (4)
- October 2008 (5)
- September 2008 (2)
- August 2008 (5)
- July 2008 (2)
- June 2008 (3)
- May 2008 (6)
- April 2008 (2)
- March 2008 (3)
- February 2008 (5)
- January 2008 (7)
- December 2007 (3)
- November 2007 (3)
- October 2007 (5)
- September 2007 (3)
- August 2007 (4)
- July 2007 (4)
- June 2007 (7)
- May 2007 (3)
- April 2007 (3)
- March 2007 (8)
- February 2007 (4)
- January 2007 (6)
- December 2006 (5)
- November 2006 (2)
- October 2006 (4)
- September 2006 (4)
- August 2006 (5)
- July 2006 (3)
- June 2006 (3)
- May 2006 (5)
- April 2006 (6)
- March 2006 (7)
- February 2006 (6)
- January 2006 (8)
- December 2005 (7)
- November 2005 (5)
- October 2005 (8)
- September 2005 (19)
- August 2005 (11)
- July 2005 (4)
Other Sites
- DreamHost Status - An off-network page for all DreamHost Announcements! (Not as silly as this blog.)
- DreamHost! - The host behind the blog.
- Forum - Discuss DreamHost and web hosting!
- Web Control Panel - Manage your account.
- Wiki - On DreamHost and web hosting.
Powered by WordPress. Pool theme by Borja Fernandez, modified by DreamHost.
Like WordPress? Consider attending WordCamp LA.
Entries and comments feeds.
^Top^
September 12th, 2005 at 6:42 pm
Any updates?
September 12th, 2005 at 6:49 pm
You guys are doing a heck of job -
/ *Edit this out*
and please don’t make me look like a fool in front of paying customers for a 3!rd time -
Get that San Jose office a big fat fiber pipe that can co-locate **at least mail** MAIL is critical
end edit/*
September 12th, 2005 at 6:50 pm
Any updates indeed…. it is disappointing that DH is still down, all these hours after the power outage.
September 12th, 2005 at 6:55 pm
Yes, this kind of down time for mail REALLY needs to be dealt with. I can easily live with losing the servers, but having my mail go down is extremely bad.
September 12th, 2005 at 6:56 pm
Get Gmail! :D
September 12th, 2005 at 6:57 pm
My site appears to be back, but is returning a MySQL error. I can reach all the other pages fine, but the php pages (like the index) are poop.
September 12th, 2005 at 6:58 pm
Well done on getting everying (mainly) back together again. I had to laugh, as I was just about to start a meeting introducing some people to the wonderful dreamhost. We had a power failure in Adealide, South Australia for around 5 hours two weeks ago, which was the final straw in finding external hosting. So there I was about to demonstrate how much better off we would all be with this new host and … nothing :)
The status site saved my butt, and all concerned at this end are happy enough.
cheers all
September 12th, 2005 at 7:00 pm
updates? one of my sites i still down :(
September 12th, 2005 at 7:01 pm
Wow, thanks for status.dreamhost.com! I’m still waiting for my site to come back up, I hope everything is going well over there at Dreamhost HQ, it seems like you all are handling the situation very well.
September 12th, 2005 at 7:04 pm
Does anyone know if email sent to dreamhost customers during the outage is now completely lost? Did outside senders to dreamhost recipients receive a delivery error message?
September 12th, 2005 at 7:04 pm
We’ve all got gmail, yahoo ,hotmail, ICQ, this that and the next waste of time( well at least some of us)
When the power is off ..
we don’t get email forwarding, autoresponding, etc etc -our stupid little blackberry gets nothing our pathetic palm SYNCS nothing and I’m sure the next IPOD will be worthless also
September 12th, 2005 at 7:04 pm
Yes, status.dreamhost.com has been very informative/helpful - and so has Gmail ;)
September 12th, 2005 at 7:05 pm
Uh oh.. everything is more or less up, but the building JUST told us.. the power MAY GO AGAIN.. the generators are running now, but we’re on UPS.
IF we do go down again, it’ll be much quicker coming back up, and hopefully the building guys are just scaring us, and us in turn are just scaring you.
Stay tuned.. to status.dreamhost.com! Blog.dreamhost.com would go down if power goes..
September 12th, 2005 at 7:10 pm
“Everything” doesn’t seem to be including my sites or mail. Maybe a little more detail on the status site would be helpful.
September 12th, 2005 at 7:11 pm
Hi Guys,
Looks like the mail servers / mysql servers my sites use are still out of commission.
Any word when we’ll have service again? My customers are getting cranky.
Hope you’ve not been to badly affected.
Relevant Domains:
http://www.flickkerbug.com (email especially!)
http://www.accesshealthcare.co.nz (database especially)
http://www.pacificpassages.com (email especially!)
http://www.emporio.co.nz (email especially!)
Cheers
Cam
September 12th, 2005 at 7:13 pm
Every time I find a new host, my site goes down shortly. Sometimes the server is hacked because of poor security (not to hack my site, but they were shared), but most often because I chose crappy providers who just have tons of downtime.
So I finally found the best hosting company in the world, and they have a citywide power outage!
I take full responsibility for putting my cursed web site on your servers. Sorry. =)
Having helped run an ISP, I can understand that no matter how perfectly you set things up, something WILL go wrong. I can also say from experience, that it is really cool to at least have an actual news event to explain your outage, rather than having to just say, “oops.”
(It is also REALLY cool for it to be someone ELSE’s problem, rather than hosting my own site!)
September 12th, 2005 at 7:15 pm
Hey there.
The nice thing is that my site, or at least the architecture thereof, is up and going. However, the MySQL servers aren’t talking to my site — which is bad news for WordPress. So keep up the good work and hopefully we’ll all be happy tomorrow.
September 12th, 2005 at 7:16 pm
The people are working hard to get everyting up again, ya’ll need to chill out, go out for a walk, take a breath till it’s up again. :)
September 12th, 2005 at 7:16 pm
Please fix the NTP servers at
66.33.216.{11,130,131,23}. Only two (out
of six) do work correctly (66.33.206.5; 66.201.54.67).
Just trying to help your customer (hopefully, they
wont become ex-customer if you fix it soon).
Thanks and regards.
September 12th, 2005 at 7:16 pm
I think you guys handled the situation as best you could. You’re not the only hosting company that went down. Thank you for keeping us up to date and for getting customer sites back up before worrying about your own site (except for this one, which has been very informative).
I have 10 domains on my account and my phone and IM were a little nuts today with folks who couldn’t get to the email or sites. It helped that I was able to give them your status page URL.
I think email during the outtage was held and will come through. I had someone mention that they sent me an email 5 minutes after DH went offline around 4 pm (EST) and I did receive it at 7:50 pm.
September 12th, 2005 at 7:22 pm
Arg, I can understand your frustrations and that you generally have system backups and that a city-wide power outage is pretty, well, catastrophic, but I’m thinking you need to invest in more backup generators or something. And test them often so that you know they kick in when they’re supposed to. My clients depend on me, my email being backed up caused a headache on several levels (one got stuck in outbox, a deliverable web graphic expected by end of day, juuuust as things went down - curse my metal body…I wasn’t fast enough…), and I CAN’T USE MY BLOG! Must. Have. Blog.
I know you understand. :D
Is it possible to split up your site into a couple different cities so that, say, mail doesn’t go down when websites do and vice versa? I could stay sane that way. Well, I hope we all learn something from this anyway. (I still like you guys.)
Lynne
September 12th, 2005 at 7:25 pm
Well my web sites (and e-mail, announcement lists, blogs, etc.) are still down. :(
Can someone please verify that our e-mail is at least bouncing back to senders now rather than going into a black hole (as mine did in 2003)? Thanks!
Good luck getting everything back up and running.
September 12th, 2005 at 7:26 pm
This happens so rarly that putting servers in another city would be a waste of money and Im sure everyone wanna pay less possible for the hosting…
September 12th, 2005 at 7:27 pm
Is it possible that a MySQL database could have been erased due to the power outage? One of my databases is empty, and the Wordpress site that uses it is broken, whereas another of my Wordpress sites which uses a different database is working fine.
September 12th, 2005 at 7:29 pm
My poor students are high and dry, and serves them right for thinking they could leave their assignment books in school because my website always has the assignments :) Hope we’re back up soon, though.
September 12th, 2005 at 7:30 pm
Sorry, but your comment has been flagged by the spam filter running on this blog: this might be an error, in which case all apologies. Your comment will be presented to the blog admin who will be able to restore it immediately.
You may want to contact the blog admin via e-mail to notify him.
What is this?
September 12th, 2005 at 7:31 pm
Wow. status.dreamhost.com has been a great tool, especially in finding out why my site and another popular one (halobabies.net) was down. Thanks for the updates!
September 12th, 2005 at 7:31 pm
I’m just glad the service was down for a pretty good reason! Can’t fight major power outages I guess. My web site is now kicking and hopefully my email service will be restored soon ;-)
September 12th, 2005 at 7:31 pm
Dreamhost should hire the guy from DirectNIC. He’s moving to California, you know, and looking for a job.
He managed to keep DirectNIC running - that is, running without a single second of downtime throughout Hurricane Katrina and its aftermath. Dreamhost could take a few lessons from that.
September 12th, 2005 at 7:31 pm
They must be happy, no hws for today! :D
September 12th, 2005 at 7:32 pm
Site is up, but it is pretty much useless because MySQL is still down. I hope this restart is almost finished.
September 12th, 2005 at 7:33 pm
I’m a long-time customer and I have a lot of sites hosted on Dreamhost, many of which are on different servers, using different IPs.
Well, -none- of my sites is up. No Web, no mail, nothing.
And guess what… my main competitor, Dreamhost’s customer from last March, has its only site up and running! :-D
September 12th, 2005 at 7:33 pm
LOL why not get Harry Potter to help them out while we’re at it?
September 12th, 2005 at 7:36 pm
Actually, I was wrong: my web panel and e-mail are up and running now! Only my websites are still unavailable. We’re getting there…
September 12th, 2005 at 7:37 pm
Or just put it on http://www.blogger.com….
September 12th, 2005 at 7:38 pm
Better yet, Mike - pick a friggin’ place that isn’t on a coast at all - something between a large city and a smaller one, that wouldn’t be a target for hurricanes, floods, landslides, attacks, or rattlesnakes. Why rattlesnakes? Because they are nasty.
Or Boston. Pick Boston. Then I can knock on your door when this stuff happens. ;P
September 12th, 2005 at 7:39 pm
email pleaaaazze :)
September 12th, 2005 at 7:39 pm
I am no NIC expert, but regarding UPS and such, if power outage is widespread I do not think UPS does any good anyway. Certainly some piece of data equipment down the line would be effected. Sure UPS is important for data integrity and to make it easier to come back on line when networking is restored, but I don’t think it would have saved Dreamhost’s bacon in a case like this.
September 12th, 2005 at 7:39 pm
Canada would be better. Btw http://dreamhost.com/ is back up!
September 12th, 2005 at 7:40 pm
From what people are saying the whole damn pipeline went down, correct?
Decentralize, people, decentralize. Or the rattlesnakes will get us.
September 12th, 2005 at 7:41 pm
Granted, it has only been an hour since the last update, but it is about 30mins too long.
Any idea about when anything will be up?
Email? Database? Sites?
September 12th, 2005 at 7:43 pm
By the way, can I get some of the crack that these stats are on? On the system stats page it reports that all of my servers have been up 100% of the time in the last day. What’s up with that?
September 12th, 2005 at 7:45 pm
Woot! I’m back up! Thanks Dreamhost! Now to back up my database, just in case.
September 12th, 2005 at 7:49 pm
Not fair!
September 12th, 2005 at 7:50 pm
My webmail is still down:
# MySQL Error: Lost connection to MySQL server during query
:-(
September 12th, 2005 at 7:50 pm
My database just came back up too! It’s on the slow side, but it’s there.
September 12th, 2005 at 7:51 pm
Thanks guys. :) We appreciate all of the hard work you’re doing. It means a lot.
September 12th, 2005 at 7:51 pm
E-mail and control panel are back, although I agree with Ruby about the crack the stats page is running on. Now if I could just get the sites back. (7:50 pm PT).
September 12th, 2005 at 7:53 pm
This is one of those few instances when having the time on comments would help, so that when Josh says things might go down again we’d have an idea WHEN he said that.
September 12th, 2005 at 7:54 pm
[...] Update: Dreamhost has (of course) blogged the event here. [...]
September 12th, 2005 at 7:54 pm
All three of my web sites are still down hours after power has been restored…no updates on the emergency status page or blog…what gives ??
September 12th, 2005 at 7:55 pm
My sites are back up, but:
Site Temporarily Unavailable
We apologize for the inconvenience. Please contact the webmaster/ tech support immediately to have them rectify this.
error id: “bad_httpd_conf”
Glorianon.com
graphicbuds.com
tamlincat.com
Besterp12.com
It’s great to see some sign of life though.
September 12th, 2005 at 7:58 pm
Question…
The boxes should not only be on generator power, but also on UPS. If the UPS start to run out of battery power, all the servers should begin their shutdown before the UPS lose power.
I assume something went wrong? No UPS software installed correctly? Obviously when machines go down there is always a problem with something not coming up, but that’s normally only if you remove the drives or work on the hardware or move the servers. Even if you lose a drive you should have multiple hot spares setup to rebuild without loss of data.
I am very interested in knowing why you ‘lost’ some equipment and why some DB are not coming up. I do computer adminstration for a living and we have gone through similar problems but when the generator runs out, then the huge UPS battery back for our floor kicks in. We could probably go 24 hours without power, depending on temperature.
If you could answer some of these questions I would appreciate it.
Ray
September 12th, 2005 at 7:59 pm
Um. I realize y’all are probably having a grand ole time, having been in the trenches when an international company is down and my boss is outside my cube running interference for his boss, who is talking to his boss because of what I haven’t gotten back online yet….
But. Once you have it back up, I’m gonna expect an explanation, including who owns the generators, how often they test, who owns the UPS, how often they get tested, and an explanation of why the servers either weren’t shudown gracefully, or why there wasn’t sufficient backup servers or cluster resources to bring everything up in a limping mode while the hardware that wouldn’t power up gets fixed.
From here, it looks to me like you lost power on running servers due to lack of an automated shutdown, or inadequate UPS capacity, neither of which are things I want to keep paying you to not plan for, and I look forward to a post-mortem report that says different.
September 12th, 2005 at 7:59 pm
Hey guys - time for a new update post - this is 50 comments… also - can you confirm if email has been bouncing or is stored…?
September 12th, 2005 at 8:00 pm
Hey, folks, just an FYI about UPSs — their purpose isn’t to keep electrical devices constantly running during a power failure. That’s what generators are for. UPSs are big batteries that give you those precious 5 minutes to properly shut down connected devices instead of grabbing that extension cord in both hands and yanking with all your back muscles.
September 12th, 2005 at 8:02 pm
DH - you guys have had a busy day. Get a beer, run over the lessons learned so you can do better next time.
Everyone else who is kvetching - what do you expect? You don’t pay that much for the service. DH is for the money the best provider we’ve had.
If you want 24×7 uptime, redundant network paths and etc you’re can get that - but you’ll pay for it.
September 12th, 2005 at 8:03 pm
Not Terrah!
The whole of neurobashing.com was off the grid for a long time today… due to power outages in L.A. ……
September 12th, 2005 at 8:04 pm
Def. DH needs to look deep into what went wrong with their UPS system and why things did not shut down gracefully. I hope they do.
But IMHO if you are running a Web server for an International company at DH I think you are nuts.
September 12th, 2005 at 8:04 pm
And I meant to say BIG international company. I.e. some Fortune 5000 (if that exists) or the like.
September 12th, 2005 at 8:05 pm
Oww! Talk about embarassing!
Some snotty web guy that is taking over one of my customers sneered at DH today for lacking proper “environmental controls”.
F@#$ him, but really guys — where’s all the cool data center power backup stuff when you needed it???
Embarassed, but will get over it soon.. /jad
September 12th, 2005 at 8:08 pm
Now e-mail works but all my web sites are still down. :( It’s an improvement, though.
September 12th, 2005 at 8:10 pm
It’s just my personal mail ond domain that’s down, sorry if I wasn’t terribly clear.
Although I’m suddenly wondering how much Directnic costs….if anyone has generator expertise right now, it’s them :)
September 12th, 2005 at 8:11 pm
everything working but notoriously slow!!
September 12th, 2005 at 8:12 pm
My websites are all still down and mail forwarding is down as well. 10 pm central time US. Please update us, thanks.
September 12th, 2005 at 8:13 pm
I guess I’m a little confused. What’s the point in having big UPS systems and generators if everything goes dark anyway? *Sigh*
September 12th, 2005 at 8:16 pm
Cut them a little slack about the server shutdown thing. I’ve had plenty of servers (Dell 2650’s, 6650’s, etc.) just decide to not start after a proper shutdown. Usually the disks decided to give up the ghost. A few times we lost the controllers in the NAS devices, resyncing disks, restoring terabytes from backup. These things take time.
Keep up the good fight Dreamhost. (and i’d recommend renegotiating your contract with the company in charge of the generators. at the least yell and scream at them for 20 minutes)
DR’s a pain
September 12th, 2005 at 8:17 pm
Regarding e-mail non-deliveries - it’s going to be a function of the total time DH is down and the settings on other providers’ servers. On the mail server I run for work we’re set to try to deliver mail for up to 3 days. So as long as a remote server gets its act together within 72 hours, mail we send will be delivered. If I had set the trigger to 6 hours, however, I’d already be bouncing mail that was destined for DH servers. But that would be a very, very unusual scenario. Most MTAs would be set for at least 2-3 days.
Many systems *will* send back warning reports to local users informing them that mail has not been delivered *yet*, which is not the same as saying it will not be delivered. Again, this is configurable by the remote server admin.
Hoep this helps alleviate some fears. Chances are, assuming a relatively quick return to normalcy (tick, tick, tick…) all e-mail should eventually be delivered.
September 12th, 2005 at 8:17 pm
One domain up (inspiredsnl.com), one down (spreadburys.com), web panel functional. Standard webmail page can’t connect to MySQL; “aftermarket” SquirrelMail is functional. Mail hosts are also down.
September 12th, 2005 at 8:18 pm
Okay, FTp access isn’t there which makes sense if they’re working on quid. Sites are still down. Attempts to go to the main page on any sites get the above message, attempts to go anywhere else get 404′d. Hopefully it will all be fixed before tomorrow morning, or I’m really going to be getting complaints. (teamb.tamlincat.com hosts a listing of assignments for the local Jr. High and some of the parents can be very annoying. ONe of the reasons I dropped PHPWebhosting actually. This is the first time DH has given me any problems so I’ll give you the benefit of the doubt and assume you’ll be faster at recovery if this every happens again.
Heh, also, the panel is back up, but you may want to update the stats at Status->Server Status
Up Forever
Up Forever
Up Forever
September 12th, 2005 at 8:18 pm
Just a quick report: now all my sites and their MySQL databases are up. Webmail is not (Can’t connect to 10.3.67.49 as squirrelmail).
Things are definitely moving!
Back to work, now. I have to publish my daily news! :)
September 12th, 2005 at 8:19 pm
Get rid of the blog and educate these peePz as to what life really is
most have no idea what it’s like to be down
every webhost has downtime
and always will (deal with it my puppy’s)
every site will go down (and)
every site will go back up
EVERY webhosting service will see this
EVERY WEBMASTER will deal with this
alert your customers and life will return to normal
once again *forthosethatmissedit*
GIVE US MAIL
September 12th, 2005 at 8:24 pm
I’m back up, I’m happy to report. And given that my previous hosting provider used to go down simply because it was Tuesday, and would never respond to email queries, and would get their servers back up pretty much as soon as the trained gerbils got the wheels powering the whole operation back up to speed, I’m not going to complain about a little blackout-induced site withdrawal.
But I do hope the DWP figures out who crossed the streams and has a good, long talk with them.
September 12th, 2005 at 8:28 pm
Outage
The Map Room was unavailable for nearly seven hours today; my hosting provider was knocked offline by the power outage that hit Los Angeles this afternoon. All my sites were offline until just a few minutes ago. More posts tomorrow….
September 12th, 2005 at 8:30 pm
My site is still down :( I actually recommended DH to a friend yesterday and created some referral code. When do you guys think things will be fixed? Thanks.
September 12th, 2005 at 8:31 pm
All my sites are still down. At least I had the emergency status page to send to my domain mail users… How I miss my e-mail. I’d only suggest you to write down GMT hours in the emergency status page, as I had to Google for PDT to know when BRT was it updated.
September 12th, 2005 at 8:32 pm
Received no mail between 3:44 and 8:06PM. Nothing’s trickled in since then, which is rare for me. What happened to the mail that was lost during this time?
September 12th, 2005 at 8:34 pm
Mail delay info …
I believe many mail servers use a decay algorithm to attempt resends. So the longer a site is down (DH), the longer it will take for the mail server that has mail bound for you to attempt redelivery.
I searched and qmail — a popular MTA — uses a “quadratic retry schedule”, which can mean a long time for mail to get delivered after an extended outage. See www . lifewithqmail . org/lwq.html#retry-schedule
September 12th, 2005 at 8:34 pm
I can totally see that it was through no fault of your own (unless DH can cause city-wide blackouts?), and of course things happen. Sucks that the generators went down, but that’s life. Glad everything is being fixed. It’s things like this that made me come back to DH… As opposed to hosts that went down and never came up (ugh), went down daily, etc. Ya’ll rock. Keep it up!
To some commenters who are insisting that the Server Status is off… There’s a reason for:
:)
September 12th, 2005 at 8:37 pm
It’s 10:30 pm central and my email is still down. Website is up, for which I am grateful. But with local primary elections tomorrow, it is really unhelpful to not have email.
September 12th, 2005 at 8:37 pm
Oh yeah still much better than other servers I’ve been on. Although all of them started out as having great service. I was on Prohosting for a long time. Time was we could call them up if we had a problem and always get the same tech guy and we’d always be right back up. We even worked together to get the (then) new Mod_rewrite installed on our server when leechers started realy hitting our bandwidth.
Later on as they expanded, they got new people, one in particular knew less about their systems than I did and things started taking longer to get fixed. Finally their billing dept started billing me on random dates and screwing up my balances(two bank withdrawls withing one month for someone who lived month to month at the time is very bad.)
So I looked around to see what was widely recomended. That was PHPWebhosting. They too were great for a while. Then around last Jan they turned useless. So I decided to try DH. I’m really hoping this not shutting down safely was a one time thing because they were caught off gaurd and not the start of a downward spiral, but I’ve just been burned in the past.
It will take much more than one outage and a few complaints from visitors for me to switch though. Especially at this price.
September 12th, 2005 at 8:38 pm
One would think the data centers and network providers would have been a bit better prepared with back-up to the back-up and/or power disaster recovery in place.
Many DCs were taken off line today…wake up call to all providers, data centers and ISPs.
September 12th, 2005 at 8:42 pm
I figured something like that, just nice to find one bit to laugh at in this.
September 12th, 2005 at 8:43 pm
All of my clients sites are still down, all 8 of them,,
email is still down for all of them..
September 12th, 2005 at 8:45 pm
I’m certainly not ready to throw in the towel with DH - hell, I’ve been with them since January 1999 and this is only the second downtime of any real duration I’ve ever experienced with them (and this is much worse than the last I recall).
Keep up the hard work guys, but please do get my site, clanhobbs.org, back as soon as you can - it’s not much but it’s home :)
September 12th, 2005 at 8:47 pm
So, do we get some credit? :)
I left Hosting Matters two weeks ago because they were becoming totally inept at what they were supposed to be doing. I still have 41 sites there, and the site I have here at DH is my “test” before I decide to move everything.
Up until today, I had really no complaints — except for a couple of minor items that I am learning to live with. How DH handles this issue will go a long ways in helping me make my decision to move everything over here.
It is now 8:45 PDT and my site is “up” but minus the SQL server. Being a WordPress site, not having the SQL means that it STILL won’t work though.
I suppose I will have to put an index page up to explain to my readers that Sh*t happens and the pooper scoopers are on their way…
September 12th, 2005 at 8:47 pm
Just an update (8:45pm PDT) .. we’re down to about one half rack of servers (a dozen or so) still down, due to a wonky switch. Hopefully everything will be ironed out within the next hour.
Here’s a quick run-down of how our power works.. we’re in a building downtown that is sort of specialized as a data center. There are lots of data centers in the building, and the building itself (which used to be a large bank’s internal data center) provides the power to the tenants.
This is (supposed to be) “good” power. IE, the building itself has huge UPSes and four large backup generators that _are supposed_ to be enough for all the data centers.
When the power went out for the city, the UPSes came on, and things were fine (that’s when Nate made that last post about us being okay). However, as they started to run out the generators had to kick in. It turned out, and this is the boo-boo, the generators could NOT generate enough power for all the data centers after all! And so, rather than just doing some sort of lower-level of power output, they are apparently configured to all just shut themselves off when things are overloaded.
So, basically a mini version of the LA power outtage happened right inside our building.
The grid is definitely back on now, and the reason the generators were on earlier was the building engineers are testing things and messing with stuff.
Ironically, the building had been telling us they were in the process of getting an extra generator installed because they were afraid they were starting to get to their power limits.
Yup.
September 12th, 2005 at 8:48 pm
QUOTE: “Hey, folks, just an FYI about UPSs — their purpose isn’t to keep electrical devices constantly running during a power failure. That’s what generators are for. UPSs are big batteries that give you those precious 5 minutes to properly shut down connected devices instead of grabbing that extension cord in both hands and yanking with all your back muscles.
Comment by Rome — September 12, 2005 #
”
HAHA 5 MINTUES? If you are running a small one out of your house. At our work our UPS can keep up our whole floor for 8 hours. Even my home UPS, which I have two 1500VA UPS and some smaller 1000VA UPS can keep me up for over an hour with my equipment.
5 MINS LOL, this is supposed to be a DATACENTER!
September 12th, 2005 at 8:50 pm
Oh thanks, not fair at all.. Dreamhost gets their page back up but I still have mulitple client sites down at 8:50 PM pst..
September 12th, 2005 at 8:55 pm
Whoa, if I try to access dreamhost.com I get redirected to:
http://www.dreamhost.com/dedicated/dedicated/dedicated/dedicated/dedicated/dedicated/dedicated/dedicated/dedicated/
and its not found. This is as of 11:55 est.
September 12th, 2005 at 9:02 pm
12:00am Eastern and traceroutes to DH nameservers are still choking on: fe1-mz.dreamhost.com (216.193.192.50).
AFAICT, personal domains are equally fucked.
Still that wonky switch or is there another grid-related outage? Why not update http://status.dreamhost.com/ instead of taking the time to *not* blog about what’s still not up?
September 12th, 2005 at 9:03 pm
You don’t own your own UPS, then?
September 12th, 2005 at 9:03 pm
9pm PST- 8 hours of total downtime, all services and sites.
I chose Dreamhost. I like Dreamhost. I want to stay with Dreamhost. Tell me why Dreamhost is located in a city that is widely acknowledged to be a leading candidate for a major earthquake, one of the choicest terrorist targets, and in the middle of a miserably fragile electrical grid (I lived there through the rolling blackouts a few years ago), yet is not adequately prepared for the electrical failure that could be caused by any of those three events? I include your choice of buildings, network providers, whatever in your blame- you chose them as part of your job, just as I chose Dreamhost as part of mine. What is important is what we choose to do now.
I recommend in the strongest possible way you investigate moving the location, decentralizing, or convincing us this won’t happen again through your extensive new equipment, planning, and new network providers. It doesn’t matter how good the deal is or how cool the ISP may be if my site isn’t there when it needs to be. No jokes about rattlesnakes here, just a dead serious note that you can show to your network providers- to the building landlord who maintains the generators- to anyone else responsible for this outage, so they realize this is very serious for some of us.
If in the end, after all the reports on what happened are filed, and the general electrical infrastructure is the sole culprit and makes it not possible to provides secure services from Los Angeles, then one of us will be moving on- your choice.
Thank you for your efforts, PLEASE get my site back up soon, PLEASE insure this won’t happen again.
David
September 12th, 2005 at 9:09 pm
Well, congratulations to the people who have sites back up … for awhile one of my four was available, but now they’re all down and email is punk as well. I wish Dreamhost was into a little more customer communication–the “fully operational very soon” posted on the status page at 6:46 pm has proven not to be true. (When that was posted, I at least had one site that was still up!)
Could ya at least update the status page? Jeese.
September 12th, 2005 at 9:09 pm
No offense guys, but I’m a bit disappointed at the lack of back up power… you should take some notes from directnic… they were able to keep their customers online throughout Hurricane Katrina. Read their blog: http://www.livejournal.com/users/interdictor/
It’s a tough situaton, I’m sure, but I’ve been down over 12 hours…
September 12th, 2005 at 9:11 pm
[...] This site and related sites, including inhisname.net, caffeinated.org, and newtomorow.org sites were down for a few hours due to a severe power outage in the Los Angeles area today. This brought down our hosting provider, DreamHost, when issues with backup generators led to a shutdown of systems and indeed entire major routing nodes. Human error strikes again. [...]
September 12th, 2005 at 9:13 pm
Take this quote from the post…
“Unfortunately, shortly thereafter we figured out (thanks to EVDO!) our sites were no longer accessible, as our three network providers Level 3, Global Crossing, and Mzima were all down. Our physical servers were okay, and probably really happy sitting in the data center (we were no longer able to access) with nothing to do.”
Narrow it down to this…
“our sites were no longer accessible, as our three network providers Level 3, Global Crossing, and Mzima were all down.”
It doesn’t matter if their servers themselves were online - their network providers were all down - they couldn’t have had (much) access if they wanted to…
Besides - being in one of the largest cities in the world - in one of the best weather areas (Southern California - aka “no seasons - year round calm”) should be plenty safe enough for a datacenter… Then some idiot from the utility company screws up the city - and that’s proved wrong…
Basically - I may be wrong, but Dreamhost theoretically couldn’t have had our sites up anyways… as the network providers were down too… At least, how I’m reading it…
-Derik
http://www.derikolsson.com
September 12th, 2005 at 9:18 pm
“Just an update (8:45pm PDT) .. we’re down to about one half rack of servers (a dozen or so) still down, due to a wonky switch. Hopefully everything will be ironed out within the next hour.”
It would be useful to have a bit more detail. A dozen or so out of how many servers? And what about email? That still doesn’t appear to be working.
September 12th, 2005 at 9:18 pm
Derik: Agreed, yet how can Dreamhost claim to have any semblance of “control” or “accountability” by way of this blog or the status-dot… both of which are apparently still working… yet no updates to either… and no service to customers.
If it’s not their fault… that’s fine. Tell us what’s going on. If it’s not their fault, then they have the time to tell us what’s happening.
September 12th, 2005 at 9:21 pm
“…fragile electrical grid (I lived there through the rolling blackouts a few years ago)”
You do know those were fake, If I remember correctly that was part of the whole Enron mess.
As for this situation, I know that all the people in the datacenter are going to be working extra long and probably throughout the night. Murphy’s law anyone? Give them a tiny little break. Maybe they’ll throw in a gift and x10 our space and bw… ;p
September 12th, 2005 at 9:22 pm
Been with DH for the past two years, and I’ve been very happy. Today has been a mess, but overall I’ll still recommend DH.
September 12th, 2005 at 9:26 pm
Please, I need some help. I have a dedicated server that lost mysql & mail service for a few dozen sites (for whatever reason) 48 BEFORE the power outage. I don’t know why it was taking so long to get a support response, but now with extra time added on for the outage, my customers are getting irate (especially over the email).
Are the old support requests still being dealt with, or are they completely on the back burner for now? (I know, I know, bigger fish to fry…)
I’d love to at least have an acknowledgement that somebody has read my support request.
Thanks.
September 12th, 2005 at 9:26 pm
Re: David
Earthquakes aren’t going to take down Los Angeles - any Californian knows that - you’d need one of the largest earthquakes on record to damage the city… A rarity that is about as likely as another Katrina hitting.
“Fragile Elecricity”
The rolling blackouts are a thing of the past - I too lived there, and have gone back many times since then. There are ~11 more nuclear power plants in California - use up as much juice as you’d like :).
Re: Gary - they’ve been explaining what’s been going on throughout the time since they’ve gone back online… Now apparently they’re fixing some servers that are having problems from the power surge - and are almost back to 100% again. Ctrl-F, “josh”, you’ll find his updates.
-Derik
http://www.derikolsson.com
September 12th, 2005 at 9:30 pm
My power went out and came back on around 2pm. It is now 9:30 and 7.5 hours later I still have no email. Are you rebuiding these servers from the ground up? We were hit by a power outage, not a cat 5 huricane. This amount of downtime is unexpected.
September 12th, 2005 at 9:30 pm
Hooray, all my sites are back up. Now I can go to bed, sleep and have sweet dreams. :)
September 12th, 2005 at 9:32 pm
i’ve been watching the downtime today and have seen several of my peer’s sites go live one-by-one over the past few hours. so, it is obvious that the DH guys are working hard. give them a break. granted, mine is still down, but hey, what can ya do?
thanks for your hard work, guys. as for the technical problems, we know you’ll reassure us after all this mess that it has been taken care of.
thanks again for years of great service.
September 12th, 2005 at 9:33 pm
My three sites are up including email and ftp access. I second the x10 space!
September 12th, 2005 at 9:34 pm
My sites seem to be back up.
I think Dreamhost did allright, I mean, the data didn’t get erased, which is key, but there’s definitely room for improvement. I’m sticking with Dreamhost on this one, hopefully it’s not a mistake. Just promise me you guys will get those generators.
September 12th, 2005 at 9:36 pm
Derik: I’ve read all of Josh’s comment posts. So, I grant that the comments section has been updated… but not the original blog post (quick-and-dirty, up-front update on the post will save a lot of customers a lot of time with the scrolling and the apple-F “josh” bullshit).
The status-dot is still hours old.
I’m not at all displeased with the service that Dreamhost has provided us over at (currently down neurobashing.com).
I’m not at all happy about the abject lack of current information/communication being conveyed by obviously-unaffected websites such as this blog and the status-dot website.
September 12th, 2005 at 9:41 pm
I have a dedicated server and it’s still down. I also can’t get to the web panel. Presumably, you guys know which customers are still affected–an email would be nice. When I sent a note to the sos address, I got an auto-response.
September 12th, 2005 at 9:42 pm
And shazam! I complain, and I am heard! It’s up now. Thanks!
September 12th, 2005 at 9:43 pm
Did any sites get hacked? A client reported seeing something like ’spy kids own you’ or something along those lines….
Thanks, Sam
September 12th, 2005 at 9:49 pm
Ever hear of backup power?
September 12th, 2005 at 9:52 pm
I’d like the JMK treatment please.
September 12th, 2005 at 9:52 pm
“Derik: I’ve read all of Josh’s comment posts. So, I grant that the comments section has been updated… but not the original blog post (quick-and-dirty, up-front update on the post will save a lot of customers a lot of time with the scrolling and the apple-F “josh” bullshit).” - Gary
I completely agree - it would be wiser - and easier for customers… these updates should also go on the emergency page… I hope your site(s) get(s) up soon, Gary. This could have been executed better in some areas, but I’m sure DH is working hard to get everything back.
September 12th, 2005 at 9:53 pm
My email is working now, but not too well. Everytime I try to do something, it takes me to the login page again. Is this happening to anyone else?
September 12th, 2005 at 10:01 pm
My sites are back up and webmail appears to be working as well. Everything’s a little slow, but it beats error messages. I don’t picture DH as having a ton of people working there, so cut these guys a little slack, if they were blogging all the time with updates they wouldn’t be getting things fixed.
September 12th, 2005 at 10:02 pm
10pm California time, and both of my sites are still down.
http://www.cancer-news-watch.com and http://www.missionremission.com
Any news coming out of DreamHost HQ? Even a couple of words would be nice.
September 12th, 2005 at 10:02 pm
10:00 PM PST and my sites are STILL down. Am I REALLY that lucky? At least my email is up so people can bitch me out for the sites being down.
September 12th, 2005 at 10:06 pm
Derik et al (esp. DH staff): I have no doubt that DH is working as hard as they can to get things back to normal. It’s really the only thing they can do to keep their customers from calling for their heads on spikes.
Not their fault? Keep us updated during idle time waiting for upstream providers to fix *their* shit on the live sites they have available (here and status-dot).
Maybe their fault? Be straight with the customers.
Websites being down isn’t the end of the world; but the service provider is beholden to its customers. I’m not saying they need to bend over backward to make us happy (the provider’s performance is subject to customer scrutiny and judgment); but it is their responsibility to keep us informed… if they wish to keep our business.
Working hard to fix a problem is one thing… but nobody will ever realize the effort put forth if there’s no communication.
September 12th, 2005 at 10:06 pm
aha :)
Everything is up :)
DH is fast nice …. From a major power outtage everything got restored quickly and nothing lost I guess !
Im wondering if any servers got wrecked and why didn’t the 4 power ups didn’t power up :s I thought it should power up in a case like this :)
September 12th, 2005 at 10:08 pm
yeah, my site is still down as well (http://www.semidigital.com). is there any news that all of this will be back up by midnight? if not, when? you’ve mentioned “in the next hour,” several times… now when?
i think we’re all losing a bit of patience.
thanks.
September 12th, 2005 at 10:13 pm
It’s too funny that the people with attitude posting here are mostly those running blogs :) I have a business site with 11 client web sites under mine and I just now got in here to read. I’ve been posting emails w/status updates to my clients all evening instead of bitching about it. We don’t know what happened, nor can any of us do much about it. Patience is a virtue.
September 12th, 2005 at 10:14 pm
My sql is still down - which means my blog, by far the most popular part of my website, is dead. Oy.
I know y’all are doing the best you can, but let me add my voice to all the other folks suggesting that if something like this ever happens again, you should make sure to be providing regular updates. Even someone posting once an hour to say “nothing new to report, we’re still working on getting stuff back up, check for a new update an hour from now” - just that and not one word more - would be very reassuring, compared to what you’re doing now.
Good luck getting our sites back up! :-)
September 12th, 2005 at 10:19 pm
MB, http://www.semidigital.com/ works for me.
September 12th, 2005 at 10:21 pm
Not complaining, exactly, but curious: my site was down, then came back, and now appears to be down again (although this time my e-mail still works). Just wondering why.
September 12th, 2005 at 10:22 pm
beejereeno: You’ve been posting emails w/ status updates to your clients all evening. You’re doing it right and being–IMHO–a good service provider. That’s the point.
September 12th, 2005 at 10:23 pm
scott,
yep, it just went live. :)
hooray! thanks DH.
September 12th, 2005 at 10:24 pm
I have a few personal sites up and a few professional ones as well. As much as it’s frustrating that they all went down, I know the deal and you plan for the worst, hope for the best and when all is said and done sometimes things go awry.
I have no doubt DH staff is doing all they can. And I want to echo another posters comments. If you’re SNAFUed professionally by this, call and e-mail clients. In my case simply saying “The Los Anegles blackout knocked us out…” was more than enough to handle it.
As for blogs. Damn. This proves what most of us know. I read blogs as well, but good lord are bloggers collectively a whiny bunch. I’m personally sorry the world will not be able to read about your cats latest antics and such, but enough already. In the great scheme of things events like this are par for the course.
Bloggers, please. Learn to cope and deal. Not to bitch and moan.
September 12th, 2005 at 10:25 pm
Have had email since 5-ish CDT and have been able to see my files for over an hour, but FINALLY apache has been restarted. Thank you, DH.
(My husband works for…another hosting company…and at least explained everything that I didn’t quite understand about what COULD be wrong. That helped, too.)
September 12th, 2005 at 10:28 pm
Just an “update” folks…
We’re still working on machines, and if you’re still having site problems, we know! We’re doing our best to get everything sorted out as fast as possible.
I’ll make one more (final) post when we think we’ve really fixed absolutely everything. If you’re still having problems then, please contact support through our panel!
(Or if that’s not working, post here! :))
September 12th, 2005 at 10:32 pm
Hmmm, methinks maybe some sites have been moved to different servers? I just tried connecting via ssh to my site and have been warned that my cached hostkey info does not match what the server is presenting me. Anyone else seeing the same?
September 12th, 2005 at 10:40 pm
LA Power Outage Dims Discourse
The LA blackout seems to have taken the site down with it. Supposedly Dreamhost has an emergency backup system, but it seems to have failed completely. Probably FEMA-approved hardware… Update: Details if you care……
September 12th, 2005 at 11:06 pm
Thanks! We are back on the air!
September 12th, 2005 at 11:11 pm
I’ve been on both sides of this sort of failure - as an end-user and as the project manager/engineer responsible for premises equipment serving the fiber connection to the real world for multiple major communications carriers.
After the first failure of the building provided power backup at one of the buildings I’d engineered, most of the tenants, including us, got together to protect ourselves.
With the cooperation of the building management company we built UPS sets for each customer (8 hour minimum), engineered and installed multiple redundant generator sets, AND put in a massive external power connector so we could roll up a trailer mounted 1MW generator to handle ‘essential power’ if all else failed.
The entire building was divided into ‘essential power’ and ‘everything else’ so we could load shed if things got sticky.
We DID survive a four and one half day power outage that took 31 square blocks of the city out.
We had dry ice in some areas to cool overheated electronics, we had hardly any lighting, we had one elevator for emergency use, but everyone kept running.
Had we not had a previous failure we would not have made ourselves ready for ‘the big one.’
This is your opportunity to learn, guys. Look around at what did and did not work after you get everything up again. Fix what didn’t work and get guarantees from the folks backing YOU up.
If you contract with the building for backup power, make sure they can provide it - and include penalties in the contract if they don’t.
Overall? Yeah, you had a failure. It took too long to recover. It won’t happen again, right? ;o)
Learn from this and you’ll be ahead of the game.
Oh, yeah, GOOD JOB! Really! Until the first one like this you really don’t have that gut feeling for what’s needed. You performed well getting it all back together when outside folks let you down. Thanks.
September 12th, 2005 at 11:41 pm
Today’s Downtime
Err.. sorry bout that… It appears that Dreamhost uses servers and internet connections that require electricity.. so, after the (probably now unemployed) Power Company worker shut off half of Los Angeles, Dreamhost no longer had a connection to the…
September 12th, 2005 at 11:55 pm
Ok, I’ll let it go with this one final rant:
“Note: System Status is not guaranteed accurate!…”
I’ve said this before to DH management — the status tab should be deleted, unless it’s going to be *honest*.
To give the disclaimer that, essentially, not all downtime events are going to be counted or reflected in the stats is a total luse, worthy of the current administration in Washington.
Otherwise where I come from it’s called a lie. Please either be honest about the numbers or take them away, ’cause right now they aren’t worth the diskspace they occupy.
/jad
September 12th, 2005 at 11:58 pm
[...] DreamHost Blog Power Outage Update [...]
September 12th, 2005 at 11:58 pm
Just a note, as of right now everything is believed to be good to go! If you’re still having any site or email problems, please contact support (after you’ve verified it’s still broken for sure!)
Thanks for the patience everybody.. more tomorrow!
September 13th, 2005 at 12:40 am
Good lord some of you guys are whiny.
that is all.
September 13th, 2005 at 2:32 am
[...] Check out the CNN article and the DreamHost blog for more information on the power outage. [...]
September 13th, 2005 at 2:37 am
its nice to know i was right and that it wasnt dreamhosts fault but their network providers… :D
September 13th, 2005 at 2:56 am
[...] Lo que le paso a Dreamhost: Okay, this sucked. Literally 10 seconds after Nate posted that last entry, we received an announcement over our loud speakers that our building was being evacuated and we had to leave immediately! [...]
September 13th, 2005 at 3:55 am
[...] Apparently there was a power outage in L.A. that took the site down yesterday. You can get details from it in the DreamHost blog at DreamHost Blog » Power Outage Update. [...]
September 13th, 2005 at 4:08 am
So much for all those 100%’s on the server up time screen…. they’d BETTER reflect this outage.
September 13th, 2005 at 5:49 am
shouldnt there have been a backup servers preferably somewhere in another location to deal with this type of “emergency”?
September 13th, 2005 at 5:52 am
Any ideas what this is?
Site Temporarily Unavailable
We apologize for the inconvenience. Please contact the webmaster/ tech support immediately to have them rectify this.
error id: “bad_httpd_conf”
September 13th, 2005 at 8:08 am
Where is the mail that was lost in the interim?
Can you guys get it together to post a single, comprehensive update? I’m getting tired of these “Update coming soon!!!!” messages.
I just want to know WHERE my mail is that was lost in the downtime.
September 13th, 2005 at 8:49 am
blackout
Much of Los Angeles “went dark” yesterday during the middle of the afternoon. I have to quote this phrase because, well, the sun was out. But for much of the afternoon, power was off for nearly a million people. Luckily,…
September 13th, 2005 at 8:54 am
All the RB fans will be delighted to see it’s back up no doubt.
September 13th, 2005 at 11:09 am
[...] DreamHost Blog » Power Outage Update Technorati Tags: Dreamhost [...]
September 13th, 2005 at 11:54 am
It’s kaputt, not caput.
September 13th, 2005 at 3:47 pm
Nice that your back :D
September 14th, 2005 at 12:55 pm
[...] http://ask.metafilter.com/mefi/23949 http://blog.dreamhost.com/2005/09/12/power-outage-update/ http://status.dreamhost.com [...]
September 14th, 2005 at 2:31 pm
Panicked a bit at home and at work - we probably lost business because email was down too. Some of the other hosts boasted redundancy systems. Think it’s time for that? Not blaming - after all it was out of your control - but making a suggestion for the future.
September 21st, 2005 at 12:40 pm
[...] This is awesome news, because as you may know, I LOVE EVDO! [...]
November 7th, 2005 at 9:34 am
[...] We’re sorry about being offline for part of the day. Our host was affected by today’s power outage in Los Angeles. [...]
December 11th, 2005 at 9:04 pm
Relacore Lipovarin Estrin-D Cortislim Suvaril Trimspa Zantrex-3 Avlimil Enzyte Magna-Rx Ogoplex Altovis for the lowest price on all of these name brand product just visit link above today!
December 12th, 2005 at 11:05 am
New Liquid Capsule Design Assists In Rapid Weight Loss!
Lipo-6 by Nutrex is the first and only fat-burner that utilizes maximum strength liquid capsules for superior absorption and unprecedented results. Never before has a fat-loss formula offered such an advanced method of delivering powerful fat-burning compounds, quickly, effectively and safely. We guarantee, with Lipo-6 Liquid Capsules, you will experience far greater fat-burning results than with any other fat-burning product on the market today! http://www.a1anutrition.com/cgi-bin/item/LIPO6
December 14th, 2005 at 2:41 pm
If you looking for the right ONE you should get your free profile from eharmony today.
http://free-blog-site.com/eharmony/
December 15th, 2005 at 9:47 am
If your looking for the right person to date you have to go to ehormony for your free profile today!
http://free-blog-site.com/ehormony/
January 1st, 2006 at 2:27 am
[...] Down Time Two major incidents happened in 2005. The Power Outage in entire Los Angeles in September and a massive DDoS attack in December. Major crisis like this is impossible to avoid, but when it does happen, it’s nice to have a web host that is quite informative about what’s going on. [...]
January 3rd, 2006 at 7:06 pm
[...] Our power costs even more than yours because it’s “good” power. That is, it’s backed up by huge diesel generators and UPS batteries in the supremely unlike event of say, a Los Angeles power outage. People estimate that Google must pay about $50,000,000 a month for power for their 100,000 servers. That sounds about right to us! [...]
January 26th, 2006 at 5:17 am
You know guys, you’re all complaining about the downtime, a lack of suitable redundant power etc, but you’re forgetting that Dreamhost is one of, if not the most reliable of all the cheap web hosts out there. If you want redundant UPSes, more bandwidth, co-located racks etc, then why not pay for a more premium service?
July 29th, 2006 at 3:02 pm
[...] DreamHost was affected by a similar incident last year, when the generators didn’t work. [...]
August 1st, 2006 at 12:29 pm
[...] Ironically, all the recent disasters stem somewhat from us attempting to take some proactive steps to head off any sort of future power outages like the kind we experienced last year. [...]
October 17th, 2006 at 10:38 pm
[...] Transparency. When a server crashes or a catastrophic power outage disrupts service, they’re quick to post to their status blog. The only time I heard from any of my previous hosting providers was when I owed them money. Dreamhost isn’t afraid to flaunt it’s faults, it embraces them. [...]
January 8th, 2009 at 11:17 pm
hi
st4hrwqv5m8xvnmf
good luck