Well, things could be worse.
We’ve pretty much got our whole network under control now.. the ongoing problem mentioned last post was finally figured out by Cisco support. It turns out it was a bug undocumented feature in IOS dealing with how they learn MAC addresses.
There was also another network problem we got resolved yesterday that was causing general slowness on web and mail servers. It’s complicated (i.e. I don’t understand it exactly myself), but in the end we took a distribution switch out of the network and that fixed it.
We still have one open ticket with Cisco for our core routers having some HSRP problems. It doesn’t seem like that’s having any real effect on our network, but we want it fixed!
We are also installing two new Ciscos to offload the BGP duties from the core routers so they’ll just have to handle switching. This set-up should be able to handle about 300% more traffic than our entire network now pushes at peak times!
Thanks to these network problems being resolved, we’ve also begun re-deploying in Alchemy, who at least didn’t have the second power outage.
We’re also still in the process of getting real UPS power on our network cabinet, plus our internal databases and a few internal servers. Basically, everything that keeps all the customer mail, web, database, and file servers from coming right back up quickly should there ever be another outage.
So, um.. that’s how it stands now! We hope this will all soon be nothing more than a long bad dream (that was real).
August 11th, 2006 at 1:34 pm
ummm… huh?
i think i understand.
thanks for the info.
August 11th, 2006 at 2:49 pm
LOL, I’m totaly gonna hot link that simpsons image the next time im working on a site!
And technicaly I could argue that I am paying for the bandwidth. I think I’m only up to 1% of my limit so far this month…
August 11th, 2006 at 2:49 pm
Thanks for the update!
August 11th, 2006 at 3:19 pm
Glad to hear that things are on the road to recovery. Hopefully the last few issues will be easily taken care of and stability will regain the solidity it previously had. Keep up the hard work.
August 11th, 2006 at 6:35 pm
I’m wondering, will Dreamhost stay in Los Angeles for long? The power is expensive and unreliable. There’s a whole write up in a recent Fortune magazine about data centers moving to rural areas that are served by hydroelectric power.
August 11th, 2006 at 6:45 pm
Nathan, I think it might have something to do with the cheap bandwidth in california, because of high competition.
August 12th, 2006 at 3:34 am
It looks like things are improving, I’m no longer getting constant downtime notifications from the uptime checking service I use (To be fair these weren’t actual downtimes most of the time probably just timeouts affecting the service’s checker, but still a timeout for a particular visitor is pretty much as bad as a “down” website)
Keep up the good work DH, I’d hate to have to move.
August 12th, 2006 at 5:36 am
I’m glad to hear that things are (almost) back to normal. Thanks for the communication during this whole process.
August 12th, 2006 at 5:58 am
hehehe :-D
Wheres the Dreamhost Blog Post Decoder
August 12th, 2006 at 9:10 am
I think the choice of Los Angeles for the datacenter has more to do with the weather, the scene, and perhaps the beach than either cheap bandwidth or electrical power.
I wouldn’t have it any other way, myself…
/j (in Laguna Beach)
August 12th, 2006 at 10:31 am
Kewlness.
Glad to hear you have it under control. When I saw that our sites were slow 2 days ago, I was ready to pull out what few hairs I have left on my head! D’oh!
August 12th, 2006 at 2:28 pm
If DreamHost moved their data center somewhere else, they’d also have to get people to work at the somewhere else. Two hour drives to the data center or hiring the only three sysadmins in the town would be a Bad Thing.
August 12th, 2006 at 4:09 pm
Interesting insights in what goes on at your end of the cables. Seems like you are doing a great effort to keep things smooth in the future.
Maybe you should pick some environment/global warming effort for the next set of charities… see if that does something about the blackouts… :)
August 13th, 2006 at 2:28 am
Thanks for the update, must be good to see the light at the end of the tunnel.
August 13th, 2006 at 9:10 am
Well, my site has had no hits at all the last 17 hours. And Dreamhoststatus reports no errors?
August 13th, 2006 at 9:18 am
How about faster support for sites that are down? My ticket’s been open over 5 hours now and no response. Some of us rely on our sites for income so these long downtimes are NOT appreciated. Good luck with your stability.
August 13th, 2006 at 11:25 am
PoRtCuLLiS, if you rely on your income from a $7.95/month hosting account then you get exactly what you deserve…
August 13th, 2006 at 12:55 pm
Ross,
I doubt he relies on it for his entire income.
August 13th, 2006 at 3:07 pm
Hey! Thanks for the awesome disaster photo collection. And yeah, it sucked being down, but I think we all know….stuff happens. I’ve been with you folks for almost five years now, and overall the service has been nearly flawless…..thanks so much.
s.
August 13th, 2006 at 8:47 pm
July was hell, Dreamhost offered an explanation which succeeded in at least restoring some confidence. There was a lot of bad luck inflicted on DH and after 3-4 very, very happy years of being a DH customer, I can easily forgive a bad patch.
There is *still* the issue, however, of DH saying their stuff is fixed when it clearly is not.
At the time of writing this, the problems with email were first mentioned on dreamhoststatus.com “6 days, 13 hours ago”. “4 days, 8 hours ago” there is a second message to say the problem is almost fixed (it wasn’t). “3 days, 4 hours ago” there is another message to say it should be 100% OK now. My email and webmail is now in its 7th day of not working properly.
I ask for one thing and one thing only: Please do NOT take 36 hours to respond to a support request with the reply, “we’ve looked at your account and fixed the issue”, when you have NOT fixed the issue. I’d rather you told me you have no idea what the problem is than to lie. I sent a reply to this message 26 hours ago saying the problem persists and I am still waiting for a response.
I worked in IT support way back when and I know precisely how difficult it can be to trace intermittent errors – you DO have my sympathy. But it is fundamental that an IT Support desk doesn’t lose credibility amongst those it is there to support.
Best of luck, DH, we’re counting on you.
August 14th, 2006 at 12:48 am
Looking ahead to the next incident:
What happens when an earthquake rocks the west coast of USA? Can it hit your data centre? Is this the cue for us to lose everything in your datacentre and have websites down for the next few months?
August 14th, 2006 at 3:18 am
Ross: Regardless of how cheap it is, I don’t think expecting a sensible amount of uptime is asking a lot. Reliability took a nosedive lately, I accept it was just a bad patch. But I still think “site down” support tickets should be given more priority and not take 5-6 hours to fix (even if it is Sunday). Good luck DH, I love you really :P
August 14th, 2006 at 8:02 am
ah yes, a bad dream. though sleeping with dreamhost is fun :)
with a name like josh jones you should be an mc. josh jooooones.
August 14th, 2006 at 8:29 am
still lots of timeouts, slow servers, errors in webmail,… i just renewed my subscription hope i won’t regret it.
August 14th, 2006 at 9:42 am
IMAP folders are horribly slow, webmail doesn’t even work at all – server drops the IMAP connection before the page can load. And all of this as DreamHost says everything is peachy. I think 5 weeks of patience is a lot to ask of us. It’s time to get your shit together.
August 14th, 2006 at 9:47 am
Unfortunately I can not share and understand the optimism. Just like Hyde I am still experiencing a very slow (often not reacting) dreamhost server (overland.dreamhost.com).
August 14th, 2006 at 1:14 pm
Um, yeah. There were about 5-6 hours of system-wide outages today and about 3 more that affected at least the server I’m on yesterday. Great…
Prove to us that you’re serious and put your money where your blog is. Offer refunds to any current DH user for the prorated amount of their pre-paid time. If you think your product still stands up, let the market really decide. I think it’s either that or your pre-paid users soak the Internet with “Avoid Dreamhost” reviews until they feel like they took their money’s worth back out by steering away potential customers.
Those of us long past the 97 days are tired of being DH apologists. All people care about (me included) is that the sites work. I’m willing to give the benefit of the doubt as I know how technology is, but this is beyond ridiculous. We’ve been patient and rolled with this, but this has gone too far.
Oh, and dreamhost status misses about 9 out of 10 outages so we have no clue what is going on most of the time. Thanks.
August 14th, 2006 at 1:23 pm
I agree with the Jon’s comments about the situation — sometimes lately it’s looked like Support is using a Dial-A-Problem wheel and just telling me whatever comes up first.
I’m three weeks shy of my first two years with Dreamhost. Back then I was satisfied enough with it to pay for two years up-front. Now I’m a few days worth of intermittent service away from switching hosts.
August 14th, 2006 at 3:44 pm
Uh.. I put in a support ticket the other day to ask where I can find my customers’ email account passwords. They got back to me within 30 minutes! Never seen support that fast from Dreamhost. Usually it takes a LOT longer but I was glad to have it.
My only gripe is that webmail is HORRIBLY slow. God it takes forever to load.. and we are on a 3mb cable connection. I would hate to think how long it would take with a dial-up connection.
August 15th, 2006 at 12:20 am
I do not ask for a refund, I do not ask for getting more bandwith every week or more webspace, I gladly give that up in exchange for a good working hosting. My current support tickets are (thre are two) open for more than 12 hours now. On the dreamhoststatus.com site I do not see any mention of the imap problems (One of my tickets, and apperently something that was noticed here as well) or about timeouts. The server I am on is wasabi and it’s just not working.
I wonder if DH reads the comments on their blog…
don’t give me a refund. Just make it do what I pay you for.
August 15th, 2006 at 8:36 am
Webmail is crap. That’s just how it is, unless you have like under 100 messages. But taht’s SquirrelMail’s fault.
August 15th, 2006 at 9:59 am
@hyde. I suggest you start looking for a different provider then. With my important sites and email accounts I am at the same provider since many many years. Problems and downtime extremely rare and that support answers quickly is the normal situation. It costs a bit more of course (I am paying $17.95/month). I came to DH for the cheap bandwidth and Ruby On Rails support.
August 15th, 2006 at 8:32 pm
Verified mail outage for almost an hour. Anyone else having trouble?
August 16th, 2006 at 3:59 am
Thanks for giving all the details and I hope things get better powerwise.
I know what it is like when things break. I had a sandwich toaster fail last week and I have not eaten for days.
August 16th, 2006 at 8:58 am
It was fixed
Now it’s worse
August 16th, 2006 at 10:38 am
Things are not fixed. Mail is down AGAIN. Has been for over an hour. I think you should stop advertising that email is included in your hosting plans.
August 16th, 2006 at 1:44 pm
I set up an automatic monitor on one of my sites (on babyruth) – set up on Monday, now it’s Wednesday and there have been five outages reported with a downtime of 11% (polling every half hour).
I’m paying $16/month for that.
I have seen other hosting providers that show the (independently collected) statistics for all of their servers. I would be interested to see how my stats compare with other Dreamhost users.
August 16th, 2006 at 2:49 pm
Lol @ “Undocumented feature…” thats why I like dreamhost… it is the only host that I know of that has an appreciation for humor!
August 23rd, 2006 at 4:18 pm
You were running BGP on your core routers?
Um.
That’s retarded.
Kind of basic knowledge too, for anyone with a few years of real network experience.
Do you employ anyone that understands networking?
August 24th, 2006 at 7:49 am
I don’t know what you consider to be “under control,” but I signed up a few days ago, and my site has been down more than up ever since. I was basically told by a support rep that although my site was indeed down, it was only because of a normal spike in traffic, so I guess that doesn’t count.
I’ve had multiple “verified outages.” Support has nothing useful to offer. It’s probably time to address this and acknowledge that the problems aren’t solved. I admire the effort to be straightforward, and I’m sure you were hopeful that things were improving, but they’re not.
August 24th, 2006 at 9:53 am
Never mind that email is horribly slow and our websites are sluggish to the point of tears. There is a humorous cartoon and a wacky attitude by the staff. That’s what’s important in a web host.
August 24th, 2006 at 9:54 am
I don’t understand how the last person doesn’t believe that the problems are “under control”?? My site has only been down 11 times since they fixed everything (and it’s down again now). Now most people would think that’s a lot, but it’s clear to me that they’re doing something right because it was down 26 times last month. Yay Dreamhost!! … friggin idiots …
By the way, I like the cartoon. I think it’s very appropriate. I’ve always compared Dreamhost to Homer — likable, but thoroughly incompetent.
August 24th, 2006 at 11:23 am
Humorous anecdote of the day:
Client e-mails me and says, “Wow, the site is running a lot faster and has worked really well the last few days. What did you do?”
I reply sheepishly, “I moved the site from DreamHost to somewhere else…”
DreamHost is now pretty much a large hard drive to me. I’m stuck with it for several more months so I might as well use it for something.
I really don’t know where people get that all these updates (which still miss about 9 out of 10 outages if my experience is consistent with others) are informative and straightforward. They are well-spun and self-deprecating (and perhaps even purposefully distracting – “Ooo, ooo, forget that silly downtime stuff, listen to this funny guy calling us c0ck$ucker$”), but no matter how you spin it, downtimes have been terrible and DH is getting panned all over the Internet for it.
Simple rule of thumb, guys: Downtime is not funny. Spend your time fixing the damn network and forget the jokes. If I want to laugh, I already have cable TV.
August 26th, 2006 at 6:31 am
Fair play – 100% uptime for the last week. That’s more like it.
August 29th, 2006 at 5:50 pm
I’m going to echo something said above. DH has serious problems. My sites appear to be back to normal now. Maybe just a little slower, but they’re working. My faith in DH, however, is not back to normal.
It’s time to stop being wacky and zany. I want less humor and more concern. Period.
When my sites are performing poorly and I come here seeking serious answers but instead find jokes, you’re damn right I feel offended – both personally and professionally.
September 8th, 2006 at 4:34 pm
My site’s been down all evening without even being given a reason so far. Awful. Just awful.