DreamHost Blog » Reconstruction Efforts

Reconstruction Efforts

August 11, 2006 on 11:01 am | In Foobars, Hardware, Insider View, Updates by Josh Jones |

Hey, Homer came in with a very competitive bid.

Well, things could be worse.

We’ve pretty much got our whole network under control now.. the ongoing problem mentioned last post was finally figured out by Cisco support. It turns out it was a ~~bug~~ undocumented feature in IOS dealing with how they learn MAC addresses.

There was also another network problem we got resolved yesterday that was causing general slowness on web and mail servers. It’s complicated (i.e. I don’t understand it exactly myself), but in the end we took a distribution switch out of the network and that fixed it.

We still have one open ticket with Cisco for our core routers having some HSRP problems. It doesn’t seem like that’s having any real effect on our network, but we want it fixed!

We are also installing two new Ciscos to offload the BGP duties from the core routers so they’ll just have to handle switching. This set-up should be able to handle about 300% more traffic than our entire network now pushes at peak times!

Thanks to these network problems being resolved, we’ve also begun re-deploying in Alchemy, who at least didn’t have the second power outage.

We’re also still in the process of getting real UPS power on our network cabinet, plus our internal databases and a few internal servers. Basically, everything that keeps all the customer mail, web, database, and file servers from coming right back up quickly should there ever be another outage.

Less like a disaster, more like a field of wildflowers.

So, um.. that’s how it stands now! We hope this will all soon be nothing more than a long bad dream (that was real).

46 Comments

1

ummm… huh?

i think i understand.

thanks for the info.

Comment by vince — August 11, 2006 #
2

LOL, I’m totaly gonna hot link that simpsons image the next time im working on a site!

And technicaly I could argue that I am paying for the bandwidth. I think I’m only up to 1% of my limit so far this month…

Comment by Nathan Friedly — August 11, 2006 #
3

Thanks for the update!

Comment by SR — August 11, 2006 #
4

Glad to hear that things are on the road to recovery. Hopefully the last few issues will be easily taken care of and stability will regain the solidity it previously had. Keep up the hard work.

Comment by Ian Clifton — August 11, 2006 #
5

I’m wondering, will Dreamhost stay in Los Angeles for long? The power is expensive and unreliable. There’s a whole write up in a recent Fortune magazine about data centers moving to rural areas that are served by hydroelectric power.

Comment by Nathan — August 11, 2006 #
6

Nathan, I think it might have something to do with the cheap bandwidth in california, because of high competition.

Comment by Sam — August 11, 2006 #
7

It looks like things are improving, I’m no longer getting constant downtime notifications from the uptime checking service I use (To be fair these weren’t actual downtimes most of the time probably just timeouts affecting the service’s checker, but still a timeout for a particular visitor is pretty much as bad as a “down” website)

Keep up the good work DH, I’d hate to have to move.

Comment by Hentaikid — August 12, 2006 #
8

I’m glad to hear that things are (almost) back to normal. Thanks for the communication during this whole process.

Comment by Michael — August 12, 2006 #
9

hehehe :-D

Wheres the Dreamhost Blog Post Decoder

Comment by Rajesh Ahuja — August 12, 2006 #
10

I think the choice of Los Angeles for the datacenter has more to do with the weather, the scene, and perhaps the beach than either cheap bandwidth or electrical power.

I wouldn’t have it any other way, myself…

/j (in Laguna Beach)

Comment by Jad — August 12, 2006 #
11

Kewlness.

Glad to hear you have it under control. When I saw that our sites were slow 2 days ago, I was ready to pull out what few hairs I have left on my head! D’oh!

Comment by Craig Cantin — August 12, 2006 #
12

If DreamHost moved their data center somewhere else, they’d also have to get people to work at the somewhere else. Two hour drives to the data center or hiring the only three sysadmins in the town would be a Bad Thing.

Comment by Matt Nordhoff — August 12, 2006 #
13

Interesting insights in what goes on at your end of the cables. Seems like you are doing a great effort to keep things smooth in the future.

Maybe you should pick some environment/global warming effort for the next set of charities… see if that does something about the blackouts… :)

Comment by Andreas — August 12, 2006 #
14

Thanks for the update, must be good to see the light at the end of the tunnel.

Comment by David — August 13, 2006 #
15

Well, my site has had no hits at all the last 17 hours. And Dreamhoststatus reports no errors?

Comment by Oyvind — August 13, 2006 #
16

How about faster support for sites that are down? My ticket’s been open over 5 hours now and no response. Some of us rely on our sites for income so these long downtimes are NOT appreciated. Good luck with your stability.

Comment by PoRtCuLLiS — August 13, 2006 #
17

PoRtCuLLiS, if you rely on your income from a $7.95/month hosting account then you get exactly what you deserve…

Comment by Ross — August 13, 2006 #
18

Ross,

I doubt he relies on it for his entire income.

Comment by bwd — August 13, 2006 #
19

Hey! Thanks for the awesome disaster photo collection. And yeah, it sucked being down, but I think we all know….stuff happens. I’ve been with you folks for almost five years now, and overall the service has been nearly flawless…..thanks so much.

s.

Comment by Susana Gallardo — August 13, 2006 #
20

July was hell, Dreamhost offered an explanation which succeeded in at least restoring some confidence. There was a lot of bad luck inflicted on DH and after 3-4 very, very happy years of being a DH customer, I can easily forgive a bad patch.

There is *still* the issue, however, of DH saying their stuff is fixed when it clearly is not.

At the time of writing this, the problems with email were first mentioned on dreamhoststatus.com “6 days, 13 hours ago”. “4 days, 8 hours ago” there is a second message to say the problem is almost fixed (it wasn’t). “3 days, 4 hours ago” there is another message to say it should be 100% OK now. My email and webmail is now in its 7th day of not working properly.

I ask for one thing and one thing only: Please do NOT take 36 hours to respond to a support request with the reply, “we’ve looked at your account and fixed the issue”, when you have NOT fixed the issue. I’d rather you told me you have no idea what the problem is than to lie. I sent a reply to this message 26 hours ago saying the problem persists and I am still waiting for a response.

I worked in IT support way back when and I know precisely how difficult it can be to trace intermittent errors - you DO have my sympathy. But it is fundamental that an IT Support desk doesn’t lose credibility amongst those it is there to support.

Best of luck, DH, we’re counting on you.

Comment by Jon — August 13, 2006 #
21

Looking ahead to the next incident:
What happens when an earthquake rocks the west coast of USA? Can it hit your data centre? Is this the cue for us to lose everything in your datacentre and have websites down for the next few months?

Comment by Peter — August 14, 2006 #
22

Ross: Regardless of how cheap it is, I don’t think expecting a sensible amount of uptime is asking a lot. Reliability took a nosedive lately, I accept it was just a bad patch. But I still think “site down” support tickets should be given more priority and not take 5-6 hours to fix (even if it is Sunday). Good luck DH, I love you really :P

Comment by PoRtCuLLiS — August 14, 2006 #
23

ah yes, a bad dream. though sleeping with dreamhost is fun :)
with a name like josh jones you should be an mc. josh jooooones.

Comment by E — August 14, 2006 #
24

still lots of timeouts, slow servers, errors in webmail,… i just renewed my subscription hope i won’t regret it.

Comment by Hyde — August 14, 2006 #
25

IMAP folders are horribly slow, webmail doesn’t even work at all - server drops the IMAP connection before the page can load. And all of this as DreamHost says everything is peachy. I think 5 weeks of patience is a lot to ask of us. It’s time to get your shit together.

Comment by steven — August 14, 2006 #
26

Unfortunately I can not share and understand the optimism. Just like Hyde I am still experiencing a very slow (often not reacting) dreamhost server (overland.dreamhost.com).

Comment by Harald Walker — August 14, 2006 #
27

Um, yeah. There were about 5-6 hours of system-wide outages today and about 3 more that affected at least the server I’m on yesterday. Great…

Prove to us that you’re serious and put your money where your blog is. Offer refunds to any current DH user for the prorated amount of their pre-paid time. If you think your product still stands up, let the market really decide. I think it’s either that or your pre-paid users soak the Internet with “Avoid Dreamhost” reviews until they feel like they took their money’s worth back out by steering away potential customers.

Those of us long past the 97 days are tired of being DH apologists. All people care about (me included) is that the sites work. I’m willing to give the benefit of the doubt as I know how technology is, but this is beyond ridiculous. We’ve been patient and rolled with this, but this has gone too far.

Oh, and dreamhost status misses about 9 out of 10 outages so we have no clue what is going on most of the time. Thanks.

Comment by Agent Smith — August 14, 2006 #
28

I agree with the Jon’s comments about the situation — sometimes lately it’s looked like Support is using a Dial-A-Problem wheel and just telling me whatever comes up first.

I’m three weeks shy of my first two years with Dreamhost. Back then I was satisfied enough with it to pay for two years up-front. Now I’m a few days worth of intermittent service away from switching hosts.

Comment by frank — August 14, 2006 #
29

Uh.. I put in a support ticket the other day to ask where I can find my customers’ email account passwords. They got back to me within 30 minutes! Never seen support that fast from Dreamhost. Usually it takes a LOT longer but I was glad to have it.

My only gripe is that webmail is HORRIBLY slow. God it takes forever to load.. and we are on a 3mb cable connection. I would hate to think how long it would take with a dial-up connection.

Comment by Chris — August 14, 2006 #
30

I do not ask for a refund, I do not ask for getting more bandwith every week or more webspace, I gladly give that up in exchange for a good working hosting. My current support tickets are (thre are two) open for more than 12 hours now. On the dreamhoststatus.com site I do not see any mention of the imap problems (One of my tickets, and apperently something that was noticed here as well) or about timeouts. The server I am on is wasabi and it’s just not working.
I wonder if DH reads the comments on their blog…
don’t give me a refund. Just make it do what I pay you for.

Comment by hyde — August 15, 2006 #
31

Webmail is crap. That’s just how it is, unless you have like under 100 messages. But taht’s SquirrelMail’s fault.

Comment by Jon H — August 15, 2006 #
32

@hyde. I suggest you start looking for a different provider then. With my important sites and email accounts I am at the same provider since many many years. Problems and downtime extremely rare and that support answers quickly is the normal situation. It costs a bit more of course (I am paying $17.95/month). I came to DH for the cheap bandwidth and Ruby On Rails support.

Comment by Harald Walker — August 15, 2006 #
33

Verified mail outage for almost an hour. Anyone else having trouble?

Comment by ste — August 15, 2006 #
34

Thanks for giving all the details and I hope things get better powerwise.

I know what it is like when things break. I had a sandwich toaster fail last week and I have not eaten for days.

Comment by Norm — August 16, 2006 #
35

It was fixed

Now it’s worse

Comment by OiOi — August 16, 2006 #
36

Things are not fixed. Mail is down AGAIN. Has been for over an hour. I think you should stop advertising that email is included in your hosting plans.

Comment by LB — August 16, 2006 #
37

I set up an automatic monitor on one of my sites (on babyruth) - set up on Monday, now it’s Wednesday and there have been five outages reported with a downtime of 11% (polling every half hour).

I’m paying $16/month for that.

I have seen other hosting providers that show the (independently collected) statistics for all of their servers. I would be interested to see how my stats compare with other Dreamhost users.

Comment by Baz — August 16, 2006 #
38

Lol @ “Undocumented feature…” thats why I like dreamhost… it is the only host that I know of that has an appreciation for humor!

Comment by Price Crapper — August 16, 2006 #
39

You were running BGP on your core routers?

Um.

That’s retarded.

Kind of basic knowledge too, for anyone with a few years of real network experience.

Do you employ anyone that understands networking?

Comment by DH customer — August 23, 2006 #
40

I don’t know what you consider to be “under control,” but I signed up a few days ago, and my site has been down more than up ever since. I was basically told by a support rep that although my site was indeed down, it was only because of a normal spike in traffic, so I guess that doesn’t count.

I’ve had multiple “verified outages.” Support has nothing useful to offer. It’s probably time to address this and acknowledge that the problems aren’t solved. I admire the effort to be straightforward, and I’m sure you were hopeful that things were improving, but they’re not.

Comment by Linda — August 24, 2006 #
41

Never mind that email is horribly slow and our websites are sluggish to the point of tears. There is a humorous cartoon and a wacky attitude by the staff. That’s what’s important in a web host.

Comment by DH customer — August 24, 2006 #
42

I don’t understand how the last person doesn’t believe that the problems are “under control”?? My site has only been down 11 times since they fixed everything (and it’s down again now). Now most people would think that’s a lot, but it’s clear to me that they’re doing something right because it was down 26 times last month. Yay Dreamhost!! … friggin idiots …

By the way, I like the cartoon. I think it’s very appropriate. I’ve always compared Dreamhost to Homer — likable, but thoroughly incompetent.

Comment by Mike — August 24, 2006 #
43

Humorous anecdote of the day:

Client e-mails me and says, “Wow, the site is running a lot faster and has worked really well the last few days. What did you do?”

I reply sheepishly, “I moved the site from DreamHost to somewhere else…”

DreamHost is now pretty much a large hard drive to me. I’m stuck with it for several more months so I might as well use it for something.

I really don’t know where people get that all these updates (which still miss about 9 out of 10 outages if my experience is consistent with others) are informative and straightforward. They are well-spun and self-deprecating (and perhaps even purposefully distracting - “Ooo, ooo, forget that silly downtime stuff, listen to this funny guy calling us c0ck$ucker$”), but no matter how you spin it, downtimes have been terrible and DH is getting panned all over the Internet for it.

Simple rule of thumb, guys: Downtime is not funny. Spend your time fixing the damn network and forget the jokes. If I want to laugh, I already have cable TV.

Comment by Agent Smith — August 24, 2006 #
44

Fair play - 100% uptime for the last week. That’s more like it.

Comment by Baz — August 26, 2006 #
45

I’m going to echo something said above. DH has serious problems. My sites appear to be back to normal now. Maybe just a little slower, but they’re working. My faith in DH, however, is not back to normal.

It’s time to stop being wacky and zany. I want less humor and more concern. Period.

When my sites are performing poorly and I come here seeking serious answers but instead find jokes, you’re damn right I feel offended - both personally and professionally.

Comment by Rob — August 29, 2006 #
46

My site’s been down all evening without even being given a reason so far. Awful. Just awful.

Comment by PoRtCuLLiS — September 8, 2006 #

Sorry, the comment form is closed at this time.

Reconstruction Efforts

46 Comments

Categories:

Archives:

Meta:

Reconstruction Efforts

46 Comments

Categories:

Archives:

Other Sites

Meta: