Anatomy of a Disaster, Part 2
September 19, 2006 on 3:00 pm | In Foobars, Insider View by Dallas Kashuba | 112 CommentsFor the past several weeks many of you have been faced with slow or unusable websites and email. The original cause of that series of issues was detailed in Josh’s great Anatomy Of An Ongoing Disaster post. The network issue we were left with once the power outage problems were mostly resolved ended up being an especially nasty one. We were essentially caught with our pants down at just the wrong time and we’ve been taking our lumps for it.
The evidence we were seeing all pointed to one of the two routers as the primary troublemaker so we focused on that one. Configurations were changed with some improvement but without resolving the main issue. Ultimately, 6 separate Cisco support engineers and a Cisco Certified Internet Engineer were all unable to determine the cause of the errors we were seeing on our routers. That, along with the recent power outages, eventually led everyone to believe there was a hardware fault within the router somewhere. That started our process of replacing and/or upgrading every component. Once that was done and the main problem was still there we were able to finally pinpoint the point of network congestion and resolve it, and that’s where we are now.
The problem ending up being the connection between the two routers. Our network was set up so one router was primarily responsible for some of our servers, and the other router was primarily responsible for the rest of the servers. Both routers are connected to outside network connections and they share those roles providing wide-area network redundancy, but the inside of our network (our LAN) relied on both routers working together and passing bits back and forth. Some of you did not experience the problems because all of the servers your service relies on were on the same core router and were not bottlenecked by that inter-router link. Once one of the routers was fully upgraded we were able to move all traffic to that single router thereby removing the bottleneck and restoring service completely for everyone.
Our routers were not redundant and that hurt us. If our routers had been redundant we could have much more easily moved all traffic to one router or the other just to eliminate some variables. Having that option would have saved us a lot of time and you a lot of painfully slow service.
Establishing Power Redundancy
In searching for this solution we wasted a lot of time uncertain about the integrity of our equipment. Whenever a piece of electrical equipment suddenly loses power there is always a chance of some component failing and when you’re dealing with a device as complex as a router that’s a lot of components to worry about! If our data center’s UPS and generator setup had worked properly and the routers had not lost power, we could have instead focused on the new evidence at hand, confident that nothing else had changed. Knowing that, and knowing the track record of our data center, we are already in the process of adding an additional layer of power redundancy for our most critical (and expensive to replace!) equipment. The DC powered equipment housed in our data center is backed by a secondary UPS system and did not lose power throughout the recent power fluctuations. To take advantage of that ourselves, we are converting to DC power at the core of our network. We have the power supplies sitting and waiting to be installed and we’re currently waiting for the power to be wired into the racks we need it in.
We are also expanding our space in our Alchemy Communications Data Center. Alchemy has set up their own UPS backed power feed and were not hit as hard by the power outage that took us down. All of our future data center expansion is going into Alchemy.
Establishing Network Redundancy
Looking back, our worst mistake of this ordeal was allowing our network hardware to end up in a state where we could not redirect all of the traffic to one router or the other. Having that option earlier on in the process would have allowed us to debug the problems more easily and ultimately we would have solved the problem faster. There’s no doubt about that.
When our two current core routers were originally deployed either one of them was able to handle the full load of the network. They were set up to share networking duties and we could have redirected traffic to one or the other if that ever became necessary. Unfortunately, the routers were not upgraded when they should have been and we ended up in a state where one of them was not able to handle the full load of the network. That situation combined with the problems beginning with the power outages led to the nasty network congestion that was difficult for us to diagnose and resolve.
Currently we are using a single router at the core of the network. Every component has been replaced and most of them have been upgraded so it is essentially brand-new and very able to handle our network traffic for the time being. We are in the process of re-establishing core router redundancy now and expect that to be done in the next few weeks. As we proceed into the future we will ensure that one of the two routers is always handling the full load of the network and the second router is standing by idle as a hot spare, should the need for it arise.
Into the Future
While investigating this issue we have been forced to look more closely at our network than we have in a long time. That has uncovered more issues that may become larger problems for us down the road and we are already working on a large scale network reorganization to both improve overall performance and make network issues easier to detect and troubleshoot. If there’s a silver lining on this dark cloud, this may be it.
Our primary local area network setup is really two separate networks, one for traffic that never leaves our network (the private network) and the other for traffic that does mostly leave our network (the public network). When you access your website traffic has to go over both the public and private networks (possibly multiple times) before you will see it come up in your browser. During our network problems it was primarily the private network responsible for the high server loads and slow website load times and email access.
The first step we are taking to improve our network setup is to completely separate out our private network from the public network. That will immediately reduce the amount of traffic going through our core routers and additionally make it easier to track down problems. More equipment will be involved but network traffic will be more isolated. As part of this process we will also be rearranging network links in as close to an optimal way as possible to further isolate traffic and improve performance. Unfortunately due to limitations in our current network architecture the best we can do is about 30% optimal and it’s likely we will not even do that well.
So, the next step in the process is a complete rethinking of how we have been deploying our servers in our data center. For ease of deploying servers and efficient use of data center space we had architected our network to essentially allow any type of server (web server, email server, file server, mysql server, etc) to live anywhere on the network. That sort of setup has worked well for us for awhile but we are now starting to see the early signs of network bottlenecks arising. For future server deployments, we will be assigning physical areas in the data center for different types of servers to facilitate a more optimal network layout between them. That will essentially localize the network traffic as much as possible and allow us to continue scaling for quite some time into the future. Overall network flow will be reduced as well, better utilizing the available throughput. This step is currently being planned and will be implemented first for the next set of servers we deploy.
All told we will be investing somewhere in the neighborhood of $300,000 into our network upgrades, not to mention all of the human time involved in planning and implementing these changes. Now that we have gotten this issue behind us we are fully committed and prepared to maintain network stability and do the work needed to improve network performance and continue to scale with our growth.
We are very sorry for all of the headaches this has caused everyone. Believe me, there was no one who wanted this problem resolved more than we did. Providing sub-par service is no fun and isn’t the way we like to spend our time. This problem took longer than it should have to resolve, but coming out of it we are now in a much stronger position as we look ahead.
112 Responses to “Anatomy of a Disaster, Part 2”
Categories:
Archives:
- November 2009 (3)
- October 2009 (1)
- September 2009 (3)
- July 2009 (1)
- June 2009 (3)
- May 2009 (3)
- April 2009 (4)
- March 2009 (2)
- February 2009 (3)
- January 2009 (3)
- December 2008 (3)
- November 2008 (4)
- October 2008 (5)
- September 2008 (2)
- August 2008 (5)
- July 2008 (2)
- June 2008 (3)
- May 2008 (6)
- April 2008 (2)
- March 2008 (3)
- February 2008 (5)
- January 2008 (7)
- December 2007 (3)
- November 2007 (3)
- October 2007 (5)
- September 2007 (3)
- August 2007 (4)
- July 2007 (4)
- June 2007 (7)
- May 2007 (3)
- April 2007 (3)
- March 2007 (8)
- February 2007 (4)
- January 2007 (6)
- December 2006 (5)
- November 2006 (2)
- October 2006 (4)
- September 2006 (4)
- August 2006 (5)
- July 2006 (3)
- June 2006 (3)
- May 2006 (5)
- April 2006 (6)
- March 2006 (7)
- February 2006 (6)
- January 2006 (8)
- December 2005 (7)
- November 2005 (5)
- October 2005 (8)
- September 2005 (19)
- August 2005 (11)
- July 2005 (4)
Other Sites
- DreamHost Status - An off-network page for all DreamHost Announcements! (Not as silly as this blog.)
- DreamHost! - The host behind the blog.
- Forum - Discuss DreamHost and web hosting!
- Web Control Panel - Manage your account.
- Wiki - On DreamHost and web hosting.
Powered by WordPress. Pool theme by Borja Fernandez, modified by DreamHost.
Like WordPress? Consider attending WordCamp LA.
Entries and comments feeds.
^Top^
September 19th, 2006 at 3:25 pm
This is great. The entire post was really well written, and I appreciate all the time put into detailing the history of the entire event, what you did to resolve it in the short term, and the short and longer-term steps you’re taking to prevent this kind of thing in the future.
I especially appreciate that it seems like you’ve taken a really grown-up approach (i.e. re-examining core-architectural decisions) and not just thrown some hardware at the problem.
September 19th, 2006 at 3:38 pm
Thanks.
September 19th, 2006 at 3:41 pm
Good job on finally tracking it down. I guess you know that your problem is serious when seven Cisco guys can’t figure it out. Here’s to hoping that the network and overall service is very much more stable for a very long time.
September 19th, 2006 at 4:00 pm
It’s kinda tough to decide. On the one hand, I’m glad that the irate, crazy nutbag customers bust your guys’ balls because it does provide the motivation for you guys to keep on top of your game, but at the same time, you guys are all so freakin’ likable that I hate seeing you get your balls busted.
What a conundrum.
Thanks for the update, as always, Dreamhost is teh rocks.
September 19th, 2006 at 4:22 pm
Wow. Thanks for a great, clear update, Dallas.
I very nearly became one of the “nutbag customers” Nate mentions, but I’m glad I’ve hung in there with you through the EIGHT times my site has gone down in the last three weeks. (Looks like fix number eight is doing the trick so far…)
Looks like overall, Dreamhost has handled this very well. “Worst case scenario” doesn’t even begin to cover what’s been going on over there, and it sounds like you’re investing enough in infrastructure — and learning enough from previous mistakes — that we stand a decent chance of getting the great hosting we’ve become accustomed to from you folks. When everything works, DH kicks ass.
Good luck in there.
September 19th, 2006 at 5:16 pm
[...] In the post “Anatomy of a Disaster, Part 2” the folks at Dreamhost explain what all caused some of the problems seen with their service lately. The problems were obviously complex, considering seven different people from Cisco couldn’t even put their finger on it. But it seems that things have settled down for them, and are running much smoother (I posted before about the low load on the two shell accounts I have, and sure enough I’ve seen my sites loading faster and performing better than they have in quite some time, even before these problems were noticed.) [...]
September 19th, 2006 at 5:47 pm
Python has now been upgraded to version 2.5. Dreamhost, please uppdate.
http://python.org/
And since I know you guys use debian, here’s the link for the package
http://packages.debian.org/testing/python/python2.5
September 19th, 2006 at 6:04 pm
This was the most useful & helpful post regarding the hell over the last few weeks.
Thanks
September 19th, 2006 at 6:12 pm
GO DREAMHOST!!!!
September 19th, 2006 at 6:36 pm
“Some of you did not experience the problems because all of the servers your service relies on were on the same core router and were not bottlenecked by that inter-router link.”
I am one of the lucky ones, it seems. Apart from some minor difficulties with email, I have not noticed any significant problems at all. It goes a long way to explaining why some people in the forum were screaming their heads off, while others were baffled at all the commotion.
The mature and professional manner in which these difficulties have been dealt with has convinced me that my faith in the DreamHost team is not misplaced. The candid, well-written missives on this blog by Josh and Dallas do them much credit.
September 19th, 2006 at 10:20 pm
Dear Upgrade Please,
This is not the place to get packages upgraded. :) Also, we only install packages out of the Debian Stable repository. If they have a stable version please let us know! Otherwise, you can knock yourself out installing Python in your home directory and running it from there.
The best spot to request things is: https://panel.dreamhost.com/index.cgi?tree=home.sugg&
Then get all of your friends and neighbors to vote on it.
September 19th, 2006 at 10:54 pm
Thank you for the frank and open statement of your difficulties and for explaining it in language even I could follow so I could momentarily feel smarter than I am.
;-)
I had experienced frustration and had even researched your competitors for a possible migration… however, it occurred to me that I may be throwing the baby out with the bathwater and this may seriously motivate you to improve your network.
Since you announced that your network difficulties are under control, my websites have been fast and webmail accessible.
Further, I had occasion to lean upon your customer service folks and ask them what became of my “disappearing comments” at DreamHostStatus.com and while they don’t know yet, Michael S. has been incredibly dilligent in trying to find out and keeping in touch with me.
The service and communication (especially the communication) is way above average and I say that as an outstanding, yet modest, customer service representative with experience in multiple industries.
You should be proud of your customer service team. They saved one customer from migrating off your network — invest in “our” network and don’t let them down!
Migrating’s a bitch.
:-p
AFTERTHOUGHT: Besides, I also checked out your competitors’ CEOs’ blogs and they just had no panache.
September 19th, 2006 at 11:30 pm
Glad to hear things are better. I’ll be setting up work related sites on my account eventually (instead of just goofing-off related sites) so I hope everything stays nice and stable for a long time (I’m sure you guys do as well).
Other than a couple of times when my sites were down I’ve had a mostly good experience with you guys and I’ve always gotten really fast replies to my support requests/questions even when they weren’t all that important overall (Even more impressive since I am in about as different a time zone as I can be from you guys).
Cheers.
September 19th, 2006 at 11:37 pm
I can’t really bitch about your service since I was one of the lucky ones but from day one I got the feeling that your network was extremely ad hoc and unplanned. I guess you did not handle growth that well and that was the break point we just experienced the past weeks.
Now that you got over it please, please I beg you take your time and plan things out this time around and build something that scales well. We (the customers) that waited out this rough spot are bound to stay with you no matter what (you do offer amazing service after all) so dont rush into things.
We like you and we dont want you to fail us again.
A happy-you-pulled-thru customer
September 20th, 2006 at 3:06 am
I just received an email today from InternetSeer saying my dreamhost hosted web site had 100% uptime :P
September 20th, 2006 at 3:53 am
thumbs up guys \m/^^\m/
Just a quick note to the guy requesting the update of Python:
Look man this is not the place to do that. Nevertheless, take in consideration that the link you sent for Python is in Debian’s “testing” branch:
http://packages.debian.org/testing/python/python
DH is using the stable branch:
http://packages.debian.org/stable/python/python
Alltough the stable branch is slow with updates to new versions, trust me, you don’t wan’t to run on “testing” branch if you are a hosting provider. :)
on the side note: this is why Subversion is still at v1.1.4 and not at 1.4. (1.1 is not even supported by developers any more)
September 20th, 2006 at 6:02 am
During all of this I had not seen any problems in my neck of the woods, I guess I am lucky. But problems are bound to happen of people can not understand that. At least you are trying to resolve the issue and future issues that might occur.
Back up plans for the back up plans.
September 20th, 2006 at 6:20 am
[...] recommend bookmarking both if you are a current Dreamhost customer. Posted at 9:19 am | Comments (0) Post yourcomment: [...]
September 20th, 2006 at 6:59 am
I knew I decided to migrate my domains to you guys for a reason. Keep up the good work!
September 20th, 2006 at 8:55 am
Thanks for the candid update, Dallas. I now better understand how some customers had so much better experience than others.
Luckily, Dreamhost’s growth has, while pointing out infrasturcture and network architecture issues, been strong enough to fund the upgrade and reconfigurations necessary to address those issues; I’m looking forward to a return to the stability and reliablility I have enjoyed at Dreamhost for years.
I really appreciate Dreamhost’s intelligent posts on these matters. It makes it a lot easier to make intelligent business decisions when one knows what is going on. Thanks!
September 20th, 2006 at 10:22 am
Great post, Dallas. I’ve stuck with you guys through all this stuff because I was confident you’d resolve it successfully. And speaking of growth, I just referred a new customer to you yesterday. Long live Dreamhost!
September 20th, 2006 at 10:48 am
@Kelly
I don’t understand this. You won’t update Python to the current version but you will update Ruby on Rails (which is no where near getting into debian stable)?
What gives?
No love for us Python guys.
September 20th, 2006 at 11:51 am
Your transparency in your operations is a refreshing change… thanks for sharing.
And to Upgrade Please: dude, WTF? Figure it out- this isn’t the venue for requesting upgrades.
September 20th, 2006 at 12:17 pm
To Upgrade Please: As has already been explained to you, there are mechanisms in place specifically for this kind of request. Posting messages in this totally unrelated forum is a complete waste of time. Use the suggestions dialog in the control panel.
September 20th, 2006 at 5:43 pm
I almost (but not quite) wish there would be another “disaster” so we could have another one of these great posts.
But maybe I can settle for just looking forward to stories on how things work to satisfy the geek side of me.
Thanks for you support DH!
September 20th, 2006 at 6:54 pm
What was so great about this post?
Talk is cheap. Once you fix the problem, I might then say “great post and thanks”.
-A disgruntled customers, who’s customers yell and scream at for advising them to use DreamHost.
September 20th, 2006 at 8:02 pm
Hey Dan, while I respect your opinion, I’m grateful that DreamHost fixed the problem, while communicating commendably, and keeping up relatively decent customer service (great agents, but too darn busy!) for the very reasonable amount I’ve paid.
If all goes well, I intend to renew with DreamHost.
September 20th, 2006 at 9:01 pm
Re: Upgrade Please
Ruby on Rails was installed due to overwhelming demand in the suggestions section of the panel I linked. ;-)
And the various upgrades were due to some really bad problems we had with the unstable releases we installed, and then I think three security point releases.
Plus, you can install it yourself if you like. It will be slightly slower over NFS from your home directory, but if you work some FastCGI magic that is a moot point!
September 22nd, 2006 at 2:04 am
Hi dreamhost, i was planning to post something really nice but by reading these comments i figured out that everything was said :)
I really find it interesting how your “behind the scene” things work. See me as a technical interested person/freak/geek.
It is indeed very nice to read that you have discovered what the bug in your network was and that you are busy avoiding that bug and others related by investing in and improving your network.
It’s also nice to read what other companies like to call inside information about the problem. They mostly just tell you: “the problem has been discovered and is being resolved” and no more info than that.
That is the most convincing reason why i stay with you, not the price but the idea that we count for you as people and not as some other costumer with a number. It makes me feel as i’m actually having a hosting dream :)
I really hope you get the strength of the network back and that you can keep it in the future! And as you communicate this well, it’s easier for us to appreciate when our server has to go down for an update, because we know it’s gonna make our server and our sites much better.
Thanks!!
September 22nd, 2006 at 9:32 am
As somebody in the computing industry its nice to see a real report of what is happening reaching the customers. I only wish my managers were as transparent with our customers when something breaks in our infrastructure. Sometimes things happen that make you go Huh! and guess what you’ve planned for that . . And guess what something was screwed up which meant your plan didn’t work! (Having dual site resilience with 4 teamed NICs in each server means that your AD Domain controllers shouldn’t go down with an AD Corruption . . Or at least not BOTH of them . . . Unless your hardware partner didn’t configure them the way they had agreed with your network partner . . . A simple shutdown to install an additional shelf in the SAN shouldn’t have taken down our hosted environment for 24 hours, but it did)
Well done on getting to the bottom of it, and then for not resting there but taking the time to optimise hardware placement to minimise bottlenecks.
September 22nd, 2006 at 12:20 pm
When do we get anatomy of a disaster, part 3, since so many of us are down right now?
http://www.dreamhoststatus.com/2006/09/22/filer-issue-causing-a-few-service-interruptions/#comments
Read those comments… so many of us have sites that are down. And yet, when you go to the emergency status page, here’s what it says:
“Network Problems Resolved”
Come on. What’s going on?
September 22nd, 2006 at 1:13 pm
a note from the future referred to in the post (like, right now)
I don’t really care why the sites are down. And truly, explaining how you’ve fixed them — which I’m reading now that they’re down again — is not how PR works.
Apropos, I’m redirecting a press inquiry into our own site (which is down) to your own press folks, Dreamhost. Who can they talk to? (seriously)
September 22nd, 2006 at 1:52 pm
What seems odd is that all this stuff happens during the middle of business hours. Who makes planned major changes to production equipment during business hours? With 100,000+ sites hosted you simply can’t be making changes during the day. It’s too risky, no matter how much you “think” service won’t be affected.
C’mon DH, you’re a business. Start acting a little more like one. I’m sorry that being a sys/network admin requires lots of after-hours work, but that’s the way it is.
September 22nd, 2006 at 10:45 pm
@cricket
Think global! The internet does not have or know ‘business hours’. I usually have to do updates after midnight but luckily all of our customers are in the same timezone.
September 22nd, 2006 at 11:59 pm
No offense, while it’s great that you guys are working on it, we’ve heard that before. A lot. We reported one of our sites down around 9am Friday. We get ignored for multiple hours, then get told were getting our connection increased to help balance load, and our site should be back up. It’s now 1am on Saturday and it’s been more than 8 hours since we were told things would be back up in a few minutes, and 6+ hours since we last emailed support to get an update. We’ve had enough, and are moving at least our most important sites off Dreamhost to some place more reliable. It’s a good thing I can’t find anything about a uptime guarantee, or I’m sure there are hundreds of customer that would be asking for refunds for today alone.
September 23rd, 2006 at 11:02 am
WHY DOESN’T DREAMHOST SIMPLE SAY, “WE ARE A WEB HOST PROVIDER FOR INDIVIDUALS, AND NOT FOR BUSINESSES”.
BECAUSE WITH THE AMOUNT OF DOWNTIME THEY HAVE, NO BUSINESS WOULD EVER (OR SHOULD EVER) CONSIDER THEM AS A HOST.
-TED
September 23rd, 2006 at 11:05 am
One slight suggestion—change that $300,000 in network re-investment to $250,000, and add another customer service person or two. When things are blowing up and bosses are breathing down your customers necks, having a honest-to-gosh living person to respond to tickets makes a huge difference. It’s not the downtime that bothers my company so much as the lack of communication.
September 23rd, 2006 at 11:50 am
[...] A recent post on their main blog goes into more detail about what went wrong recently, and what they’ve been doing to rectify it, and to prevent further problems. I suspect yesterdays outage was related to the attempts to provide a permanent fix. [...]
September 24th, 2006 at 1:39 pm
Thanks, Dallas, for this post. The information is appreciated.
September 24th, 2006 at 2:01 pm
Ted was an angry young bloke / And we see that his caps-lock is broke / What he said I don’t know / I did try to read it though / But all the yelling gave me a stroke.
September 25th, 2006 at 11:29 am
In response to Ryan: “One slight suggestion—change that $300,000 in network re-investment to $250,000, and add another customer service person or two.”
We hire regularly as our support volume increases. This month we’ve hired 4 or 5 new techs alone. This is above and beyond the $300k Dallas noted for network improvements.
DH does not run on-the-cheap, in any respect. :)
September 25th, 2006 at 4:20 pm
We hire regularly as our support volume increases. This month we’ve hired 4 or 5 new techs alone. This is above and beyond the $300k Dallas noted for network improvements.
DH does not run on-the-cheap, in any respect. :)
@Karl
Maybe DreamHost should start to consider hiring LESS, but hire more HIGHLY skilled employees.
This on-going, “we fixed the problem” then days later say, “whoops, we didn’t know what was wrong” indicates to me that DreamHost needs higher skilled employees who “know” what’s going on with the constant downtime.
Just my thoughts, with all due respect.
September 26th, 2006 at 1:53 pm
This is basically why even with the issues, I don’t really contemplate changing from DH. I’ve never had a host this honest, or try this hard.
September 27th, 2006 at 8:46 am
Still not working. Hope you guys get this sorted out soon!
Abi
September 27th, 2006 at 1:33 pm
I was quite unhappy with the way the service was getting and how unreliable, but lately things have seemed a lot better and am very pleased that you’re taking time to put in redundancy. Cheers to dreamhost for fixing your problems and being able to assess where the problems were!
September 28th, 2006 at 7:02 am
I second what Shaun said. I read an academic paper from amazon indicating they manage their entire online system with about a dozen systems administrators (run=operate, not design/build new stuff). They hire super-senior guys (and gals) who proactively look for ways to architect things so they don’t break and who create advanced tools to investigate and diagnose problems as they occur.
This post, which is refreshingly transparent and honest, is a good first step. But I fear that while you’re clearly admitting that you’re out of your depth when it comes to systems and network architecture, you’re going to have the same people come up with a solution to your problems.
Given that your entire business is hosting, it seems like you need to step up the level of people you’re hiring. Or at least go out and get some consultants who know what they’re doing for a short-time to get a “real” architecture (not one that just works out of luck, but one that is engineered to work). It couldn’t hurt to get some non-vendor, outside advice for something as critical as your network architecture.
September 28th, 2006 at 2:35 pm
[...] On their part DreamHost openly discussed their problems on their blog (part 1, part 2) which they were lauded for (Mike Davidson, Robert Scoble) because it, of all things, kept the dialogue open between them and customers. [...]
September 29th, 2006 at 3:51 am
Hey guys, I am impressed with your level of transparency in this matter. I am considering hosting with your service and this is the kind of information that personal customers require.
I also could not agree more with your post about being able to handle the full network load with one core router. HSRP was designed to provide redundancy, and most people load-balance the VLANs for S&G’s. When it comes down to the wire, you have to be able to use it for redundancy – redundancy first!
This is all the consulting you need:
http://www.cisco.com/en/US/netsol/ns656/networking_solutions_design_guidances_list.html#anchor3
And remember, if it’s not a hardware issue, the problem is almost always the load balancer! Never trust a load balancer.
September 29th, 2006 at 3:53 am
oh yea, that’s CCIE – Cisco Certified Internetworking Expert ;)
September 29th, 2006 at 11:01 am
I’ve been with you guys since 1999. I joined just after you’d been hacked into and a large proportion of your customers’ websites totally wiped out – only after the event did you discover a large % of your automatic back up systems weren’t working. Remember?
I only mention this because if I were a cynic, I’d say your ability to predict and avoid disaster hasn’t got any better as the years have gone by. Maybe you should hire a futurist – someone whose job it is to predict the worst case scenario, and then guard against it!
Anyhow, I am glad things are looking up for you. But I do agree with the observation elsewhere in these comments that maybe you need someone with more technical expertise in your support department to complement the current staff.
And finally – if I had known about the hacking event (it happened a week before I signed up, as I recall) I would never have joined. So being prepared for disaster and then avoiding it is most likely good for your business too.
Good luck,
Rod
September 30th, 2006 at 6:32 am
Fine, I know what it’s like and all that (I have a software background). But as I find that the shop that we depend on for a living is down AGAIN, please give me three good reasons (okay, give me one) why I should stay with you – after all I don’t think you’ve even offered any compensation for all this, and **** knows how many orders we’ve lost through it.
September 30th, 2006 at 8:07 am
I moved my important work off DH during the original FUBAR that happened and just happened to check a dummy site I still host here and see that it is down! I’m so glad I’m moving my business away from this piss poor hosting company. DH used to rock, I was the biggest DH fanboy there ever was, but now it’s just completely pathetic that this sort of thing is still going on. People commend you on your transparency, but they obviously do not understand what they are seeing. They are seeing a completely incompetent webhost struggling to keep mom and pop websites online. This is such a huge issue for a hosting company that it baffles me why people are still in DH’s corner at this point. I guess you do get what you pay for and if my website was down for almost 20 days of a 365 day calendar year, that is just, quite frankly, piss. poor.
September 30th, 2006 at 4:55 pm
every month John pays eight bucks / to complain that his hosting sucks / but we’ve heard it before / and if you need so much more / then switch already, ya cheap f**k!
October 1st, 2006 at 5:15 am
I’m still waiting for parts 3, 4, 5 and so on, as this story doesn’t seem to have come to an end… My sites are still down every few days for pretty much no reason at all — all of these problems are verified by other users, but none of them show up on dreamhoststatus.com. I’m getting *really* tired of this. I decided to give you guys a chance to work through your problems, but this has gotten ridiculuous. I think you owe your customes another (more complete) explanation as to the source of these ongoing problems and what you are doing to solve them.
October 1st, 2006 at 7:40 am
Our sites are down AGAIN?
It’s nice to try – but it’s better to succeed. Meantime I’m looking for another host. Anywhere has got to be better than this. The fiasco been going on for what, best part of half a year now?
October 1st, 2006 at 7:42 am
PS – anyone talking Class Action yet? I just wondered.
October 1st, 2006 at 9:17 am
I’m starting to finally get fed up. I’ve been patient with all of these issues because of the great communication and I’m not running anything mission critical. However, I’m on a tight schedule and finally found time to work and the site is having a plethora of issues again (I’m on selma). So, here I am sitting in a coffeeshop just waiting….
I’m about ten minutes away from giving up and finding a new host.
–
Tom
October 1st, 2006 at 9:45 am
Karen: there are no grounds for a lawsuit (not even close), and if you had even the slightest clue, you would have already known that.
Some of you don’t seem to understand how much worse most hosts handle this. If you had spent the last month at a-once-great Site5, you’d be willing to pay DH double for what you’re getting here. They’ve been deleting records of their problems, locking threads that even mention servers with trouble, ignoring tickets and completely lost a bunch of incoming email before customers ever got to see it… all from botching a simple upgrade.
Anyway, put all of your eggs in one basket–regardless of where that basket is– and you are asking for trouble, so don’t be surprised when you find it.
October 1st, 2006 at 12:21 pm
Transparency doesn’t mean accountability, and tons of apologies don’t make profits return.
Get your act together.
October 2nd, 2006 at 1:15 am
Karen: there are no grounds for a lawsuit (not even close), and if you had even the slightest clue, you would have already known that.
Wrong in fact. And flaming (personal abuse) is childish.
October 2nd, 2006 at 1:56 am
But aside from responding to the flame, let me be quite clear what I’m saying. I’m not calling for a class action, I am ASKING if one is in the pipeline yet. There seems to be plenty of talk about it and it’s a very valid question at this point.
I’ve stuck with Dreamhost thus far partly because I’ve found one particular service person very helpful (if often slow). I’ve also probably been over-optimistic about the company’s ability to sort all these problems out. Although as the months go by (and today I have three complaints about our sites being down) I am now looking elsewhere.
The question about a class action is relevant because it’s something I want to take into account in my decision about what to do next. Clear? Good :)
October 2nd, 2006 at 10:35 am
Karen: I hate to burst your bubble on this, but Mike happens to be correct. While it is true that “anybody can sue anybody for anything”, in order for a class action suit to proceed, someone has to identify a cause of action. Sometimes that is a lawyer, who is eilling to “drive” the thing in hopes of ultimate financial gain, though occasionally it is a “deep pocketed” individual who has sufficient resources to fund the process. Either way, such people generally have to feel they have a reasonable change of prevailing, or of at least extorting a settlement, before devoting their resources to the endeavor.
A simple, but careful, reading of Dreamhost’s Terms of Service (TOS) makes it obvious to all but the most “clueless” that there is no cause of action for “downtime” of a Dreamhost account. Mike’s comment to this effect was far from a “flame” and even further from being any kind of “personal abuse”. If you choose to make ignorant statements on public forums, you should expect someone to identify them for what they are (and if your statement evidences that you don’t have a “clue”, that is likely to be pointed out also!) :-)
Quoting Karen:
“The question about a class action is relevant because it’s something I want to take into account in my decision about what to do next. Clear? Good :)”
Wow. If whether or not there is a class action is “in the pipeline” really is “something you want to take into account” in your “decision about what to do next”, you have some serious problems with your though processes.
You’ll find life, and business, a lot easier and more productive if you rely your own resources, decisions, and actions to acheive your goals than you will waiting to participate in some adjudicated “windfall” Clear? Good :)
October 2nd, 2006 at 3:27 pm
Karen: You’re already two posts past where an attorney would have advised you to remain silent. You need to go read the TOS you were supposed to read when you signed up. You know, the TOS that you are legally bound to and accepted when you signed up. Remember that?
“There seems to be plenty of talk about it and it’s a very valid question at this point.”
And that is all the proof you need that there is no chance of anything happening. “Talk” replaces action, rather than leads to it. Besides, that “talk” you’re hearing is actually called “whining,” and it never leads to anything productive.
Not a single person with a legal clue would be talking about it on the internet. The reason they make threats of legal action, instead of just doing it, is always one of two things: 1. They already know they don’t have a case (very true here–only an idiot would think otherwise) and/or 2. They can’t even afford a lawyer, so they think mentioning one is actually going to scare someone (probably also true in most cases here, since we’re talking about a very cheap service).
Here’s another thing you can safely bet on: Dreamhost can surely afford better lawyers than whiners whose lives can be reduced to rubble by 5 minutes of downtime on a $9 hosting plan.
“I’ve stuck with Dreamhost thus far partly because I’ve found one particular service person very helpful (if often slow).”
That’s cute, but it’s not how you choose a host or a good way to run a business. Here’s another much-needed clue for you: The fact that you CHOOSE to stay here IMMEDIATELY disqualifies you as a victim deserving compensation.
Dreamhost is a web host–not a babysitter. If you can’t make your own decisions yet, then you might want to sue your parents for not doing a better job of guiding you through life.
“Although as the months go by (and today I have three complaints about our sites being down) I am now looking elsewhere.”
Doesn’t sound like you’re even close to qualified to run any type of business. That has nothing to do with Dreamhost. That’s 100% your fault for relying on a single shared hosting account to keep a business up 100% of the time. I guess you never heard of “Plan B” before?
You have a lot of learning to do. Focus on that instead of asking dumb questions about make-believe lawsuits. Head over to Google and start typing in words like: redundancy, failover dns, off-site backups, etc…
“The question about a class action is relevant because it’s something I want to take into account in my decision about what to do next.”
No, it’s not relevant–its mere mention is a sign of ignorance… nothing more.
“Clear? Good :) ”
The only thing you made clear is that Dreamhost doesn’t have a minimum IQ requirement when it comes to signing up for web hosting.
Here’s a free tip: Go learn something before you come here to teach. If you’re already thinking up what you consider to be another witty reply, probably ending with “Clear?”… it’s probably not going to sound as smart to anyone here as it does in your head.
Clear? Good. :)
October 3rd, 2006 at 2:03 am
Wow some people have a lot of time on their hands.
October 3rd, 2006 at 1:37 pm
[...] In less than a year, Dreamhost has already upgraded their already insane webhosting plans with up to 10 times more webspace and at least 2 times the bandwidth – an upgrade that comes to all existing Dreamhost customers. In other words, Level 1 plans start with 200 Gigs of webdisk and 2 TB of bandwidth. But is this really a good thing? I mean, we’ve already experienced a wackload of problems with Dreamhost that seem to be ongoing. Now, I’m not sure if the previous increase in bandwidth and diskspace was a direct correlation with the network issues that Dreamhost is facing now. One thing that’s for sure is that these “insane” deals are in fact bringing Dreamhost a lot of new customers. According to their blog and webhosting.info, Dreamhost hosts over 300, 000 domains and ranks 24th among the largest webhosts in the world – a nice jump from 200, 000 domains and ranked 39th back in January 2006. Having researched that, there is no doubt increasing bandwidth and diskspace means increased customers. Increased customers means increasing number of shared servers. Increasing shared servers means increasing network problems. Which begs the question again, is this upgrade really a good thing? I remember when some of my sites would take up to 20 seconds to load because of the high resource usage on the database server and there was little that Dreamhost Support could do except to tell me to wait as they tried figuring out the problems. [...]
October 3rd, 2006 at 3:54 pm
cricket – I hope you realize that your analogy to Amazon is completely inane. A web host, particularly one as open as Dreamhost, providing shell access and a variety of programming languages and environments for thousands and thousands of customers is going to have a HUMONGOUS work load of menial support just to deal with the people who don’t know what they’re doing. Dreamhost has highly skilled people, it’s just that they’re not manning the support email to respond to your $7.95/month question. They’re dealing with the real issues.
For everyone bitching about how bad Dreamhost is, please go try some other shared hosts. You are likely to get burned anywhere you go. If you have a great experience somewhere, just remember that it’s probably a lot of luck. Also, you should keep in mind that anecdotal evidence is complete bullshit. I have tried dozens of shared hosts to host hundreds of websites, and there is no shared solution that lives up to your expectations. If you are serious about uptime you need to get on a dedicated server, period. Also, it’s worth noting that very few hosting companies have dealt with all the problems Dreamhost has faced and solved over the years. How will they measure up? Only time will tell.
October 3rd, 2006 at 3:57 pm
Wow, guys.
I’m ashamed of the way Karen has been treated here.
A lawsuit would probably not succeed and I personally wouldn’t go that route, but DreamHost really has sucked for me for the last nine or ten months and I’ve moved some of my business elsewhere (bluehost).
I “like” DreamHost and I appreciate their transparency, but they’ve lost a chunk of my business this year.
October 3rd, 2006 at 4:52 pm
Rob:
Karen may not have been greeted with a big, warm, fuzzy, hug, but she was not treated badly either. She was told she was wrong, and that she was clueless.
She would not have had to “suffer” either of those “mistreatmeants” if she had the slightest sense of netiquette; commenting on a corporate blog by asking, “Anyone talking Class Action yet? I just wondered.” is the kind of “drive by” idiocy that pollutes most every blog that allows comments these days.
I’m sorry you felt she was treated badly, but that is no reason for you to be ashamed.SHe brough it on herself, and fueled it with her own responses. Maybe she learned something, maybe(probably)not, but I bet she will think at least a little more before she next types (and thinking before one types is a “Good Thing”tm).
October 3rd, 2006 at 5:30 pm
what a great read. thanks!
karen, stfu.
i *heart* dreamhost.
October 3rd, 2006 at 6:19 pm
Personally, I do think they suck. So, yeah. We were cheap and had 35-50 websites hosted on two monster plans. We couldn’t take customers bitching at us all the f*cking time about the email or webiste being down.
Becuase we had two plans, one hosting plan NEVER had problems, but the other ALWAYS had problems. No matter how many times we would suggest that the server itself was bad or the equipment (router) that was controoling its access was bad we were told everything was okay.
Really, it was our fault, we stayed when we should have left. So we’ve moved most of our sites to liquidweb on a virtual private server with cpanel and total freaking control of the last detail and I say we couldn’t be happier.
Adios Dreamhost…keep dicking around with your crap: non-redundant equipment, non-working generators, generators that catch on fire. You say your in the same data center as MySpace. Get a clue! They’re down all the time too!
October 3rd, 2006 at 6:21 pm
I’ve jumped to a few different hosts and I gotta say, DH is still the best. I would say that the downtime is roughly the same, but I get SO much more for my dollar. (shell access, DNS access, actual updates on what is going on…)
It also comes down to “you get what you pay for”. These businesses that are losing all these sales and whatnot are trusting their whole web store to a $7/mo account? I’m sorry, but if you need more 9’s* you better have a thick wallet. I ended up moving my critical services to a colo server somewhere else. Since I pay for bandwidth there I still host all my large files here at DH. The best of both worlds!
My hat is off to DH. This is hard work but you are doing a good job. Keep it up!
-paul
* as in (99.99*N)% uptime
October 3rd, 2006 at 6:52 pm
Thanks for the great info on what has been going on. I still love DH!
October 3rd, 2006 at 7:15 pm
Thanks guys. Great job.
October 3rd, 2006 at 7:59 pm
[...] In the networking world much has already been written about Dreamhost’s Summer of Discontent. Read this two part episode on how Dreamhost ran into an unending cascade of hosting issues. Dreamhost hosts over 350,000 websites, so when they went off the air, large chunks of the Internet went away. There is a lot to learn from the repeated outages that Dreamhost experienced. First, the importance of building for redundancy is key to any networking architecture. Second, Murphy’s law rules in IT and networking. If things can go badly, they generally will and with networking in two’s and three’s. All these point to the importance of both change processes and proactive engineering of the network. Interestingly not all the issues Dreamhost ran into were the result of process. There is a story within the story. [...]
October 3rd, 2006 at 8:07 pm
“I’m ashamed of the way Karen has been treated here.”
Yeah, we were too easy on her, but we try to keep things family-friendly around here.
“I’ve moved some of my business elsewhere (bluehost).”
You do realize that most people that hate DH blame it on them being a “cheap host.” You just moved to an “even cheaper host” that has a lot of current complaints including a lowered CPU cap. Remember how much crying people did when DH had a CPU cap?
“I “like” DreamHost and I appreciate their transparency, but they’ve lost a chunk of my business this year.”
That’s fine–but understand that every single shared hosting company on the planet is always losing a chunk of someone’s business.
If uptime is important, host-hopping isn’t quite the fix that redundancy is.
October 3rd, 2006 at 8:21 pm
“We were cheap … We couldn’t take customers bitching at us”
Take DH out of the sentence and it still makes sense. Did you think being “cheap” was going to get you the same results that people get when they spend $$$ on redundancy to keep their customers online?
“So we’ve moved most of our sites to liquidweb on a virtual private server with cpanel and total freaking control of the last detail and I say we couldn’t be happier.”
I’ve used LW as well. They’re fine, but they’re not perfect either.
And the relevance in comparing a shared hosting account to a VPS would be… what? Do you realize that your VPS isn’t that great compared to multiple dedicated servers in various datacenters with failover DNS? And that a Corvette is faster than a skateboard? Are you seeing the apples/oranges pattern?
And try to find a control panel that has been exploited more times than CPanel. Nothing personal against CPanel… but they’ve probably had a lot more problems than DH, so I wouldn’t use them as ammo in a DH-sucks argument.
October 3rd, 2006 at 8:38 pm
[...]ken Says:
September 30th, 2006 at 4:55 pm
every month John pays eight bucks / to complain that his hosting sucks / but we’ve heard it before / and if you need so much more / then switch already, ya cheap f**k![...]
learn how to read, buddy. I said I already moved my sites off of DH, dipsh*t.
October 3rd, 2006 at 9:45 pm
I program for many clients whose sites are hosted at many other places…they have ALL had problems in the past 6 months.
DH still rocks, cause look what I get for $10/mo!
-Tom
October 3rd, 2006 at 9:54 pm
Glad to hear you’ve got the problems on the run. I’ve just renewed for two more years, and I’m confident I won’t regret it. (Not to mention I’m too lazy to move anywhere else). Now I really must get to work on consuming all the transfer bandwidth I’ve got now. Tetrapixel pictures of PeeWee Herman anyone?
October 3rd, 2006 at 10:16 pm
[...] 0. Introduction.1. It’s our birthday? Double the bandwidth, tenify the disk!2. Shared + Dedicated? Shedicated!3. Down with DRM? Up with Files! Forever!4. RSS Feeds? The Discussion Board!5. Who needs catch-alls? Not+you@dreamhost.com!6. Transferring a .com? .net? You need an auth code!7. Old Charities? New Charities!8. Network problems? Fixed!9. DHSOTM? DHSOTM!####################################0. Introduction.Did you SEE that Table of Contents? Holy Cram-oly! Maybe I should donewsletters like last month’s more often! I don’t know where all thisstuff came from, but I PROMISED something SUPER last month, and whatbetter chance to DELIVER something SUPER than by delivering LOTS of SO-SO? It’s the shotgun method of great web hosting features!It’s also like my mama always told me. She told me, “Son, better to dolots of mediocre things than one thing that’s really good. That way,people will know you were trying, even if you are a little dumb.”That crazy Mama, she made me! Or maybe it was all my Papa, with hisfailed shotgun method of birth control.And uh, my apologies that there’s NO TIME for a silly THEME this month,I’ve got WAY too many average, run-of-the-mill, ho-hum features to plowthrough!###############################1. It’s our birthday? Double the bandwidth, tenify the disk!So, what feature WAS I alluding to last month anyway? Could it that it’snow been NINE years since we registered dreamhost.com, and to celebratewe’ve upped everybody’s disk and bandwidth, AGAIN, wellll beyond therealm of possible use: up to TEN times the disk (including growth rate)and at LEAST double the bandwidth (also including weekly growth rate)!And as usual, all existing customers get it too, and you don’t even haveto do anything to upgrade. It’s … already … happened.Anyway, that puts us at, um. Er. I’m not sure. A lot. Check it out:http://www.dreamhost.com/hosting.htmlBut WAIT! Is that a new site design for dreamhost.com I see? And what’sthat on the main page? Pictures of employees and RSS feeds? GRADIENTS?Amazing! What version of the web would you even CONSIDER a site likethis? It’s gotta be more than 1.5, possibly even 1.9..This is some kind of killer upgrade, for sure. Possibly even SUPER.And for those of you who say, “SCREW all the features, ALL I *really*want is website, email, and database stability!” I say, “You are smart.”#####################################2. Shared + Dedicated? Shedicated!Okay smarties. If you’re so SMART, and you hate features, why do youkeep hosting with us? Who’s SMART NOW?! Don’t worry, you still are.Because we’re half-way back to dedicated servers! And everybody KNOWSdedicated servers are The Best Way to get all the LAME features ofDreamHost plus all the AWESOME admin skills of YOURSELF.Except wait! You’re not an ADMIN (some of you). And DreamHost is notreally THAT bad.. now that we fixed our network problems (see eightbelow). That was not a subliminal ad for the movie (with Paul Walker),that was to point you to section #8 below. (Snow dogs are cute.)What I’m saying is, maybe you DON’T need root access to a server to behappy. Maybe you just need your own “shared” machine.. just not actuallyshared with anybody else. And maybe, just maybe, you’ve got $395/monthburning a hole in your butt.In which case, “Shedicated” hosting is for you! OR, as the less creativecall it, “Strictly Business Enhanced”:http://www.dreamhost.com/hosting-dedicated.htmlGet all the FUN of your own server, but without having to deal with anyof the DUMB. Apart from the fact you’ll still get this newsletter.And THAT is a SUPER feature, desu ne?######################################3. Down with DRM? Up with Files! Forever!Nah, I didn’t think so EITHER.What would be REALLY super is if there was an iTunes-like digital storethat didn’t put any DRM on their songs and let you re-download them ifyou ever lost them. Or, if there were an online file backup servicewhere you only had to pay once, instead of monthly, THAT’D be super. OrACTUALLY, if DreamHost offered an easy way to email large files to myfriends. Now THAT would be super.But all three, in one, that’d be complicated, stupid and POOPER.Well, you know me. All in three!https://files.dreamhost.com/Try it now while it’s still in BETA.. before it gets GOOD!Oh, and if my NEWSLETTER doesn’t make any SENSE:http://blog.dreamhost.com/2006/10/03/itunes-music-soreFor those who like pictures. And don’t already have it all RSS-ified up!######################################4. RSS Feeds? The Discussion Board!RSS? On the Discussion board? Yes; it’s true, “REALLY SUPER SYNDICATION”is here at last, here at least, thank lord almighty, it’s here at last!http://discussion.dreamhost.com/wwwthreads.plSee all those little orange peels all over the place? That’s some superfeature for yah.#######################################5. Who needs catch-alls? Not+you@dreamhost.com!I think we’ve already had more than our fair share of totally awesomefeatures for the month of September, plus three days, but I’ve beenwrong once before (I’m sure of it).Which is why it never hurts to add a few more not-your-run-of-the-millfeatures. Say for example, this one… a feature I’m pretty darn sure NOOTHER WEBHOST ON THE PLANET OFFERS. And I say this after literally yearsof checking:Starting right now, all your email addresses have a sneaky new feature: Built-in catchalls!How does it work? It’s easy! For every email address we host you can adda + sign and then ANYTHING before the @, and it’ll go to the same place.So, if you have an email address like say:josh@ilovemoneyandbunnies.comThe following addresses now all go to the same place:josh+ferrets@ilovemoneyandbunnies.comjosh+jones@ilovemoneyandbunnies.comjosh+dreamhost_spams_dont_they@ilovemoneyandbunnies.comWhat’s the point of that?Well, now every time you have to give out your email address to somebodyyou hate, you tell them it’s got a + in it and then some uniqueidentifier like:josh+damnyousteve@ilovemoneyandbunnies.comAnd then when they start spamming you some day, you can easily set upyour mailbox to auto-delete messages sent to that address!SUPER? More than its fair share.########################################6. Transferring a .com? .net? You need an auth code!Finally, the feature we’ve all been waiting for! I know you saw it upthere in the table of contents and were just DYING to get to it! But youhad to wade through all that other stuff first, because to skip aroundlike that would be like getting a Mona Lisa beach towel.It would violate the sanctity of my art.I thank you for respecting my wishes, and reading this newsletter in theorder it was originally intended.You will not be disappointed. And I will make you wait no longer..It’s TRUE, .com and .net are finally switching to EPP.. which meanswithin the next few weeks (it should be by October 15th) to transfer a.com or .net domain *REGISTRATION* you’ll need to get an auth code fromyour existing registrar.You can see your existing auth codes for domains registered through usalready at:https://panel.dreamhost.com/?tree=domain.transferFinally, you can relax. The superest of all supery features is past us.##########################################7. Old Charities? New Charities!Sigh, it seems like it’s been like a quarter of a year since we had apost like THIS in the newsletter. I’m getting all doe-eyed. SNIFF.We’re switching up the charities available for you to donate to and getyour donations matched by us at:https://panel.dreamhost.com/?tree=home.charityThe first place we’re letting you donate to is “Defective By Design” ..the ones who are having a “Day Against DRM” which is what INSPIRED our“Files Forever” service. At least to be launched today, probably a goodthree to five years before it’s ready.http://defectivebydesign.org/The other place we’ll match you up on is “Santa’s Helpers” (of GonzalesCounty, Texas, for some reason) .. a place that’ll help get some kidssome presents this Christmas if you give them some money. At the rate wechange charities, I figured NOW IS THE TIME.http://www.santas-helpers.org/In retrospect, in July-September we got $904.21 from 36 people for theWorld Wildlife fund, matched it, and also got $383.32 from 17 people forThe Colorectal Cancer Coalition, and decided to match that too.Aw…#####################################8. Network problems? Fixed!Do you know I actually moved this section to number eight just so Icould do that LAME eight below joke above? I now realize how pitifulthat is. In fact, this whole section is nothing more than a cheesy wayto bulk up this newsletter to be more SUPER after last month’s dismalexcuse for a newsletter darkened your e-mailbox.All I’ve got to say here is really, we’ve turned the corner and theperformance and stability problems we had much of the last two monthsare slowly fading into the setting sun. Literally. I’m not sure how itcould be literal, but I always like it when football announcers saystuff like, “He literally knocked his head off on that hit!”Anyway, Dallas splained it all a couple of weeks ago on ZE BLOG:http://blog.dreamhost.com/2006/09/19/anatomy-of-a-disaster-part-2/SUPER?!########################################################################9. DHSOTM? DHSOTM!Finally, the REAL secret of this month’s newsletter has been revealed!The DreamHost Site of the Month winner is none other than:http://www.santas-helpers.org/Look familiar? I promise, they actually won! In fact, that’s why I madethem a charity. Believe it or not, I wrote this section before sectionnumber 7. That’s just the way my artistic soul had me do it. I can’tcontrol it. I just have to be happy I’ve been blessed with such a giftand do my best to share it with the world.Get your own site all popular and make lots of money, bunny:https://panel.dreamhost.com/?tree=home.dhsotmThat’s what I’d do, if my soul weren’t just in it for the love of mycraft.#########################################Now THAT’S a newsletter?Josh Jones!Now NOT a newsletter?https://panel.dreamhost.com/id/?tab=contactUnsubscribe, foo’! Rate This Post: [...]
October 4th, 2006 at 12:55 am
We have been with DH for almost five years now. In that time, we have experienced little down time and the times we did, it was fixed within hours. However, the past four to five months have been very troubling for us. With the email servers going nuts and having to wait sometimes up to ten minutes just for the page to load, then another five more for an email letter to open….it was very frustrating.
I wrote to DH support and in two days received a reply that it was in issue and they were trying to locate the problem. In the meantime, since they dropped the catch-all forwarding to an alternate address, I started losing valuable emails and my business relies on email in order for guests to contact me.
It was decided to drop the webmail and have DH mail come to my primary account with an alias created just to keep the requests coming in.
Now, the worst of this was during the heat wave and the power outages, but the problems with the email being slow and non-responsive started way before that. It was when I got a call from one of my guest who asks why I didn’t return his email, I told him I never got it. I later discovered that many emails never made it to my inbox and I am wondering now, just how many reservation requests I lost due to the mail server being weird.
After many discussions, my partners and I decided to give it a wait and see to see if things would work out. Well, some of it has, I still check the webmail because I have an address book on there with five years of customers that cannot be transferred to my Verizon account. So, I have to log in there to email out an address and it gets forwarded to my Verizon account.
Should I stay with DH? For now, YES….but let me tell you. I am very faithful to who I use for my business needs and when I get so disgusted with a company to the point of leaving them, it really must have been bad.
Guys, you have been great and the newsletters have always been fun to read. But the one that came when you were in Hawaii and I was not able to get my emails….well…it didn’t sit well with this old man and you almost lost a valued customer. And trust me, when I decide not to use a company or contractor something really had to go wrong, because as a small business owner, I know the value of losing “one” customer.
Keep us posted, be honest with us as we are faithful to you. But when major problems come down, let us know, and in a way that some of us non-computer learned people can relate to? I know some, but thank goodness for webmasters.
My rant is over and I am stepping down from the soap box.
Andy
If its funny, enjoy the laugh. If it says please forward to all your friends, remember I have more than one friend and most likely have received it already.
October 4th, 2006 at 1:01 am
Awesome, I really love this increased bandwidth and space. Hopefully you guys arent over-extending yourselves here. Its much appreciated!!
October 4th, 2006 at 1:50 am
Thanks for the education, Mike. 8 ]
I’m really quite amazed at how vigorously some folks are defending the downtime at DH.
Everyone’s mileage will vary. Having said that, clearly not all shared hosts perform equally well.
A simple googling will give us several comparisons (possibly with varying degrees of impartiality).
http://www.hosting-review.com/
http://www.thehostingchart.com/
http://bestwebhosting2006.com/
http://www.rubyonrailswebhost.com/
For example, I selected my replacement hosting company not because I’m a bonehead, but rather because:
a) they are on every top 10 list I’ve seen
(I’m sure someone will inform me that they paid to be on all the lists.)
b) they are ranked #1 for ruby on rails hosting.
(Now someone will tell me Rails is hype and PHP Rulez!)
c) they post their support phone number in bold type on their home page and give me great service.
That’s good enough for me. Do what you like.
October 4th, 2006 at 2:02 am
Well … now we’ll see if the $300k investments will keep up with the new disk spaces and bandwidths …
October 4th, 2006 at 5:32 am
Much like “Andy”, I was getting a little towards the end of my patience. My site is more personal than business in nature, but my ability get to email is *critical* for work and personal contact – so these recent outages were getting old. I was *this* >
October 4th, 2006 at 9:30 am
I think some of you need to take a look at DreamHost’s log of redundancy problems lately and, if you run a site that your living depends on, create a mirror of it on another host. Then when you have problems on DH you can fail-over to the other until DH fixes the problems.
October 4th, 2006 at 10:24 am
Thank you for the complete story. Now I can go to my boss with a better explination fo what was going on and how you are fixing it. She was ready to go back to a local host who is real bad because of all the problems you had. The reason we switched to you was because of all the problems we had with our previous host. Which was mainly e-mail problems. We are in the news bussiness and count on our e-mail for daily stories. So when e-mail is slow or doesn’t make it at all, it’s a very big deal. So when you started having all the e-mail problems, I really started to get into hot water here. Our web site is very important to us, but e-mail is critical. I sure hope this is the last of the problems.
October 4th, 2006 at 11:31 am
[...] In response to this comment on a recent entry at DreamHost: Ken says: [...]
October 4th, 2006 at 12:26 pm
I am not a Dreamhost client, but after reading through all the postings on this blog, I am considering hosting one of my smaller sites when ready. I have just spent 11 months in hosting hell and no matter who agrees and who doesn’t, I find it refreshing that a host is up front about the problems. I also agree, however, that a real troubleshooter needs to be hired to look for possible future problems.
I joined and then left a top hosting site because I at first managed to access my control panel and proceeded to upload my site. I use dial-up (have no choice in my area) and had to quite for a few hours because of an expected call. When I again got online and tried to access my cpanel, google no longer recognized the login URL. After 5 days of back and forth nonsense about what to do, I was asked why I used that method instead of another method. I used it because it was what the sales team gave me, and it worked the first time.
No explanation was given as to why it didn’t work the second time. A visit to their forums answered my question. Several of their servers had been blaclisted. This was just one hosting issue. I checked user reviews all over the net and they were glowing about this site. (I don’t trust webmaster reviews). My big mistake was not checking the sites own forums.
Several have made the point here about putting your eggs in one basket. A very good point. I have a dozen sites that I am working on and I plan on using 4 or 5 different hosts. Another item to consider, is when you are allowed to host more than one domain in your package, for heavens sake don’t host sites that are in direct relation to each other and therefore having the same keywords. You will cause the server to spam the search engines.
Many of the suggestions offered on this blog were great. Those of you who choose to host all of your sites in one package or with one host are begging for trouble. I am tired of moving around the web. The situation is the same just about everywhere. Sites expand too quickly before they are ready to handle it. I applaud this site for having the courage to admit their problems.
Only one other site on the net has done so as far as I have seen, and that is Bluehost.
I will be keeping my eye on Dreamhost, and hope that I see steady improvement so that I can place one of my sites here.
The only failures in life are those who quite trying.
October 4th, 2006 at 11:07 pm
I signed up with Dreamhost on June 30 so naturally my first impressions left a lot to be desired. After my sites being occasionally inaccessible and unreasonably slow for a few weeks, I nearly jumped ship to another web hosting service this week.
My decision to stay was based upon the fact that the DH team was so direct and realistic about what was going on. I appreciate this kind of openness. And like someone above mentioned, the team is just so darned likeable.
My web needs don’t necessarily demand that I have dedicated lightning-fast super servers. I’m realistic about both my needs and expectations with a budget hosting company like Dreamhost. They offer exactly what I need for a price that I am comfortable with.
The reality is that machines do break down. Hardware does inevitably fail at some point or another – whether it’s ten days or ten years from now. Upgrades will always be needed and shared servers will always get finicky under heavy load. What matters most to me is the response time to these sorts of things. DH seems to be on top of what’s going on and making an earnest effort to set things right again.
October 5th, 2006 at 5:03 pm
i don’t give a shit what you say the problem is, everyone else here seems to be kissing your ass and thanking you guys for explaining, fucking fix it, don’t explain it, I’m not fucking pay for your server do be down then write some explanation fuck brains, i don’t give a damn about you guys or your feelings, this is business and i’m losing money and visitors, you guys suck ass and need stop fucking around and get the site to stay online for longer then 3 hours at a time, you guys suck!
October 5th, 2006 at 5:04 pm
seriously man, get your act together dreamhost, 45% downtime is absolutly desgraceful my goodness…
October 5th, 2006 at 5:06 pm
and why the hell can’t you have a normal front page for a host, it’s always someone sleeping or a bunch of nerds making faces and acting retarded, god damn I am so fucking pissed with you guys.
October 5th, 2006 at 5:53 pm
John, you should kill yourself.
October 5th, 2006 at 9:25 pm
my story is almost identical to comment no 90.
I signed up in april, so when i saw major probs after just a few months (a few hours downtime here and there, as well as verry slow page loading etc)… I was worried. The only reason I didnt jump ship emidently is the wealth of praise that dreamhost seems to recieve almost where ever their mentioned. Added to wich support was always helpfull and clear as to what’s going on, if not a little slow at times.
Now that the ’shit is sorted’ i gotta say im a DH fan. My sites are incredibly fast since the probs got sorted (though they were pretty slow to start with…) and now your increasing my BW and storage?!?!??! OMFG. i was stoked with what i was getting before, it matched or beat the snot out of anyone else for the price, AND it grew with time… and now i find its doubled / tentified!?!??!
wow guys. keep up the good fight. hope this new update doesnt streach you to far…
luke
ps
to all those who like to spend their time ripping DH appart, go to another company, or better yet, look up reviews on the other companies who’s service, BW and storage match DH. the majority get ratings of less than 10%, and the ONLY comments they have are neg. DH is almost all +++++. And as many others say… if you rely on your webpage for a living… then its a good idea to invest more than bloody $9 on it!!! my god… some people are IDIOTS>. > thats my little rave
October 6th, 2006 at 1:09 pm
“if you rely on your webpage for a living… then its a good idea to invest more than bloody $9 on it!!! my god… some people are IDIOTS>. > thats my little rave”
Dear Moron: The folks paying $70 a month for the “Strictly Business” plan (you know, the one targeted toward businesses) are getting the same crappy service as everyone else for the last 4 months. DH is *almost* worth $9 a month given its current state. It’s not worth anywhere NEAR $70.
October 6th, 2006 at 3:31 pm
To the moron that entered post #96:
Putting the word business in a the name of a shared hosting plan doesn’t change the fact that it’s a shared hosting plan. It’s on the same computers as a $9 plan, so why would you think paying for more features would equal more reliability?
$70 is still cheap. If it’s not worth investing in redundancy, then maybe it’s time to quit pretending that your site/business is important.
I can’t think of a single business that credits their success to crying instead of thinking, and putting the entire site’s uptime in the hands of a single shared hosting account… regardless of who the host is.
October 7th, 2006 at 2:56 am
[...] (Perhaps our web host was at fault [that’s what I would like to believe]. We use Dreamhost. [This guide to deploying Rails on Dreamhost is invaluable.] This past summer has been pretty difficult for Dreamhost, and, as a bonus, Rails support isn’t the best. [...]
October 8th, 2006 at 3:08 pm
Now that kind of honesty and transparencey… that’s what I’d expect from a bunch of Mudders. Keep up the good work guys. I can’t express how nice it is to be kept in the loop.
October 9th, 2006 at 11:33 am
Here’s a crazy idea for disgruntled users. Instead of spending $9 a month for web hosting exclusively with DH, why not spend $18 and sign up with a second hosting provider. It’s not really a big deal to move new stuff back and forth at night. Then, if a provider has a particularly bad run of luck (and it is luck many times!), you’re covered. Pretty cheap insurance if your living depends upon your web sites. For those of us that have been in the business a long time, we really appreciate the honesty and up front discussion of problems.
October 9th, 2006 at 3:42 pm
Or, to make #100 even better… sign up for a DNS failover service and have everything automated.
Anyone that can’t afford a minimum of two shared hosting accounts and DNS failover doesn’t have an important site and shouldn’t whine about downtime.
You can kid yourself and pretend your sites are important, but no important site/business relies on a single shared hosting account for 100% uptime. This has been said many time, but some people just like to cry because it’s easier than thinking for once.
October 10th, 2006 at 6:59 pm
Guys,
I wish I knew the problem earlier! I could’ve helped you with equipment upgrades for your network, and my sites wouldn’t have been offline so sporadically!
Anyway, ggod to hear things are looking up!
~ Charles
October 11th, 2006 at 5:10 pm
This is all very interesting. I don’t have an account with dreamhost but I have had accounts with Mindspring, Earthlink, nomonthlyfees.com, Globat and currently AnHosting.com / MidPhase.com. These were business accounts and shared hosting accounts.
So far, every one of these listed has had problems. Globat, specifically, was the worst of the worst, never answering one e-mail in a year, never giving any of the promised support, and placing our sites on the slowest servers and routers in the known universe. AnHosting, our current host, has excellent customer service except that they rarely can figure out the problem or give me an answer that makes any sense. And out of the 4 accounts I have there, 3 are pretty slow or miserably slow.
So, I don’t think these issues are dreamhost’s alone. They seem to be rampant everywhere out there on the net. I suppose that if you want to deliver streaming media and fancy server side sites, you better find a dedicated host with big bandwidth and some smart people on the other end.
All I can say is, I have yet to find a host that I can reccommend and still get to sleep at night.
It is refreshing to have someone on the IT side of things explain some of the inside issues in the server farm business. She should be given a gold star for that regardless of the never ending struggles these web hosts have in staying on line.
October 13th, 2006 at 3:44 pm
[...] Now, you may think it has something to do with this, this or maybe this. [...]
October 14th, 2006 at 3:47 pm
[...] But no commercial webhosting company I’m aware of currently offers grid-based hosting. This leads to outages and slowness for clients. DreamHost has had a number of recent problems. To their credit, they are quite open and honest about what went wrong. But there’s also a constant parade of announcements of servers going down for 10 minutes here, 10 minutes there for RAM upgrades, hardware replacements, etc. [...]
October 17th, 2006 at 2:46 pm
[...] Dreamhost has been busy with a filer migration and network upgrade. Fortunately the results are visible. bandwidth dreamhost hosting mrtg performance speedThese icons link to social bookmarking sites where readers can share and discover new web pages. [...]
December 3rd, 2006 at 1:57 am
[...] Network problems? Fixed! The network problems DreamHost experienced during the month of July and August are finally solved. [...]
December 4th, 2006 at 1:50 pm
So does this mean everything is back to the way it used to be? No more problems with email, no more slow sites, no more downtime? Have all the steps that were mentioned actually been implemented? Have the public and private networks been separated? Has there been physical space allocated in the data center for different kinds of servers? DreamHost really sounds like a dream, and I would love to host with them, but you know what they say about things that sound too good to be true…
December 5th, 2006 at 11:00 am
[...] It’s true that DreamHost has had some problems reliability in 2006. And they don’t have all the features (like dedicated servers) that some people require from a hosting provider. But I have to give them kudos for working through it while continuing to provide tools that are valuable for those who are eager to create, distribute and sell social media. [...]
March 7th, 2007 at 12:38 pm
This place is a fu
March 29th, 2007 at 3:39 am
[...] to the Dreamhost blog everything should be returning to normal now. Dreamhost have also announced that they will be [...]
October 15th, 2008 at 2:48 am
Thank you..