Another Anatomy
April 7, 2008 on 12:23 pm | In Foobars, Hardware, Insider View, New Features by Josh Jones | 125 Comments
Okay, nothing silly this time, I promise…
Some of you may have noticed that we’ve been having what a problem that is, although maybe not the worst in DreamHost history, definitely in the top 5.
There has been a DreamHost Status post about it, but it’s been going on so long, there obviously needs to be more said.
The History
The events that conspired to cause this horrible performance for everybody in our “blingy” cluster actually started to take root 19 months ago.
That was when I made this post asking our customers for some suggestions on storage. I made the mistake in that post of mentioning the name of one particular storage vendor who apparently does a search for their name in rss feeds of all kinds of blogs. I won’t mention their name again here, to test if they REALLY read this blog, but they were the one on the list right after “Netapp”.
Anyway, immediately a sales guy from there was hounding me about how great their product was. It would have super-duper reliability, super-duper performance, and super-duper ease-of-management. It was super-duper expensive compared to our current solution (about 3x the price per GB), so in the end I declined.
But, over the next year he kept hounding me and hounding me, and eventually the price came down to something in line with our current costs, so we decided to try one unit for our new cluster, “Blingy”. After we were satisfied with our internal testing, Blingy went live with the new storage solution in December 2007.
Smooth Sailing
At first, everything was fine, performance was great, everybody was hunky and dory. But then, as usage started to go up, the new file system started acting up. Around the same time every night, the system would stop responding to NFS requests for a while, which would immediately break web and mail service for everybody in the entire cluster.. thousands of customers.
Our Bad
Now, it can be a big mistake to put live customers on any new system. But honestly, we’d tested it lots, researched it a ton, and we added people very slowly at first, and it performed great.
Our biggest mistake I believe had nothing to do with what specific vendor or hardware we went with.. it was simply putting so many eggs in one basket!
Even with our Netapps (which are pretty much awesome), there are problems from time-to-time. However, a typical hosting cluster will have a dozen or so Netapps, which means any problems are one twelfth as big.
With Blingy, EVERY customer is on this one “mega” filer, which in theory should make for better performance, reliability, and ease of management. And since we got the clustered solution (in an active-active configuration)… there really is no single point of hardware failure in this thing.
But, as it turns out, there are a lot of non-hardware failures in the world.
Their Bad
Well, the techs at the vendor couldn’t figure out what was causing the NFS freezing, and so they recommended us doing a major OS upgrade to hopefully fix it.
During this whole time, the fiber channel disks were slowly filling up, and we’d been trying to move large files off to the sata pool (it’s a two-tiered solution, and there’s a feature that automatically moves less-accessed data to lower tiers).. however the thing couldn’t move the data fast enough. It couldn’t finish doing a “move job” in a single day, and every day it’d sort of “crash”, which would screw up the move job, and nothing would get moved.
Also, as the disk kept getting more full, performance kept getting worse, creating a vicious cycle. We ordered some more fiber channel disk shelves at the end of February to grow the main FC volume, since we couldn’t get things off to SATA, and it was supposed to come on March 10th and be installed at the same time as the major OS upgrade.
However, the disks didn’t end up getting installed until March 25th, and at that point it turned out we could NOT grow the FC volume with these disks (well, it was technically possible, but their on-site techs recommended VERY VERY heavily against it.. it would severly impact performance), which was sort of the whole point. So now we had a new FC volume which we still had to migrate users to.
Your Bad
Of course, this whole time, new customers just kept signing up, and being added to Blingy. What were you guys thinking?
By this point we knew this was a bad idea, but we didn’t have a new cluster ready (we’d expected Blingy to grow for another couple of months), and we try to never ever grow old clusters again once they’ve been “shut off” from new signups (because in time they stablize and have very few problems).
However, the moving people off to the new FC vol, or the original SATA vol, or even the new Netapp we also added to Blingy, just wasn’t happening fast enough. So on April 2nd we bit the bullet and switched Blingy off as the “new customer” cluster and started growing good old “Postal” again. Once we did that, we were finally able to get ahead of the curve and total usage on our first fiber channel volume has been slowly dropping ever since.
We tried at that point to contact the vendor to see if we could just get more drives that WOULD allow us to grow fcvol1, but they said their manufacturers were closed for inventory for a week after the end of the quarter and we couldn’t get anything until Friday, April 11th at the absolute soonest. Later they said they could find us some they could get us by Tuesday, April 7th, and we preliminarily said we’d take them.
This whole time we had a support ticket open with the vendor about the crashes (the OS upgrade didn’t fix it), and finally on April 3rd we received notice that they’d fixed the bug that they believed was causing it! However, the patch still needed to go through their “QA”. Finally, this Sunday April 6th they said it was all ready to be deployed, so last night we did.
What Now
Well, right now, performance is still not great on fcvol1… but mail and web should be pretty much working. One thing we’ve noticed is a website that hasn’t been visited in a long time will have a big lag still upon the first visit.. but then subsequent reloads/visits seem much faster.
At least the total disk usage is coming down now, and hopefully by tomorrow it’ll be below 85% which is supposedly a magic number where performance is fine. We’re going to keep off-loading it until things are great, though. We’ve got plenty of disk space for it, the problem is just it takes so long to move it.
We also I guess will find out tonight if the NFS freezing bug is fixed by this new patch. Hopefully so.
It’s Too Late…
I realize this is probably too little too late for many of you, but I just wanted to sincerely apologize for this whole big Blingy cluster-f*ck. Also, if you’re on Blingy (you can tell from the panel by clicking “account status” and looking at “Your Email Server”, we’d like to offer you a month worth of hosting credit.
To get it, all you need to do is contact support from our panel and make the subject of your message “Blingy Account Credit”. That’s all you have to do, and we’ll credit everybody who asks (and is actually on Blingy!) next Monday (April 14th).
125 Responses to “Another Anatomy”
Categories:
Archives:
- November 2009 (3)
- October 2009 (1)
- September 2009 (3)
- July 2009 (1)
- June 2009 (3)
- May 2009 (3)
- April 2009 (4)
- March 2009 (2)
- February 2009 (3)
- January 2009 (3)
- December 2008 (3)
- November 2008 (4)
- October 2008 (5)
- September 2008 (2)
- August 2008 (5)
- July 2008 (2)
- June 2008 (3)
- May 2008 (6)
- April 2008 (2)
- March 2008 (3)
- February 2008 (5)
- January 2008 (7)
- December 2007 (3)
- November 2007 (3)
- October 2007 (5)
- September 2007 (3)
- August 2007 (4)
- July 2007 (4)
- June 2007 (7)
- May 2007 (3)
- April 2007 (3)
- March 2007 (8)
- February 2007 (4)
- January 2007 (6)
- December 2006 (5)
- November 2006 (2)
- October 2006 (4)
- September 2006 (4)
- August 2006 (5)
- July 2006 (3)
- June 2006 (3)
- May 2006 (5)
- April 2006 (6)
- March 2006 (7)
- February 2006 (6)
- January 2006 (8)
- December 2005 (7)
- November 2005 (5)
- October 2005 (8)
- September 2005 (19)
- August 2005 (11)
- July 2005 (4)
Other Sites
- DreamHost Status - An off-network page for all DreamHost Announcements! (Not as silly as this blog.)
- DreamHost! - The host behind the blog.
- Forum - Discuss DreamHost and web hosting!
- Web Control Panel - Manage your account.
- Wiki - On DreamHost and web hosting.
Powered by WordPress. Pool theme by Borja Fernandez, modified by DreamHost.
Like WordPress? Consider attending WordCamp LA.
Entries and comments feeds.
^Top^
April 7th, 2008 at 12:44 pm
Nice to hear that you guys are on top of it. Blingy IS a big filer, running df shows 13 TB of storage. Incredible! That FC/SATA offloading system sounds pretty cool too.
April 7th, 2008 at 1:21 pm
I sure would like to know who that storage system vendor is.
April 7th, 2008 at 1:26 pm
I am really glad you guys are looking into this issue. I am a new customer, and this experience did leave a bad taste in my mouth. I was afraid I would have to just put up with these issues, but I am glad I made the right choice by going with dreamhost!
April 7th, 2008 at 1:33 pm
@Lorenzo: The vendor is BlueArc.
http://blog.dreamhost.com/2006/08/25/ask-dreamhost-customers/
April 7th, 2008 at 1:58 pm
Thanks for the update Josh – now when are we getting next newsletter?
April 7th, 2008 at 2:04 pm
Seriously, when’s that newsletter coming? I think I’m going to open a support ticket about it.
April 7th, 2008 at 2:06 pm
What about other customers who were affected due to Blingy? I have experienced almost a week of “Network Timeout”s!
April 7th, 2008 at 3:23 pm
A smarter person would have read the post before asking a stupid question.
If you’re on Blingy, you get credit.
If you’re not, you weren’t affected by it.
April 7th, 2008 at 3:32 pm
Damn, I was going to give you some good-natured crap but then you have to go and offer account credits. I can’t even tease you without feeling bad now.
Here’s hoping that the next three months are better than my first three months with y’all.
April 7th, 2008 at 3:45 pm
Why not just GIVE everyone on Blingy a credit automatically. That is the right thing to do.
April 7th, 2008 at 3:53 pm
That’s only the right thing to do if you’re located within Stupidville city limits, but that’s not where their datacenter is.
April 7th, 2008 at 4:14 pm
Do customers on spunky get anything?
April 7th, 2008 at 4:24 pm
Sounds good. My friend was on blingy, I got in before him so I wasn’t. I thought it was odd that his site was always way slower than mine, so he’ll be happy to find out.
April 7th, 2008 at 4:31 pm
Spunky was nothing compared to Blingy. Get real.
April 7th, 2008 at 4:46 pm
So, are you going back to NetApp or will now try someone new again?
Just curious
April 7th, 2008 at 5:40 pm
“and we try to never ever grow old clusters again once they’ve been “shut off” from new signups (because in time they stablize and have very few problems).”
LOL
April 7th, 2008 at 7:27 pm
@16: note “few,” not “none.”
April 7th, 2008 at 8:42 pm
Thank you for your honesty, openness, and professionalism.
Your cock is so big.
April 7th, 2008 at 8:48 pm
@18: Pics or it didn’t happen.
April 7th, 2008 at 10:01 pm
Shall I say it?
Google App Engine anyone?
April 8th, 2008 at 1:17 am
I like the tshirts :D
good to see we’ll be getting credit back and also good to see an explanation
April 8th, 2008 at 3:38 am
Can we get the T-shirt instead of the credit?
April 8th, 2008 at 10:00 am
Is there any chance that all of this filer action has somehow affected spunky? I haven’t been able to access mail on that cluster for the past ~24 hours. I’ve sent in two trouble tickets, but have had no response.
April 8th, 2008 at 10:09 am
I think I was one of the first 25,000 DH customers and stuck with them through a lot. I finally had enough about six months ago and have been slowly moving away. For the last few months, I’ve only used DH for DNS (very reliably, I might add) and thought I’d check this blog out now that I’ve finaly changed all of the DNS information in my domain registrations and am deleting the last of my domains from the DH control panel.
This post gives me no regrets about the decision to move about 250 domains to my own dedicated servers, which running in my office on a T1 are faster and more reliable than DH.
But, one good thing about DH is that they are less expensive than my current backup DNS provider. I’ve love to use DH for secondary DNS if I didn’t have to manually update the information in the control panel. If you guys could just pull my zone files, I’d continue to be DH customer for another 10 years.
April 8th, 2008 at 11:33 am
I want that shirt — talk about turning lemons into ice cold sweet lemonade!
April 8th, 2008 at 11:58 am
@24 (The Captain)
Bingo!
That’s pretty much the future of DH. With Amazon, Google (soon) and others entering the hosting market with a no-oversell-like-hell businness model and offering a very good and reasonably priced service, DH is doomed to play in the minor leagues as a cheap and unreliable service… only suitable for backups and secondary services.
I’m also moving away my domains from Blingy (you guys can smoke the free month, thank you). But I’ll still use my account for svn, DNS and other services.
Regardless of what happens wit Blingy (which I seriously doubt it’ll be running smoothly anytime soon) I woudn’t recommend DH for hosting serious stuff from now on. The rules of the game have changed…
April 8th, 2008 at 2:25 pm
so after this hell of troubles you still sell accounts? and call that ‘Your (us) bad?’
the only good thing in DH, they honor their 97 days money back.
damn it
April 8th, 2008 at 3:23 pm
Yeah.. I’d rather get a t-shirt too.. :-)
April 8th, 2008 at 5:42 pm
What amazes me is that 90% of the people who bitched on the dreamhoststatus blog won’t understand 5% of what’s explained here.
April 8th, 2008 at 5:49 pm
This is a pretty poor attempt at waving off a lot of pretty serious malpractice as a hosting provider. I have since moved on to a new hosting solution because of the constant email and website failures and delays, though I doubt I will ever see any sort of refund (much less the silly tee-shirt). The newsletter was even more pathetic for trying to laugh off the miserable support and failures. (And by the way, why should I have to ‘do’ anything to get the free month? Don’t you owe at least that much for all the screw-ups? Comes off as nothing more than another veiled attempt at sincerity.)
Seriously, you should get out of this business. Or better, someone should force you out for incompetence.
April 8th, 2008 at 5:57 pm
What I can’t understand is why you guys didn’t stop new signups from going to Blingy months ago! I mean, there’s problems with it, you knew this, and you knew that adding additional stress/data to the filer would cause more problems.
However, you guys STILL left the new signups going to Blingy and caused yourselves even more problems because of it… I mean seriously, why don’t we be a little smarter next time?
April 8th, 2008 at 6:18 pm
Umm, wow. Definately interesting reading. I just moved to DH (don’t know if i’m on blingy though, too recent for a refund anyhow, lol). At least the attitude here is much better than the last shared hoster I was at, which would communicate what I guess was their displeasure by doing “chmod 000″ commands at random files in my account and see how long it would take me to notice. Think my home page was down for a couple months…
April 8th, 2008 at 6:27 pm
I’ve got several sites on blingy, DH came highly recommended from a guy I really trust so I didn’t even research it, of coarse it wasn’t helping that DH’s site is a constant sale banner. Well anyway my sites don’t require much bandwidth because we only deal with a few clients, but we definately needed a host company that dealt in hundreds of Gb not Mb, well long story short DH’s flexibilty and space has kept me so far, but if my sites aren’t running smooth on my 95th day I’m gonna be dissapointed but definately have to leave. ALTHOUGH… I think one of those spring break shirts is in order compliments of DH if I stay!!
April 8th, 2008 at 6:34 pm
Waaaaaaaaaaaaaaaaaaaaaaaaaa!
April 8th, 2008 at 6:47 pm
I learned a few things in the week that I fought to keep my sites and email online.
1.) You can host your email with Google (Gmail). For free. It’s even integrated into the Custom MX record menu item in the mail menu.
2.) I made a bad assumption that DH had their stuff on track when I signed up in Jan 2008. I read old posts on DH downtime and slow servers, but I figured they were past that. I get in just in time for the billing SNAFU. Then the blingy SNAFU. I’m sure it’s an innocent glitch, but I can’t roll the dice with my clients’ moneys. You know?
It makes me sad because I really want to stay, but I just can’t do it in good faith. I took out a 256 slice with SliceHost a few days ago, and I’m setting it up to run a handful of low-traffic sites. My Fast-CGI Rails blog won’t suck so bad either because I’m going to front it with Mongrel on my slice.
April 8th, 2008 at 6:56 pm
To offer the amount of services the Dreamhost does at their low prices, they probably can’t afford to have much extra capacity lying around. Unfortunately, that means that when there is trouble they don’t have extra servers to pick up the slack. It would be nice if they could get their hardware a few months ahead of demand, but overall they still do a good job.
April 8th, 2008 at 6:58 pm
How about concentrating on correcting the problems and spending less time on cute newsletters and smart ass comments. I’m losing face with my clients, haven’t got a single reply to the ticket I opened and going to leave your service very soon (and demand a FULL refund).
What a joke
April 8th, 2008 at 7:15 pm
I’m still there :P I’m not paying my bill this month unless you want me to? :P However I’m still hanging in there, I have many other server and if you are down or too slow it just goes on another server (redundancy is the key…)
However do I need to pay my bill? LOL
April 8th, 2008 at 7:15 pm
Guy Angelo grow the f@#k up dude and stop demanding shit, you don’t deserve nothing like that.
April 8th, 2008 at 8:21 pm
Angelo — “Losing face with clients”, you are dumb enough to host your clients site on ANY economy hosting solution? Ever see a 99.9% guarantee here?
Yeesh.
April 8th, 2008 at 9:05 pm
I certainly am not surprise by all this shit. I have numerous problems over the last year. If anyone had paid any attention to the number of complaints coming from Blingy customers something would have been done last year…………
April 8th, 2008 at 10:18 pm
I would like to say that there were 2 problems here.
1. was the failure of blingy, which, although absurd in its proportions, was in some ways understandable as it was a combination of human and machine error.
2. The complete failure of adequate information via tickets, support and status updates for 3 weeks. I was out of (and still lack) stable email. I lost a client. I didn’t switch because I was left in the dark hoping. But as a strategy, this is pure human and DH error. I have use DH for 6 years, and I have never seen such poor service or response.
Truth be told, the second failure is much more offensive to me, and I can only say it deeply saddens me that a host such as DH, which is relatively cool, could not handle itself at least a little better. If I had had 1/10th the amount of information Josh just posted, I would have been much more patient and understanding.
Please consider this failure of communication both as machine and DH error, and hire more support staff.
nron
April 8th, 2008 at 10:35 pm
I appreciate that you are so open about this episode. However, the most important thing will be to learn from this horrendous experience.
You can’t put so much blame on the vendor. After all, vendors will be vendors! All pushing their product, offering guarantees and great service….just like you guys to us !!!!
Yes you need to remain price competitive, but not at the expense of losing customers!
One suggestion for you is to have different levels of service – bronze, silver, gold or standard / preium, etc…. Bronze would be your current shaky offering, or specifically certain clusters. Silver or premium would then be a less risky setup with more stability. For that you could charge an extra buck or two per month, or whatever.
Good luck…but please don’t leave it to luck, do more thinking as a team. Maybe you should also set up a user group to help with say 10-20 representative users – by representative I mean not just nerds but real business people?!! :o)
April 8th, 2008 at 10:49 pm
Still not fixed. 12 s for a single web page. 27 s for a simple perl script. This was now. When is it fixed?
April 8th, 2008 at 10:51 pm
I like how everyone gives pats on the back and laughs like we’re all friends. I don’t know any of you guys, I don’t know anyone at DreamHost, and they’re a business. They provide a service rather unreliably and should take some responsibility for their product. I could really give two snaps about what the problem with “Blingy” was, but I do care about why it took weeks on end to fix it. If you can’t solve your own tech problems then contract it out – this isn’t the Stone Age and things shouldn’t take more than 24 hours to be fixed. If its that bad throw the servers out and get new ones the same day. Its called a ‘business expense’ to those running serious businesses.
April 8th, 2008 at 10:52 pm
I love dreamhost with all of my heart. You guys need to understand that generally you get what you pay for and dh by no means should be deployed for a mission critical application, but for personal blogs and other web fun. If you are planning on hosting digg on a any shared services, you may want to reconsider. But for everything else it gets the job done cheap!
April 8th, 2008 at 11:28 pm
DH couldn’t organize a fuck in a whorehouse or a piss-up in a brewery.
DH tres suxor – dey is in yur servers fuckin up yur shit.
April 9th, 2008 at 12:00 am
this is why you need a PROPER storage vendor… IBM/NetApp for everything.. eggs in a basket i know, but they do know their stuff.. I do know that EMC, have and do actually page on call engineering teams (the guys who actually CODE the guts of the various arrays they sell) no matter what time of the day it is, to get a workaround/fix implemented. microcode can be modified ‘on the fly’ to change values and non-user settings to alleviate performance issues if the code is misbehaving, till a permanent fix is released…. I’ve been there, done that, got the boxers with the skid marks due to threats coming from upper management if the problem isn’t fixed in a timely fashion.
April 9th, 2008 at 12:17 am
Instead of giving credit, you could speed us up by lowering the maximum number of customers on blingy and dividing what is left over among the users.
Right now we are getting more time on a slow server rather than a faster server!
Also I do not think that blaming the troubles on the vendor is completely justified here. As the sysadmin you should have known better. Stop apologizing and give us something real!
April 9th, 2008 at 12:36 am
Been with DH for nearly 2 years, lovin it all the way! I’m on the Barqs server and I’ve had like 6 hours down time that I’ve actually noticed in nearly 2 years.
LOL’ing @ all you bitchers, whing whing whing “I’m losing face with my customers” bitch bitch bitch “I’ve no email in 24hours” If your losing face with your customers, try running your business properly then…
To Josh and crew, keep going guys, My crew & I are with you all the way.
April 9th, 2008 at 1:35 am
Josh’s recent newsletter led me here. Reading through, T1 has just gotta be Josh since his counterpoints in this thread are as childish as Josh’s newsletters. My stuff is not mission critical but why should my stuff be less important than anyone else’s? Owners of a particular brand of equipment rely on my site to do their ‘thing’. I’m relatively impressed by the techno-crap outlined earlier but, at the end of business day, who gives a rat’s ass if it’s simply not working? Am I supposed to discern, somehow, that DH is not top-tier? Is subject to economy issues? I absolutely KNOW I can pay more for hosting but to what end? I’d be more happy with DH if the newsletters were less silly in nature … there’s a thought
April 9th, 2008 at 1:42 am
Maybe all us blingies should send hate mail to B_l_u_e_A_r_c telling them how much we hate them ;)
Isaac
April 9th, 2008 at 2:30 am
Dear admin,
I´m a new customer, it was a hard time for me! More than 50 clients calling me all days, but, I persist and I haven’t jumped ship.
I was a Network System Analist, and some time ago, I lost a Novell Server and make me 5 days and 5 nights without sleep abd more than 15 days to finish the job. I did in yours shoes!
Nice job!!!
Send me a t-shit BLINGY SPRING BREAK 2008 :-)
Last go to work!
Eduardo
From Brazil
April 9th, 2008 at 3:14 am
Things are indeed running MUCH better, but there’s still room for improvement. This meltdown happened during my first 50 days with DH, and I came pretty close to getting my refund and walking, but a few things kept me hanging on:
1: I’m not using my sites for work at the moment
2: I can tell by your web panel that you really do care about the customers’ experience, and that you are in touch with what we want,and
3: I’m hoping that you get everything up and running and that Blingy eventually ’stabilizes’. I’ve gone through 4 or 5 low cost hosts over the years, and I like it here the most so far… and given that my sites haven’t been working most of the time I’ve been here, that must mean that you’re doing SOMETHING right that will pay off in the future.
Shell access + very nice panel + very nice server side installs and upgrades + up-to-date versions of php, mysql, etc. + RELIABILITY & SPEED = a very happy customer.
April 9th, 2008 at 3:29 am
Hi,
It will great if some one let me know how much time it will take more to fix this
Thanks
Davi
April 9th, 2008 at 3:38 am
What you should have been doing was COMMUNICATING with us. There were no email status updates, and your stupid posting board was filled with pranksters putting up rick-rolling links.
You lost me as a customer. Bye bye.
April 9th, 2008 at 4:54 am
“up-to-date versions of php, mysql, etc.” — amen. At WebHost4Life, they wanted to charge me to move me to a server with a working stats program. After all, they couldn’t fix it, because “someone might be relying on the current behavior” *rolls eyes*
April 9th, 2008 at 4:57 am
I was really hoping “definitely in the top 5.” was a link that would take me to a list of what you guys think the big 5 have been. Might make an interesting thing to post once every few years.
April 9th, 2008 at 4:58 am
One thing it prove. This hosting is a moron. Even a FOOLED by vendor.
April 9th, 2008 at 5:12 am
Meh, where I work 13TB is child’s play.. For one of our apps we have like 2 Petabytes configured via SAM/QFS, all FC-AL.. The tape library has like 16 drives and 4096 slots, it’s about 6′ wide, 8′ high and 9′ deep…
I’ll be requesting that month shortly, though frankly you should just add it to EVERYONE who’s on that cluster automatically..
April 9th, 2008 at 5:17 am
Thank you DH for being open about the issues – can’t say I know another host who would be so honest!
Been on for 8 years here, and won’t be leaving any time soon. Sure, there have been issues here and there, and one of my sites was down from Blingy – but the thing that keeps me here is the support. Not sure who you guys are that “never get responses to tix” as you claim.. but I’ve never had an issue getting the answers I need.
If you “hate” DH – please, for the rest of us, leave. Do you shop in stores you hate? Do you wear clothes you hate? Read books you hate? Seriously.. grow up.
April 9th, 2008 at 5:46 am
Wow,
I am still on your email list, but you never returned any of my or my web designers concerns from way back in Dec 2007, Jan 2008, Feb, March.
You cancelled mywebsite,www.kevintheplumber.com, and now I am left with 4 trucks with hand painted gold leaf lettering costing well over $4000.00.
I contacted you about billing me 3 times. In late Jan, early Feb, you dlb billed me and several others from your email to us. You sent out a form letter appoligizing about this double billing. Next my website is down, and then gone.
After several phone calls, emails to your company about this issue, My website was gone!
I had all 3 billings removed after my web page went inactive in March.
GET ME MY WEBSITE BACK!
http://WWW.KEVINTHEPLUMBER.COM
How could of avoided any of this from happening. It was all done from your end.
F-in pissed
Kevin Holloway
F-IN PISSED OFF!!!
April 9th, 2008 at 6:19 am
“the techs at the vendor couldn’t figure out what was causing the NFS freezing”
I hope you don’t plan on using this vendor again.
April 9th, 2008 at 6:20 am
Let me preface this with: I am no longer a Dreamhost customer.
1 — I logged enough tickets in February to know that there was something wrong back then. My request to be moved to a different “filer” was turned down. If I’d have known the full story I would have been more insistent. Instead they blamed my site performance on Drupal (!) .
2 — I am astounded to know that you continued to add new accounts to Blingy as late as April. I would have been astounded to hear that you continued to add new accounts to that piece ‘o crap as late as March 10th.
April 9th, 2008 at 6:23 am
Oh, and I almost forgot point number 3:
There have been 4 other incidents just as bad as this one? Wow. Oh well. My sites have been moved. So long.
April 9th, 2008 at 6:39 am
It’s good to know everything is back up to speed.
April 9th, 2008 at 6:58 am
We’re Lovin it!
Thanks Dreamhost for the credit.
x
April 9th, 2008 at 7:12 am
Not sure why I’m posting this, but after reading through the posts…I felt I had to.
I worked for a different hosting company when I signed up for my DH account – for only one reason – price. I was lucky enough that my company had phone support – so I was yelled at when stuff like this happens – not posted to.
I’ve been with DH for 2 and half years now – and I’ll never change.
It’s the INTERNET people, and we’re dealing with computers – things break, things don’t work right, and things happen. There is no way to have things 100% ALL THE TIME – and until the public realises this, these blog postings will be my morning entertainment.
If you’re going to be someone who wants perfection, host your own stuff – then you can only bitch to yourself when your servers go down, or your internet connection fails, you experience a DOS attack, and so on….
April 9th, 2008 at 9:23 am
If everything is better now then why is the load on my server (Pan, apparently part of “Janky” cluster) still showing a 400+ load on a regular basis? I understand the implications of using a shared server, but a 400+ load day after day is an indicator of an overloaded server.
We also specifically asked months ago to be moved to a server that wasn’t overloaded, and we were denied. My projects (and several employee’s jobs) are now at risk because our site’s core audience is eroding due to the frequent outages. All we run is a phpBB3 forum, nothing complicated.
It’s a very good thing that we moved our mission-critical Rails apps to a sturdier provider months ago or else I would have just lost a big contract through no fault of my own. I’ve been with Dreamhost since 1998 and I’m willing to cut you guys some slack out of loyalty, but PLEASE get these servers running again. It’s just embarrassing to explain to people in meetings that we’re loyal to a web host that can’t handle powering a phpBB3 forum.
April 9th, 2008 at 10:03 am
Wow, you’re the first person that has ever accused someone that doesn’t cry along with your stupidity of being an employee.
Seriously, where did you ever get such an original idea?
Well, since your opinion is different than mine, and that makes me Josh, then it must be safe to assume that you’re the opposite, making you one of the dirtbag criminals that won’t return part of the $170,000 over-refunds, right?
Or don’t you like braindead lack of logic when you’re not the moron throwing it around?
Top tier compared to what? Did you think a $5 shared account was the same as $10,000/month in redundant fully managed dedicated servers spread throughout multiple data centers?
Anyone whose site relies on a shared hosting account agrees to downtime, regardless of host. None of them let you pick when it happens or for how long.
April 9th, 2008 at 10:14 am
Or is it a sign of someone not knowing how meaningless that number is with NFS, when it’s artificially inflated by pending processes?
April 9th, 2008 at 10:25 am
So many people are posting about how horrible DH is and offering “business advice”. My question is how many of you people actually have any real basis to offer and kind of advice to anyone. DH offers a lot to its customers for the money. It almost seems as if half the people posting stuff here have never made a mistake, never though something would work based on their research and been totally wrong and above all never had a computer problem. Things break and people make mistakes, it’s part of life. And all those people “losing clients” over this debacle are really in no place to offer business advice or complain about DH’s customer service. If any of you had half a brain cell you would communicate with your customers using a back up method.
To the DH crew: I hope all goes well with the repairs and fixes. Myself and two friends host our sites with you and have had our own fair share of server issues. (Pre-DH) We all know how frustrating all this can be and I think I speak for all of us when I say we understand the problems taking place and plan to remain understanding as they are fixed.
April 9th, 2008 at 10:56 am
T1, if you think that’s the first time that somebody has responded to my complaints about the overloaded servers by insulting my intelligence rather than fixing the servers, then you’re sadly mistaken.
That’s really my biggest issue here. Stop calling me stupid and fix the servers. T1, I don’t know if you speak for Dreamhost or not, but I get pretty much the same response from the Dreamhost techs through the web panel. They either blame my site and then don’t look at my evidence to the contrary, or they send me a form response. I’ve got a number of different clients set up with Dreamhost accounts. When almost all of my clients’ sites stutter or go down for days I feel like I deserve more of an explanation than the run-around that I’ve been getting.
So once again, and I’m going to ignore any T1 insults this time: does this explanation about problems on Blingy also pertain to Janky? All I want to know is: what is the status on fixing the server that my site is on? Simple question. I’ve been with Dreamhost for 10 years but I’m not going to just hang around here forever if all I get in response to a days-long problem is form letters and blame-shifting and insults. I’m being very tolerant and understanding but one way or another I’ve got a business to launch. I need a working server.
April 9th, 2008 at 11:36 am
Maybe that will stop happening when you stop being stupid?
Why not do both? A broken server is no reason to stop calling stupid people stupid.
The 400 load doesn’t mean anything is broken. Go read up on NFS instead of just randomly throwing out numbers that you think favor your side of the story.
That’s because you’re not too bright. Is there anything you do know?
You’d know exactly who you were talking to if you used the appropriate channels to discuss support issues. This blog isn’t one of them.
You certainly haven’t provided any evidence of it not just your site. Again, the load number means nothing. They know that.
If it was a problem on their end, it wouldn’t just affect you. Chances are (very good) that you’re not communicating the problem clearly enough to them, but instead just dwell on what you think you know about how things work.
Is that your idea of a serious question? What do you think? Do you think that it does, but they didn’t know how to add “and Janky” to the description?
Use your head.
And yet you’re totally clueless about how anything works.
April 9th, 2008 at 11:39 am
You mean having the ability to cry in blog comments doesn’t qualify people to give business advice?
Well, in that case, I’d say the answer to your question is: 0
Even the ones that claim to have experience in a related field usually go on to say something so far off that it shows they don’t.
April 9th, 2008 at 12:39 pm
Dallas, letting this guy do your PR for you is not a good idea. The “appropriate channels” are useless T1, as is everybody’s complaint. I get nothing from tech support and some blog troll is the only guy with an opinion, about a mission-critical server. I have no idea what to tell my client. That’s just great.
I could give a rat’s ass about how NFS works because I’m busy flying around the country starting a business, I pay somebody else to make the servers go. All I want to know is whether this cute story has anything to do with my failing web sites or not. Cute pictures, but does this pertain to me or not? I’m not on Blingy but my sites are failing. When the server’s load spikes, the web site’s don’t go. That happens every day. Is that a simple enough explanation? Is Janky going to get better, or are my sites operating as well as Dreamhost will ever be capable?
April 9th, 2008 at 12:53 pm
Are you getting dumber by the minute? This is a blog that is open to the public, moron. People that don’t cry along with losers like you aren’t PR reps.
April 9th, 2008 at 1:53 pm
LOL RYAN IS AN IDIOT.
April 9th, 2008 at 2:01 pm
Gee thanks guys, I’m real impressed with you too. Those of you who are calling me an idiot, I’m curiou: Are you calling me an idiot for choosing to host sites at Dreamhost? For expecting them to work? For expecting a useful response from tech support after all of my clients’ sites have stuttered for days? For looking for something more substantial than cute pictures in the only information that I’m getting from my web host? Which of those things makes me an idiot? I’m feeling like an idiot for all of them right now.
April 9th, 2008 at 2:56 pm
IDIOT.
April 9th, 2008 at 5:44 pm
You make people monitor your blog and send you email to get the one-month credit? What a slap in the face to your customers.
April 9th, 2008 at 5:46 pm
Pfft…
I ain’t be readin’ this, but I’m special cuz I gots FOUR letters up in my user name.
April 9th, 2008 at 6:04 pm
This must be your first day on the internet, judging by how clueless you are.
First, there is no SLA, which means not a single person was entitled to anything. You like the sound of that better?
Second of all, you don’t even have a basic idea of how it works even where is an SLA.
Most want you to show proof with logs and fill out a claim, which then pends approval. They also don’t tell you when it takes place. You have to catch it, prove it, then hope for the best. They make you jump through hoops for pennies, hoping you won’t do it.
And you call Dreamhost giving out free money a slap in the face for customers, compared to that? Next time you get the urge to whine just because everyone else is doing it, at least pull your head out of your ass and put a little thought into it first.
April 9th, 2008 at 8:49 pm
Seriously, you need to get it together. The problem seems to be Q/A procedure. You shouldn’t hook live systems up to new hardware without first running the hardware through some tests, including stress tests, which should have caught this. I’ve stuck with you for a very long time (5 years now!), but the recent problems have been trying.
I know Dreamhost is cheap, but I don’t think that’s an excuse. There are other cheap hosting companies that manage not to make these sorts of mistakes.
April 9th, 2008 at 8:58 pm
“I’d be more happy with DH if the newsletters were less silly in nature”
Yeah, I agree. In general communication is lacking. There doesn’t seem to be much correlation between status info, blog, etc. and what is going on at the moment, as far as I can tell.
THEN when we finally get a newsletter where (one would assume) DH has a chance to give the big picture, it’s so silly I can’t hardly read it.
At this point I’m just getting going with my web stuff, so can put up with a few hiccups, but please make the newsletters more informative and straightforward.
In particular, use them to tell us how you’re continually improving! :)
April 9th, 2008 at 10:03 pm
People complaining in blog comments think the newsletters are silly? That’s funny. Josh should definitely grow up and be more like the super-mature comment complainers.
Reading the “Unsubscribe” link in the newsletter is almost as hard as clicking on it.
April 9th, 2008 at 10:06 pm
People complaining in blog comments think the newsletters are silly? That’s funny! Josh should definitely grow up and be more like the super-mature comment complainers.
Reading the “Unsubscribe” link in the newsletter is almost as hard as clicking on it.
April 10th, 2008 at 12:46 am
@68 Maggie
What are you? a 9 year old girl? “things break, things don’t work right, if it rains you’ll get wet…” Jesus f*cking Christ! Of course we know sh*t happens, of course we know DH won’t deliver 99,999% uptime for 8$/month… but what is absolutely NOT ADMISSIBLE is having your website down for a WHOLE F*CKIN’ MONTH.
Again: My sites have been NOT WORKING during the lasts 4 weeks. Do you understand that kid?
April 10th, 2008 at 1:47 am
Dan, do you feel better now that you cried like a little bitch?
April 10th, 2008 at 4:39 am
> Dan, do you feel better now that you cried like a little bitch?
Actually yes. Anal raping your mother also helps a lot…
April 10th, 2008 at 8:16 am
Well, reading this blog is definitely interesting, although less interesting than it was yesterday. I see people coming in here trying to find a forum to try to air grievances that they don’t believe they have other outlets to show, and run into people telling them to get out and others trying to make them scream louder.
DH has a heck of a lot of diagnostic tools on their site, more than any other that I’ve seen (connueries?!?), so if there is something wrong with your site it looks like it would be very good to track down the source of it. While I do recognize that DH has been having technical difficulties, is a shared hoster with limited abilities, and is probably overworked (hey, I get tossed “make it work” questions all day) I do also recognize that a site being down for weeks is also not a good thing and probably should at least have an explanation.
And for those that want closure by posting here, unfortunately I’m gonna say that you probably aren’t gonna find what you’re looking for. Most likely you’ll just end up attracting more lollercoasters.
April 10th, 2008 at 9:31 am
Awww, did I make the little 12 year old pull a mommy joke out of his clever arsenal? I think I see some tears.
I’m sure rape is the only way you’ll ever lose your virginity and it’s doubtful that the victim will be a female.
April 10th, 2008 at 7:01 pm
I really dont think it’s funny. It’s a bad decision bordering on being fradulent to continue to accept new users and shoved them to blingy after the problem surfaced.
I signed up 2 weeks’ ago. I dont have a busy site but Internal Server Errors were so frequent that I can’t avoid them even in the short time between opening my wp control panel, write a short post and publishi it. And it always happened after I clicked on ‘publish’ – everything I wrote was gone.
April 11th, 2008 at 3:47 am
You know, I kind of feel like some of these comments should be pruned for being unbelievably far off-topic, if not because they lack even the wit of a middle school child’ barbs.
Would it be possible to give more detail, though? What tests were run that you felt it safe to go live?
April 11th, 2008 at 7:06 am
Peon: Let’s give everybody on the cluster a month free credit.
Honcho: EVERYBODY? No, no! Let’s put it in places most people don’t read — like My Awesome Blog Posts! That way, even if half the people hear about it and all of those half apply, we’re still 50% better off.
Peon: Ah, I see! Wow, you’re smart!
Honcho: I know. That’s how I have ALL THIS FABULOUS MONEY. Oh, and you’re fired.
Peon: ?
Honcho: Before you go, get my yacht dealer on the line.
And that’s how decisions are made at DreamHost.
April 11th, 2008 at 8:10 am
I don’t think anyone at Dreamhost has a yacht after they sent out those refunds for the billing issue. Ouch.
April 11th, 2008 at 9:31 am
You guys should know that this newsletter was marked by Dreamhost’s own spam system as spam! Lucky I saw it as I am a (severely affected) Blingy customer and one month’s repayment is the least I would expect.
Peace
April 11th, 2008 at 9:40 am
#91 has no idea of how anything works–no surprise. Give reading a try. Comment #80 would be a good start.
Like it or not, they don’t owe any of those customers anything. That being the case, they made the money WAY easier to collect than any host with an SLA out there.
#92 Give the comment spamming a break already. Why would anyone want to go with a scumbag host like you that stoops to spamming? Trying to keyword spam a host for “SEO” is pretty gay anyway.
April 11th, 2008 at 9:43 am
#91 has no idea of how anything works–no surprise. Give reading a try. Comment #80 would be a good start.
Like it or not, they don’t owe any of those customers anything. That being the case, they made the money WAY easier to collect than any host with an SLA out there.
#92 Give the comment spamming a break already. Why would anyone want to go with a scumbag host like you that stoops to spamming? Trying to keyword spam a host for “SEO” is pretty gay anyway.
April 11th, 2008 at 9:53 am
RE: “That being the case, they made the money WAY easier to collect than any host with an SLA out there.”
Very true! Hosts with an SLA are the ones that make things hard. They don’t announce downtime for the world to see like dreamhost does. Not only do you have to catch the downtime, but it’s up to you to prove it with log files, even though they already know it happened! Then you have to file the claim with log proof and wait for them to approve it.
April 11th, 2008 at 12:43 pm
At these prices, what we get is amazingly good. You can pay 10 times the price and get nothing better, plus the support staff will either treat you like a piece of spit or ignore you completely
April 11th, 2008 at 1:24 pm
Knightrous, your comments are pretty rude & insensitive. I hope that you run your business properly if you have time to come on here & swear at people who are unhappy about their service not working.
Good to see so many other people on Spunky posting here with problems! It lets me know it’s not just me :)
April 12th, 2008 at 1:07 pm
Hi guys,
nobody got fired for choosing EMC!
Why haven’t you choosen them?
BR,
Fernando
April 13th, 2008 at 7:23 pm
OMG it’s been three weeks! THREE FUCKING WEEKS!!!! and they STILL haven’t sorted it out??? What complete RETARDS!!! How about you try to..
READ . THE . FUCKING . MANUAL !!!!!!!
Never heard of such incompetence. And then there’s idiots like Knightrous who admire DH for fucking it up??? It’s a fucking embarrassment to be associated with DH these days. Knightrous, suck my dick you fucking faggot – you seem to like doing that.
April 13th, 2008 at 7:26 pm
HELLO MY NAME IS JOSH. SORRY BUT I CAN”T HEAR YOUR COMPLAINTS OVER THE SOUND OF HOW AWESOME I AM.
April 13th, 2008 at 11:30 pm
What’s with all the “there’s no SLA” and and “read the ToS” because “they don’t owe anyone anything” type comments when even their own ToS say “DreamHost Web Hosting expressly limits its damages to the Customer for any non-accessibility time or other down time to the pro-rata monthly charge during the system unavailability.” Their service was basically unusable for some people for a month, so those people get a month credit, and that’s just the way it goes, no better but no worse either. As for putting it in their blog, well, I would certainly hope anyone affected by this has started to follow the situation by now, so I don’t think it’s fair to act as though it’s hidden somewhere those people will never see it. Hopefully if anything both DreamHost and its customers have learned a lesson about counting too much on things which can break…
April 14th, 2008 at 4:48 am
typo on the Exxon pic:
“nothign” vs “nothing”
April 14th, 2008 at 11:36 am
The “Fat Finger” bugged me, even though it didn’t impact me (other than a scary email). Figured it was a one-time deal.
The “Blingy Cluster-Fuck” didn’t impact me, since I’m on raplhie. But it sounds like a disaster for people in that cluster.
My service personally has been pretty good — and I may even still recommend DH to users with minor needs. But I’m left wondering — when will it me my time in the DH barrel? And how much will it cost me?
April 15th, 2008 at 7:59 am
prodigal_chicken, thanks for reminding us, we almost overlooked you. Your time is coming, and it will cost you plenty.
hugs & kisses,
The happy DH I’m-gonna-bend-you-over-and-do-you-like-a-freight-train team.
April 15th, 2008 at 8:06 am
To all you retarded DH worshipers, you are a bunch of utter morons – complete ass-munching fuckheads. Go fuck yourselves with a large badger.
April 15th, 2008 at 1:35 pm
Wow…some extremely distasteful comments, someone needs to put an ‘adults only’ disclaimer in this comments section. What is even harder to believe than some of these comments is that any person with mission critical site(s) claiming to have been down for a month (even for over a full 24-hour period – are still here. You can purchase additional hosting elsewhere (I have shared hosting here and somewhere else with my own ‘cheap’ failover.) You should have moved your sites after the first day of being down…you can add yourself to blame for your own lack of proactiveness and poor decision making. I am not on the blingy cluster (although 2 of my clients are and didn’t notice anything wrong, or they didn’t call me.) Myself and other clients reside on the randy, postal, frisky and spunky clusters. That said, if my cluster went down, i’d be hosting elsewhere and my site(s) would be up. Simply unbelievable!!!
April 15th, 2008 at 2:00 pm
thanks a lot..
April 15th, 2008 at 4:24 pm
I just got my three months of hosting credit :)
For $29.85; I can forgive them. In fact I would’ve forgiven them even if they hadn’t reimbursed me.
These guys are great!
April 15th, 2008 at 10:11 pm
@#105 – DH Refugee
Hey! Dude! Man, you are on the wrong site! You’d have a lot more fun with all the other folks over on:
http://fuckdreamhost.com – “It’s not a *real* rant unless it’s on *here*”
April 16th, 2008 at 12:59 am
I still can’t see my credit on my panel :(
April 16th, 2008 at 12:59 am
I still can’t see my credit on my panel :(
April 16th, 2008 at 4:46 pm
No less intelligent than the other whiners. I wonder why no one takes you seriously?
April 16th, 2008 at 4:49 pm
well – wtf – when is going to be fixed already
April 17th, 2008 at 8:41 pm
I was way ahead of you on the direct government idea. Like Captain Kirk in the Khan episode, I don’t believe in the leader principle, I just vote for the Propositions.
Representative democracy was started because of the problem of logistics.
You have hosted the above domain for years: http://www.direct-congress.riftmaiden.com/
I never told people about it, because I was afraid things would get too chaotic.
April 20th, 2008 at 5:17 pm
I’ve been a customer since 2002 and I don’t think that will change at all. I tried other hosting services and really DH is the best. And I’ve been around… lol…
April 21st, 2008 at 12:21 pm
Less words, more Homer.
April 22nd, 2008 at 7:56 am
Hello.
Cool the first Picture.
And like you say, All the eggs in the same basket never has been the best idea.
June 1st, 2008 at 1:09 am
I sure would like to know who that storage system vendor is.
September 22nd, 2008 at 4:48 am
So now I know I gave a good advice to my friend to leave blingy. @123: ey, turkey, don´t doublepost around here, the answer on your copy-pasted comment has been already given:
# Chris Benard Says:
April 7th, 2008 at 1:33 pm
@Lorenzo: The vendor is BlueArc.
http://blog.dreamhost.com/2006/08/25/ask-dreamhost-customers/
January 22nd, 2009 at 4:38 pm
lavidjio