You may or may not have noticed that yesterday, www.dreamhost.com was offline and unreachable for the better part of 6 hours. We can’t let something like that go without an explanation.
I should note that during this time no customer sites were affected (other than one – which I’ll get to) and the main “www.dreamhost.com” domain. Customer sites were up, our web panel was up, everything was up…including the ire of some tech-savvy Muslims!
We’ve got a fairly liberal free-speech policy here which we’re quite proud of. Speech that is protected by the United States Constitution’s First Amendment is protected by DreamHost. While we don’t always agree with the content of the sites we host, we do support their right to host it in America!
Yesterday was Draw Mohammad Day.
This did not sit well with roughly 21% of the world’s population.
We happened to be hosting drawmuhammadday.com, a site that encouraged people to draw images of Mohammed. That’s kind of a no-no in the Muslim world.
Incidentally, did you know there’s like a million different ways to spell Mohammed?
In the spirit of yesterday’s event, but without the offensive parts, I’ve drawn some pictures to show you what you might have missed!
Some people weren’t too keen on the idea of the Draw Mohammad Day website and suddenly we were the target of the largest Distributed Denial of Service attack (DDoS) we’ve ever seen. drawmuhammadday.com was the first to fall. It was the main target and it didn’t take long…based on our stats it looked like almost the entire country of Pakistan was attacking us! Well not really. But nobody in Pakistan could reach YouTube, Facebook, or Twitter yesterday, so what else were they gonna do?
These weren’t just random attacks from here and there. We saw several Pakistani groups targeting us on their blogs, often providing step-by-step directions and automated tools for launching e-assaults on dreamhost.com and drawmuhammadday.com.
They did not let up once the site was down. At one point dreamhost.com (the site itself) was handling around 20,000 requests per second. To put that number in perspective, when our customers’ sites have traffic surges a busy day might see that number get up to ten or even twenty.
Our load balancers, as great as they are, typically handle about 4,000 connections at any given moment. During the attack they made it up to 400,000 before they seized up and crapped out. We believe that even the most top-shelf battle-hardened load balancing options would not have been able to withstand an attack of this scale – a quick jump in traffic about 100x larger than normal traffic patterns we see on any given day.
Our fault-tolerant setup relied on those load balancers and they proved to be our undoing. Luckily only some services were affected by this for a very short time (webmail being one of them) before we got them going again a few minutes later.
To restore services we had to take the site down altogether while we moved it to newer, stronger hardware, beyond the reach of our load balancers. We tuned the Linux kernel on this new machine aggressively to use less memory for TCP connections. We also abandoned Apache, favoring a specialized nginx installation.
When we flipped the switch to get dreamhost.com up and running again at around 2PM PDT, the attack load had dropped to 130,000 simultaneous connections with over 20,000 requests per second. The new setup took it like a champ and continues to perform well today – even while we’re still seeing elevated traffic as a result of lingering attacks.
We’re proud to say (and repeat!) that customer sites were not affected and our control panel was still reachable during this entire debacle. And of course if you ever suspect server problems with your DreamHost account be sure to check dreamhoststatus.com!
We learned some lessons yesterday and, moving forward, we’re going to put them into practice. Thanks for hangin’ in there.
(not Mohammed)
May 21st, 2010 at 1:42 pm
I am proud to have you guys as my web host. Thank you for standing up for your customers, and their right to speak openly and freely.
May 21st, 2010 at 1:43 pm
I’ve hosted with you guys for 6+ years now. Thanks for helping to keep the internet free and open.
May 21st, 2010 at 1:47 pm
Fuck mohammed, that piece of shit.
May 21st, 2010 at 1:49 pm
:()-|-<
(above, I have depicted mohammed. you will note the lack of genitals.)
May 21st, 2010 at 1:53 pm
Two quotes come to mind.
“I disapprove of what you say, but I will defend to the death your right to say it.” – Evelyn Beatrice Hall, paraphrasing Voltaire.
“If you strike me down, I shall become more powerful than you could possibly imagine.” – Obi-wan Kenobi, taunting Vader.
May 21st, 2010 at 1:54 pm
Well, first– “Robot Lord” — that’s completely disrespectful and no better than anyone who launched an attack on the servers, and personally I don’t want you to be a representative of “free thinking” in the West.
Second–thanks for the heads up, Dhost! I have hosted with you for the past 10 years and you’ve always been the best hosting service in the biz.
May 21st, 2010 at 1:57 pm
Come on Mr.Robot Lord :S
what If I insulted your prophet (if u have) cuz on like u don’t know GOD or any respect for religious :S , people in all the world treat Muslims like animals ! and they say Muslims are terrorists ! why do you think that ? because of the Hatred which people made in them ! so stop insulting and get back to work! finally you are just a Robot :D which means you don’t know what to do in too many cases ! just like this ;)
May 21st, 2010 at 2:01 pm
Once again, top performance! Well done guys.
May 21st, 2010 at 2:08 pm
Hey, 21% is what I’m guessing you’re calling the entire muslim population, and that’s not correct. At least, according to one muslim redditor.
http://www.reddit.com/r/pics/comments/c6r2i/dear_infidels_of_the_world/
May 21st, 2010 at 2:10 pm
Great job Dreamhost! Thanks for the heads-up and for being a great and transparent company for all these years!
May 21st, 2010 at 2:11 pm
Durka Durka?
May 21st, 2010 at 2:26 pm
@Kate #6, what about the word Tokyo in my name makes you think I’m “a representative of “free thinking” in the West”? You are dumb enough to be a muslim. Go climb a mohammed.
May 21st, 2010 at 2:43 pm
Proud to be a happy dreamhost customer.
Strongly representing freedom of speech: +Infinity
May 21st, 2010 at 3:30 pm
> and they say Muslims are terrorists ! why do you think that ?
> because of the Hatred which people made in them
Er, no. People say they’re terrorists because right now they are attacking our favourite hosting company over a customer’s joke :-|
Seriously, I know that 99% of muslims are fine (just like 99% of christians and atheists are fine), but you can’t just claim the other 1% don’t exist, *especially* as a reply to a blog post saying “we are being attacked by the crazy 1% right now”…
May 21st, 2010 at 3:31 pm
Thanks for trying to stick it as long a you could guys. I’m really sorry it caused you so much downtime. Any chance you’d give me a nice powerful stand-alone server in support of free speech?
May 21st, 2010 at 3:35 pm
Well Well Well
Dreamhost, I have had problems with you in the past, things crashing that sort of thing.. then you go and do something like this and it makes me want to keep being a customer…. good job, oh yeah, how many mohammads does it take to change a lightbulb…
May 21st, 2010 at 3:58 pm
Or better yet, please don’t sue me! :)
May 21st, 2010 at 4:24 pm
I’m really proud to be hosted under Dreamhost, thanks for caring about free speech.
May 21st, 2010 at 4:24 pm
Do you mean 10 or 20 requests per second just for dreamhost.com ?
May 21st, 2010 at 4:32 pm
Can we have any pretty graphs of the traffic?
I can understand their feelings about the not wanting people to draw muhammad, but attacking a website is no way to go about solving the issue.
May 21st, 2010 at 4:34 pm
We want ngnix for our sites too! :D
May 21st, 2010 at 4:40 pm
Have had an account where most of my websites are hosted since 2005 and couldn’t be more happier today to have you as hoster!
Great job Dreamhost!
Vince
May 21st, 2010 at 4:50 pm
I love you guys! Go DreamHost!
May 21st, 2010 at 4:51 pm
@vicm3 you can get nginx on a Dreamhost PS — find out more here: http://wiki.dreamhost.com/Nginx
May 21st, 2010 at 7:03 pm
Great job Dreamhost!
May 21st, 2010 at 9:12 pm
The fact that you didn’t kick them (like my old host would of) make me proud to keep you as my host.
May 21st, 2010 at 11:29 pm
Hopefully they had some AdSense ads on those pages, wonder if they got any clickthroughs?
May 21st, 2010 at 11:46 pm
I like how the drawmuhammadday.com website DNS record now points to 127.0.0.1
May 22nd, 2010 at 2:07 am
Once again the Muslim show there real face.
Wake up free world!
May 22nd, 2010 at 5:00 am
You guys rock! The twats trying to stop free speech can go climb a muhammad :-)
Good going Dreamhost! Proud to use you for my hosting.
May 22nd, 2010 at 5:01 am
@Fred #31: free speech is important, but without education its value is much reduced. However, if “show … real face” was deliberate irony then I’ll give you a bonus mark.
May 22nd, 2010 at 6:14 am
Thank you for not caving on this important issue, and for just generally being awesome.
May 22nd, 2010 at 6:17 am
Nicely done!
May 22nd, 2010 at 7:58 am
Hey! … do have any of the proof of what you guys are hunking?, losers! … get your security fixed first, … blaming us and getting pity from your stupid customers, … do you really know what “freedom-of-speech” really means, … ashholes! … just pathetic!
May 22nd, 2010 at 10:24 am
@Faysl (36)- I find it interesting that you blather on rather incoherently about “freedom-of-speech” (sic) and yet you miss the irony of this very diatribe of yours being published.
You, fine sir, are an idiot.
May 22nd, 2010 at 11:10 am
Rock on good people — from one of your proud customers.
May 22nd, 2010 at 11:46 am
I applaud your support of the Draw Muhammad website. The world can’t be free and secure until we wipe the planet clean of this fundamentalist Islamic malignancy born in the Middle East and now being exported world wide. Those of us who are born into this world to search for our own truths, define our own destinies, invent and create new technology and new art simply can’t let a religion and race of people determined to live in ignorance and delusion as their ancestors did a thousand years ago stand in our way. We will one day face the fact and it will forced upon us by them, that we simply must eradicate them. And we should not feel any guilt or sadness when it happens, knowing how strong their belief that there is a paradise beyond this world that awaits them.
May 22nd, 2010 at 12:41 pm
You’ve become my new favorite web host service. If I ever in the future need a host, expect to be the first people I’ll contact.
May 22nd, 2010 at 12:57 pm
@Proud To Be An American: Replace “eradicate” with “educate” and I’ll be more inclined to agree with you
May 22nd, 2010 at 3:03 pm
WOW. 20,000 per second. I love DreamHost. They provide hosting for my fairly offensive atheist blog and I’ve never worried about censorship here. Glad to see somebody still believes in free speech.
May 22nd, 2010 at 3:16 pm
You guys are great. I’m happy to be one of your customers for over four years.
May 22nd, 2010 at 6:32 pm
Nice AAR! (After Action Review, it’s an Army thing)
May 23rd, 2010 at 12:33 am
Dreamhost, you are even more awesome than I thought you were.
May 23rd, 2010 at 4:11 am
Just one more reason our site’s on Dreamhost and we recommend you to everyone we know.
May 23rd, 2010 at 4:53 am
My name is Mohamed and I disagree with the way a lot of muslims behave around the world and I do believe in freedom of speech.
Yet, I feel very alienated by the way a company like dreamhost is behaving. Singling out my name for ridicule does not make my life any easier during this time. This whole thing has crossed the line from freedom of speech to downright racism.
I was planning on signing up with dreamhost but not anymore.
May 23rd, 2010 at 11:52 am
Wow Dreamhost is really serious about this free-speech thing, even the spam is getting posted!
@Mohamed: Are you really so ugly that drawing a picture of you is insulting?
May 24th, 2010 at 12:50 am
Well,
I’m really disappointing with your double standard with saying it as free speech. What so free if you have so many forbidden content in your Term of Services?
It’s always easy to do something that hurt somebody else and then use free-speech to justify it.
Now I know why sites such as http://notalwaysright.com will never run out of their content source. Most people are so ignorant and self-centric and always think that all your doing is right.
To you, dreamhost, I think I will start searching another hosting alternative now, because you are hypocrite that obviously make exceptions to your own Term of Service just for money.
Thank you for opening my eyes.
May 24th, 2010 at 6:58 am
@49 Donny Kurnia
Their TOS only forbid copyrighted material, child porn and spammers. Please leave DH then, and die soon.
May 24th, 2010 at 8:29 am
I host for 4 years with you and i´am proud that you defend our freedom of speech! Thanks Dreamhost!
May 24th, 2010 at 10:03 am
Way to take a stand! I’m happy to have hosted with you for about two years now.
May 24th, 2010 at 11:00 am
From some of the comments, it appears there may be some confusion about exactly what role we played here.
First, we have no affiliation with drawmuhammadday.com or the organized day itself. We only mentioned it to add some context to what was a pretty serious outage for our primary website.
Second, it was dreamhost.com itself that was being attacked and all of our efforts were put into bringing that site back up. The drawmuhammadday.com site is still down and is no longer pointing to our name servers so it’s out of our hands. I don’t know if the attack on it is still under way or not.
May 24th, 2010 at 11:23 am
So glad to have you as my webhost. Keep up the great work and never back down.
May 25th, 2010 at 9:04 am
Great explanation and good job guys!
May 26th, 2010 at 7:27 am
@Dallas – you guys told me to fuck off, even after being switched to a PS.
So I did… Or tried anyways. Yes the attack is still underway.
May 26th, 2010 at 10:12 pm
Its one thing to support “free speech” of a client. It is another thing to endorse the material.
By your publishing the disrespectful depictions of Muhammad on your blog it would seem that you too are being disrespectful of other people’s beliefs.
I’m sure many of your clients are of many religious origins. And any attempt to make them feel disrespected would offend them. I am not a Muslim, but I am offended by your apparent active insults of the founder of other people’s Faith.
I suggest Dreamhost take a more professional attitude.
May 26th, 2010 at 10:15 pm
Frankly I just want a hosting service.. not a war ground for ignorant religious wars.. (and both sides of this war show their ignorance in a glaring fashion)
May 28th, 2010 at 10:20 am
I’m not exactly sure why all the pictures of Muhammad would be disrespectful, and EVEN if they were it shouldn’t cause any problems with anyone. I can imagine someone being bothered if they see their deity being laughed at, but that’s everyone’s free right.
In my opinion all religious books should be moved to the fiction part of libraries!
And i really think we should start caring less about funny pictures about Ancient people who are described in ancient books written in ancient times with ancient pens and under ancient laws.
They certainly won’t mind anymore, why should we ?
May 28th, 2010 at 2:50 pm
DreamHost… make love to me, please!
May 29th, 2010 at 6:05 am
I am not a muslim.But still one thing I want to say to u.dreamhost is a hosting company ,not a religious organisation.Why the hell should u fu** others sentiments?yes,There is freedom of speech,but does it mean you will do anything as you wish.will u like if I say ur mom and ur wife is slut and whore and make a website showing their Ti** and P***.yes this is also freedom of speech,so I can say and do whatever I wish.so,in freedom of speech also there is a limitation.Freedom is everyone’s right ,but you can use it only if it did not hurt others in anyway.FU** up ur hell company.In this way,u will make little progress.
May 29th, 2010 at 6:10 am
if all christian do like this,wait one day a nuclear bomb will surely come on ur bloody head
May 29th, 2010 at 1:26 pm
Well done guys you really did a good job.. applause for you that on 20th May you refused your hosting service to drawmuhammadday.com and he went to host his website to Planet who provided him a dedicated server….You really supported freedom of speech.. thumbs up
May 30th, 2010 at 12:25 am
Hi Brett,
Wonder if i can register drawbrettmomtits.com in Dreamhost?
Or maybe brettbombisrael.com?
or brettisantisematic.com?
or brettloveterrorist.com?
or brettwillbombyou.com?
or dreamhostantijewish.com?
or dreamhosthateamerica.com?
or dreamhostisterrorristtokill.com?
or dreamhostsucksashosting.com?
or donthostindreamhost.com?
or dreamhostteachtobombingacrountry.com?
or dreamhostantiblack.com?
or dreamhostemployeeispedophile.com?
will you approve this, for freedom of speech right?
May 30th, 2010 at 12:20 pm
@drawbrettmomtits
Dreamhost wouldn’t have a problem with you registering any of those specific domain names. I’d encourage you to keep any content on the domains legal per USA law, defamation and libel could be a concern depending on if the content was consistent with some of those domain names (I’d be interested in seeing your *proof* displayed on some of those domains, everyone is entitled to their identified opinion though, no matter how wacky it may be).
BrettLoveterrorist.com could make a fun “Hot or Not” type clone. I also could see potentional in Donthostindreamhost.com as an anti-piracy or illegal fraud site .
June 1st, 2010 at 12:46 pm
Please do understand that we won’t be bogged down by these atrocious laws and pacts signed by within USA. This is cyber world and it is free space. Everyone can and must be allowed to have liberty over here and that does include drawing and displaying holocaust images as well. That’s the second level that you certainly won’t like to see us doing. N’d we can take it to that level far more easily than you can even think of . No one champions the rights to it except those whom can define their own just like we (Muslims) defined on 20th May for you.
Have a good day since few days for your liberty are left…
June 2nd, 2010 at 3:11 pm
Hey DH
Can I host a Holocaust denial website with you? remember freedom of speech :)
I’ll also host a gallery making fun of jews, christians, muslims, black people, disabled people… I’ll encourage racism, you know it’s Freedom of Speech!!
OK, Ok, I’m not serious about what’s written above, I just wanted to make it clear that Freedom of Speech is a responsibility, everyone’s freedom ends when they reach someone else’s believes….
Peace to you all
One Love
June 2nd, 2010 at 10:54 pm
I’m a muslim and I second @Holocaust . Freedom of speech is not to be use to spread hatred and attacking religion let alone make fun of it with some silly drawings. If you want to draw, go get some papers and join the kindergarten kids. Couldn’t believe such retard behavior is being endorsed by DreamHost. I mean could you just simply be nice to all to not accepting to host this type of website? You won’t lost a penny but lot more to earn, since all the muslims, christians, jews, black people will love you even more!
Well..you can actually create your own stand on the Freedom of Speech. Wny must follow the America’s one? I know that you are base in America and you guys are all from America, but sometimes you don’t need a country regulation to tell you what to do right? You can judge it yourself and do the needful , whenever wherever you see it fits. It’s your call.
I’m proud to be Muslim and I have many Christians friends too. Thus, if it wouldn’t much to request to DreamHost, could you just kindly withdraw all of the depicted images in your entry?
Thank You
June 8th, 2010 at 1:15 pm
I think you guys would make any bright british, god fearing christian fascists happy.
Look what you’ve done! you’ve gone and lost my support!
Good job! maybe you feel like good Nazi’s now.
Congratulations. oh and your advertising marketing scheme using your old childhood photos as commercials for your crappy hosting is rather jail-baiting…
what kind of content do you host?
oh yeah. hatred and fascist racism/religionism.
how american of you!
June 9th, 2010 at 8:14 pm
I am currently comparing server hosts for modest WordPress sites. I am not a programmer so I want reasonably simple user interfaces. As I did my research, I got the best personal reviews from people I respect for Dream Host. Host Gator and Media Temple were also generally well liked.
It is my understanding that there have been recent server specific attacks on Media Temple, Network Solutions, Go Daddy AND Dream Host. The Dream Host attack was addressed in this report from Danchev.
http://ddanchev.blogspot.com/2010/05/dissecting-mass-dreamhost-sites.html
I came to this blog to see how you are doing with that but I don’t see it mentioned. I know that Media Temple is still working to unravel something that happened in November because I set up an account there this Spring but haven’t yet launched a site there after I realized their troubles.
I found a list of “incidents” on the Wikipedia for DreamHost. The Danchev one isn’t mentioned but Mohammed made the list which is how I wound up here.
Can you please point me to where you provide an update on the current status of Dream Host to assure me that my small but to me precious WordPress sites would be safe, fully supported and fully functional.
I wish that Host Gator was writing about how they are working to ensure this won’t happen like Tiger Tech Net did on their blog in April. Tiger Tech was recommended by a credible source but because of their size it’s not so easy to cross check.
June 13th, 2010 at 1:58 pm
The followers of the Prophet would be better off if they took stock of their qualities as well as their faults.
I make no mistake, these are not stupid people although many are uneducated. But they have a defect. Islam has tried to use force to invade every area that the followers thought they could take by force for over 1000 years. That policy of expansion has brought them nothing but misery. Why not confront the notion that the children of the Prophet suffer greatly and stop the war as well as the stupid violence such as honor killing? Islam suffers as a consequence of these foolish aspects of their society. Sober up Islam!
June 16th, 2010 at 11:47 am
@Emma
Reviewing the link you posted I found that they clearly stated information about the attack, however did not verify their claim that it was a massive attack on Dreamhost customers. What they did go in depth with was details about what looks like an automated attack (which would effect any insecure website, regardless of host), this is why there was no related post about the “incident” this blogspot logger is writing about mentioned on any DreamHost sites.
If you’re interested in reviewing a broad level of security various hosts/internet providers have please see the “FIRE” (Finding Rouge Networks) project. Which tracks and rates network blocks based on the number of IPs on their network which are reported as involved in malicious activity.
http://maliciousnetworks.org/
July 1st, 2010 at 7:30 pm
aha! let’s watch it in the futhure……