Okay, nothing silly this time, I promise…

Some of you may have noticed that we’ve been having what a problem that is, although maybe not the worst in DreamHost history, definitely in the top 5.

There has been a DreamHost Status post about it, but it’s been going on so long, there obviously needs to be more said.

The History

The events that conspired to cause this horrible performance for everybody in our “blingy” cluster actually started to take root 19 months ago.

That was when I made this post asking our customers for some suggestions on storage. I made the mistake in that post of mentioning the name of one particular storage vendor who apparently does a search for their name in rss feeds of all kinds of blogs. I won’t mention their name again here, to test if they REALLY read this blog, but they were the one on the list right after “Netapp”.

Anyway, immediately a sales guy from there was hounding me about how great their product was. It would have super-duper reliability, super-duper performance, and super-duper ease-of-management. It was super-duper expensive compared to our current solution (about 3x the price per GB), so in the end I declined.

But, over the next year he kept hounding me and hounding me, and eventually the price came down to something in line with our current costs, so we decided to try one unit for our new cluster, “Blingy”. After we were satisfied with our internal testing, Blingy went live with the new storage solution in December 2007.

Smooth Sailing

At first, everything was fine, performance was great, everybody was hunky and dory. But then, as usage started to go up, the new file system started acting up. Around the same time every night, the system would stop responding to NFS requests for a while, which would immediately break web and mail service for everybody in the entire cluster.. thousands of customers.

Our Bad

Now, it can be a big mistake to put live customers on any new system. But honestly, we’d tested it lots, researched it a ton, and we added people very slowly at first, and it performed great.

Our biggest mistake I believe had nothing to do with what specific vendor or hardware we went with.. it was simply putting so many eggs in one basket!

Even with our Netapps (which are pretty much awesome), there are problems from time-to-time. However, a typical hosting cluster will have a dozen or so Netapps, which means any problems are one twelfth as big.

With Blingy, EVERY customer is on this one “mega” filer, which in theory should make for better performance, reliability, and ease of management. And since we got the clustered solution (in an active-active configuration)… there really is no single point of hardware failure in this thing.

But, as it turns out, there are a lot of non-hardware failures in the world.

Their Bad

Well, the techs at the vendor couldn’t figure out what was causing the NFS freezing, and so they recommended us doing a major OS upgrade to hopefully fix it.

During this whole time, the fiber channel disks were slowly filling up, and we’d been trying to move large files off to the sata pool (it’s a two-tiered solution, and there’s a feature that automatically moves less-accessed data to lower tiers).. however the thing couldn’t move the data fast enough. It couldn’t finish doing a “move job” in a single day, and every day it’d sort of “crash”, which would screw up the move job, and nothing would get moved.

Also, as the disk kept getting more full, performance kept getting worse, creating a vicious cycle. We ordered some more fiber channel disk shelves at the end of February to grow the main FC volume, since we couldn’t get things off to SATA, and it was supposed to come on March 10th and be installed at the same time as the major OS upgrade.

However, the disks didn’t end up getting installed until March 25th, and at that point it turned out we could NOT grow the FC volume with these disks (well, it was technically possible, but their on-site techs recommended VERY VERY heavily against it.. it would severly impact performance), which was sort of the whole point. So now we had a new FC volume which we still had to migrate users to.

Your Bad

Of course, this whole time, new customers just kept signing up, and being added to Blingy. What were you guys thinking?

By this point we knew this was a bad idea, but we didn’t have a new cluster ready (we’d expected Blingy to grow for another couple of months), and we try to never ever grow old clusters again once they’ve been “shut off” from new signups (because in time they stablize and have very few problems).

However, the moving people off to the new FC vol, or the original SATA vol, or even the new Netapp we also added to Blingy, just wasn’t happening fast enough. So on April 2nd we bit the bullet and switched Blingy off as the “new customer” cluster and started growing good old “Postal” again. Once we did that, we were finally able to get ahead of the curve and total usage on our first fiber channel volume has been slowly dropping ever since.

We tried at that point to contact the vendor to see if we could just get more drives that WOULD allow us to grow fcvol1, but they said their manufacturers were closed for inventory for a week after the end of the quarter and we couldn’t get anything until Friday, April 11th at the absolute soonest. Later they said they could find us some they could get us by Tuesday, April 7th, and we preliminarily said we’d take them.

This whole time we had a support ticket open with the vendor about the crashes (the OS upgrade didn’t fix it), and finally on April 3rd we received notice that they’d fixed the bug that they believed was causing it! However, the patch still needed to go through their “QA”. Finally, this Sunday April 6th they said it was all ready to be deployed, so last night we did.

What Now

Well, right now, performance is still not great on fcvol1… but mail and web should be pretty much working. One thing we’ve noticed is a website that hasn’t been visited in a long time will have a big lag still upon the first visit.. but then subsequent reloads/visits seem much faster.

At least the total disk usage is coming down now, and hopefully by tomorrow it’ll be below 85% which is supposedly a magic number where performance is fine. We’re going to keep off-loading it until things are great, though. We’ve got plenty of disk space for it, the problem is just it takes so long to move it.

We also I guess will find out tonight if the NFS freezing bug is fixed by this new patch. Hopefully so.

It’s Too Late…

I realize this is probably too little too late for many of you, but I just wanted to sincerely apologize for this whole big Blingy cluster-f*ck. Also, if you’re on Blingy (you can tell from the panel by clicking “account status” and looking at “Your Email Server”, we’d like to offer you a month worth of hosting credit.

To get it, all you need to do is contact support from our panel and make the subject of your message “Blingy Account Credit”. That’s all you have to do, and we’ll credit everybody who asks (and is actually on Blingy!) next Monday (April 14th).

I have been to the future.

It’s a very well known future, one everybody knows is coming, and yet, it is still, frustratingly, not here.

It’s not the future of nano-bots, flying cars, and hamburger toothpaste that some predict. Nor is it the future of sex-bots, self-driving cars, and hamburger frosties that others believe in. As pleasing as those two possible futures are, they are not the universally accepted inevitability that this future is.

No, this is the future of ubiquitous wireless high-speed INTERNET! Everybody knows that day will eventually arrive, it’s just a matter of when, and through what vessel.

Whether it be an 802.11 mesh, cell phone technology, sattelites, wi-max, that new spectrum Google was bidding on, or some as-yet unknown future technology, I think it’s a pretty much a forgotten conclusion that by the Year 2000, everywhere you go will have wireless high-speed connectivity, and nobody’s going to pay for it.

And of course, everybody knows that’ll be pretty cool. I mean, everybody who’s already got EVDO service (*cough cough*) knows how great it is to have a fast usable Internet connection with you at all times. It’s great because it’s fast, portable, and reliable… but for most people it’s just not $80 a month great.

But, just as cell phones have supplanted landline phones, so will “cellular” Internet replace the “landline” Internet. Once it gets price-competitve (and FREE is very competitive) no long-term contracts or cancellation fees on the planet will be able to hold back the tsunami of people rushing to escape their local telco and cable company.

You see, apart from the reliability, portability, and convenience advantages that wide-area wireless internet provides, there’s one other pro… a little something I like to call “ZERO C”

I’m not talking about the temperature in Boston right now, either. I’m talking about Zero Configuration!

Currently, to get the Internet set up at your home or office, you’ve got to have a service man come and set things up.

Then, to share that internet throughout your multitude of computers, video game systems, slingboxes, iPhones, and refrigerators, you’ve got to set up a not-exactly-something-your-mom-can-do home network.

Then, whenever a friend comes over with their laptop, they never seem to be able to get on the net without your help.

Then, whenever the cable modem drops out in the middle of the final game of a Bomberman Live match, all Time Warner has to say is “everything looks okay on our side“.

Then, whenever you’ve been using your Airport Express for too long, the music cuts out and you’ve got to unplug it, wait fifteen seconds, and plug it back in.

Then, your wireless never seems to reach to the guest bedroom reliably.

Then, you’ve got ugly cat-5 ethernet everywhere.

Then, once every two months you’ve got to do a firmware upgrade on everything for “stability” and re-do the whole thing..

But now think… what if every device you ever bought was always reliably connected to the Internet at high-speed, no matter what, for free?

You’d never have to deal with Time Warner Cable or Verizon DSL again.

You wouldn’t have to set up or manage or worry about a home network.

Visitors would already be on the Internet everywhere they went, just like you are.

Your Xbox 360 would always be able to connect to Xbox Live.

Your airport express would always stream your music reliably, and you could control it from anywhere in the world.

Everything would still work in the guest bedroom.

You would have no cat-5 cables, anywhere.

Devices could automatically get firmware upgrades because the manufacturer would always know they’d be reliably on the net, since it was free and just automatically worked.

However, as I was saying in the beginning… I’ve already been to this future. And my vessle was..

The Amazon Kindle

That’s right, this humble, $400, sold-out e-reader, is our first baby-step to technology nirvana!

Because I wanted to check it out, I got my wife a Kindle for her birthday in January, and the coolest thing about it is its barely-mentioned “whisper net”.

Shhhhh… this “whisper net” is just Amazon hiding the fact that the Kindle comes with Sprint’s 3G EVDO service for free. I’m not sure what kind of deal Amazon made with Sprint, but …. THIS …. IS …. AWESOME.

Because it’s just ALWAYS on the Internet, everywhere, the thing is like magic… and super-easy for moms and (I assume) grandmoms to use. There’s no settings, no account to create, no monthly bill, no passwords, no nothing. Just a physical switch on the back to “turn the internet on” and you’re buying e-books and browsing the full Internet at a perfectly usable speed.

Once the Kindle costs $99 instead of $399 (and there’s no waiting list), it (or something like it) is going to mop the floor as a super-cheap “internet appliance” that “just works” for “people that are old”.

My hats off to Amazon for truly making the first device that is truly always on of the Internet. By making it free, they’ve guaranteed that as long as a Kindle is working, it’s on the net.

Just imagine the other ZERO C possibilities ubiquitous, free, high-speed Internet would bring!

How sweet would it be for your next digital camera to have? No more worrying about sd cards, usb cables, or emailing your pics… the moment you take a pic, it’s backed up to some picture hosting site, shared with the world and freed from your camera’s internal memory. As a bonus, all pics you’ve ever taken would be able to be called up and previewed right from your camera’s (not-so) little LCD screen.

How awesome would it be to have a free-EVDO skype handset? That’s it for paying for cell phone calls.. and it’d be so easy to get everybody to switch from the archaic POTS system to voip when there was finally a no-monthly-fee cell phone that worked everywhere their existing cell did.

How cowabunga would it be to have a Nintendo DS with this? Anytime you’re sitting around, riding on the bus, like I am now, but uninspired from writing any meandering blog posts, you could whip it out, do a couple Mario Kart races, and then when you realize you forgot your Dr. Mario cartridge at home, shortly thereafter remember that there are no cartridges anymore, every game is just streamed directly from your account on Nintendo’s servers!

Anyway, yep, that’d all be very awesomely cowabungifiededly sweet.. but it’s still a ways off. In the meantime, I hope this little ZERO C fix will hold you.

We’ve finally made a true one-click install process, which is the way I always envisioned our one-clicks would work from the beginning, several long years ago.

Just go to our panel’s one-click installer area, and click the new “easy” mode.

From there, you just choose the domain or sub-domain you want to use and give your new site a name, click the submit button, and in literally under five minutes, you’ll get an email when everything’s done.

Previously, you’d have to already have set-up the domain or sub-domain you wanted, and you’d have create or pick a database you wanted to use, and then when you got the email there’d be some more software package-specific installation steps for you to complete.

NO MORE!

This time, one click really means ONE CLICK!

When you get that email, you are done.

It is only available for WordPress now, but believe it or not, this new one-click process we have is actually easier to implement for future software packages than the old way, so it shouldn’t be long at all before we fill in the portfolio with lots of other yummy goodies.

The only downside is, the easy mode actually hosts all the software on a centralized, load-balanced service we’ve set up, rather than in your normal webspace. This means that any customizing that requires changing files won’t be doable. Fortunately, most software packages keep all the customization you’d want to do in the database these days, so this isn’t really that big of a restraint.

The upside is no maintenance (we handle all upgrades), hopefully better reliability and performance (as it’s now effectively a “hosted” service as opposed to a local install), and of course…

ZERO C!

Starting today, every DreamHost PS and DreamHost dedicated server comes equipped with the Mongrel ruby application server. Mongrel is the preferred method of deploying Ruby on Rails applications and is typically more robust than FastCGI-based deployments. Now we’re giving you both options to choose from so you can use the one that works best for your application!

We recently posted a rant about how Ruby on Rails is not easy enough for people to use, and this is one way we’re working to fix that situation. Now anyone can use Mongrel with very little server know-how. It only takes a few minutes!

Mongrel sits behind an Apache running as a proxy server, and our web control panel now streamlines that process and handles all the details for you. You just upload your application anywhere under your web-enabled user and set up the Mongrel application server, and in a few minutes your site will be up and running. For more information, check out our Mongrel wiki article.

Note that this feature is only available for DreamHost PS or DreamHost dedicated due to its resource requirements. Due to extremely high demand our inventory of DreamHost PS servers is low but we expect to get more in soon. If you would like to get on the waiting list, you can do so on our DreamHost PS provisioning page.

Okay, I’m not exactly sure what the current state of the writers strike is, so I’m just going to make this a Happy New Years post and be done with it.

Happy 2008!

There, got that out of the way!

And now, here are a few of my personal new years resolutions:

* Eat more chocolate.
* Exercise less.
* Start smoking.
* Start really drinking.
* Save less money.
* Spend less time with family.
* Read less books.
* Be more selfish.
* Get less organized.

and of course…

* Get the January DreamHost newsletter out by my birthday.

Now, please don’t get accustomed to this, but I thought I’d go ahead and end this shorty-short post with a NEW FEATURE RESOLUTION!

Well, a new BETA feature resolution at least. Please go and test out a new webmail client at roundcube.dreamhost.com … and comment about it in this discussion forum thread … and who knows, maybe before the end of this writer’s strike we’ll have replaced squirrelmail with it!

Ha. AS IF!

We are!

Well, maybe not ALL fifth graders, but I’m sure at least A fifth grader.

Like, one who skipped first grade or something.

Like me! (I was too tall smart.)

Anyway, DreamHost is TEN YEARS OLD!!!

   Domain Name: DREAMHOST.COM
   Registrar: NEW DREAM NETWORK, LLC
   Whois Server: whois.dreamhost.com
   Referral URL: http://www.dreamhost.com
   Name Server: NS1.DREAMHOST.COM
   Name Server: NS2.DREAMHOST.COM
   Name Server: NS3.DREAMHOST.COM
   Status: ok
   Updated Date: 21-sep-2006
   Creation Date: 23-sep-1997
   Expiration Date: 22-sep-2013

In dog years, that’s SEVENTY!

In Internet years, that’s ONE THOUSAND!

In waiting-for-tech-support-to-get-back-to-you years, that’s INFINITY!

To celebrate, I’m doing this super-long blog post retrospective… and if you read the whole thing, you might feel a little less let down about the announcement I mentioned last post just being a freaking birthday announcement.

Stupid Beginnings: Pre-DreamHost

Man, I was just looking through some old emails from 1997, and one thing I can say is, boy, were we dumb!

It’s nice to know some things never change.

Date: Sun, 5 Jan 1997 15:06:32 -0800 (PST)
From: Josh “Evening Optimist” Jones
To: Honchos
Subject: Plan?

Maybe we should piece together a mission statement or guidelines for
business or goals or something. Maybe we should also come up with some
different plans for pricing web hosting. Like some amount for a small
business site (at most 500 hits a day or so) and some amount for a large
site, with a discount if we were the ones who made the site. All should
probably have 20megs of space, with $5 per 10 more or so (I’m of course
wide open to specific price suggestions, these are just round numbers).
Maybe $30 for a small, $45 for a large, plus maybe $5 more per some number
of hits. Also more if they have a domain name. If we made their site
though, how about $15 off the base price?

Anyway, I was thinking we need some reason that people would want
to use us instead of our competitors. Why would they now? We are good at
design and graphics and programming, but honestly there are plenty of
places that are good at that. We have to be something others aren’t. Like:
quicker at getting the job done, better sites, cheaper sites, or maybe
even something like better customer service or even more advertising. The
reason I thought that low price would be a good way to go is because we
have an advantage over most other places in that. We aren’t actualy
dependent on this (at least not _yet_) whereas others are. I don’t think
we have an advantage in speed, especially since we are all full time
students. We are good at design but it’s hard to sell people on our sites
being the highest quality I think. I can imagine some people really liking
our work and other people not so much, there is a lot of personal taste
involved. Another thing which I guess is sort of obvious is that it would
be good to get some large sites that we need to change a lot and sort of
constantly maintain and add things to, because we can keep charging for
that. Like when we get a job, we should outline clearly what is included
in the setup, and additional things (like more pictures, etc..) are
clearly going to cost more later. Okay thats it. I would have written this
earlier, but my connection was flakey yesterday.

Josh

But wait, isn’t that email dated January 1997?! We didn’t register dreamhost.com until September! What oh what were we doing in the meantime?

The thing is, the actual company over here is really called “New Dream Network” .. and the goal was never (and still isn’t!) to be a web host. We did some web hosting on the side to try and cover the network we were stealing from a friend, but we generally didn’t want it to ever get too big.

Buuuuuut, once we started actually raking in the dough, that mentality changed quick. Let me give you an idea of how much dough there was to be raked back when we decided to get serious and get an actual domain name..

07.29.97 pillar Pillar Communications $20.00
07.31.97 pinzler Andrew Pinzler $48.00
07.31.97 jbark Joseph Bark $46.00
08.06.97 tim Timnet $126.00
08.12.97 threnody Cheryl Dowling $136.00 VOID
08.12.97 jhb5 Vickee Sepich $46.00

Here’s an interesting little exchange I found too… the origins of the DreamHost name:

Date: Sat, 20 Sep 1997 14:06:21 -0800 (PST)
From: Dallas Bethune
To: Honchos
Subject: Re: everyhost.com

Hey,

I just discovered that dreamhost.com is not taken!

We could snatch it up! What do you think?

I’m making an ad for hosting right now, BTW…

->Dallas

> > Hey what do you guys think of everyhost.com? We could make it our mission
> > to make having a website with a domain name easy and affordable for
> > everyone from private citizens to small to large businesses. Therefore..
> > everyhost.com (it’s not taken). Also Dallas, are front page extensions
> > still installed somewhere? I’m going to take advantage of the beta status
> > of FP98 to download it and see if we can get our server to work with their
> > extensions. It would be good to put at least on our hosting server once we
> > get it.
> >Josh
>
> I’m not that taken by everyhost.com. I don’t think I would personally want
> to have my site there. It would be fine if we were trying to focus on
> sites with their own domains, though.
>
> I believe I deleted the FrontPage stuff. We never got it working right,
> and were low on space at some point.
>
> I’d almost rather not have FrontPage going on our servers. It seems kinda
> neat, but I’m still worried about what access to our server that it seems
> to require…
>
> ->Dallas

Ha, EveryHost! Just rolls off the tongue, doesn’t it?

I do feel somewhat vindicated that everyhost.com was snatched up less than two years later!

(One thing sort of funny is… we were assuming most people would be getting sub-domains of our main domain. Dallas was saying that he wouldn’t want dallas.everyhost.com as much as dallas.dreamhost.com! Of course “It would be fine if we were trying to focus on sites with their own domains, though.”)

A Nightmare is Born

Woooooheeee! Thanks the Wayback Machine I’ve been able to find and recreate the entire history of DreamHost.com and lay it out for you here, complete with what I think are the most interesting points in each design! Unending boredom awaits..

This was our first design!

Designed by Dustin Vannatter, New Dream Member extraordinaire, I’ll always hold a special place in my heart for this one!

Back then, we actually had specifically an “adult site hosting” plan.. it was $99.95/month and, like all our plans, included unlimited bandwidth! As unlimited as half a T1 can be! It came with 100MB per 5GB of transfer you used, which was really weird in retrospect.

We also had our Crazy Domain Insane plan for $9.95/month with 20MB of storage, Archive Boy for $17.95/month with 40MB, Code Warrior (we hadn’t gotten that Cease and Desist from Metrowerks yet!) for $23.95/month with 50MB and a telnet user and CGI access, and Strictly Business for $44.95/month with 100MB of storage along with 20 email addresses and anonymous FTP!

It turned out, that Adult Site hosting plan with unlimited bandwidth was the only thing that kept us solvent those early months. As soon as we put that “too good to be true” offer up there, we started getting deluged (as in, multiple PER WEEK!) with signups for it! And these were big customers too.. $100/month!

It took about a week before we realized that unlimited bandwidth plus adult content equals not good. Some of these people were using over a GB a day of transfer.. and according to an early email from michael, we needed to be making $200/GB to stay afloat! We immediately had to re-negotiate with some of those early adopters.. one guy began paying $700/month, and others left.

We did learn an important lesson though, and that was that some of those $100/month adult sites used hardly ANY bandwidth at all! And thus, the truth about overselling was realized!

(Ha, if you thought having a dedicated adult hosting plan was crazy, before dreamhost.com launched we had a dedicated warez hosting plan!)

We also had “colocation” options back then:

For $995/month you got 50GB bandwidth, 64MB RAM, a 3GB SCSI drive, on a Pentium 200Mhz!

For $3000/month you got a PII 400Mhz, 256MB RAM, two 9GB SCSIs and 300GB of bandwidth!

The deals would have been a little sweeter if I’d had my way though…

Date: Mon, 29 Sep 1997 11:35:29 -0800 (PST)
From: Dallas Bethune
To: Honchos
Subject: dreamhost.com site

I looked at it…

I’m alarmed by the colocate page.

How do you think we’re going to offer

200MHZ Pentium II
64 MB RAM
4.2 GB drive
Full Debian Linux Distribution

Only $300 setup
$300 a month.

At this ridiculously low prices?

We’re trying to make money…!

The offer I was talking about would be a 100 MHz Pentium with 16MB Ram and
2 gig drive for $1500 setup and $500/mnth

That price is even low…

Also, I’d rather not give everybody full CGI access that is a client of a
reseller. That would be a good way to open ourselves up for a lot of
attacks from people we don’t even really know at all. I think we should
develop a set of cgi scripts that most people would want to cover people’s
want or need for CGI without opening the server up. We can’t afford to
get new hardware too often…

I also feel especially unsure about granting full CGI to people with warez
pages. They would be the ones most likely to try to cause trouble,
right? (maybe I’m just getting old)

Most people don’t need cgi except for little things. If we can provide
those things, we can secure ourselves from big programs that use up the
processor or what not. I’m paranoid.

I think we may have to discuss our prices overall, as well. We are way
lower than a lot of places. This helps us get business, but we may need
to reassess our costs, and our income, and all that.

->Dallas

Yeah, early on you got ONE mailbox, ONE hosted domain, ONE ftp user, and NO cgi access unless you were at least on Code Warrior.. and you liked it! Not to mention domain registrations were $70/year from Network Solutions (and only Network Solutions!)

From the very beginning we had a “reseller program” (you’d get 20% off) and in November 1998 I started the monthly newsletter. Oh yay.

So yep, that was it, one server, four honchos, and $200 in the bank.

Two Years Pass: September 1999

First thing I noticed on this redesign.. our 1-888-261-4484 is nowhere to be found! I must have gotten tired of all those voicemails setting off my pager. Good riddance to phone support! It wouldn’t return (in the form of callbacks) for three years.

By our two year birthday, DreamHost has grown from the four honchos to 19 people.. and gone from no profit to profit to no profit again, thanks in large part to Sage’s WebRing millions!

We’d dropped the unlimited bandwidth, but added some “extra options” at this point: get an extra ftp username for $5/month, and extra mailbox for $2/month, and extra storage for $5 per 10MB!

We’ve dropped the adult plan, added a domain parking plan for $30/year .. NOT including registration (but refunded if you upgraded to full hosting!), renamed “colocation” to “dedicated” (after all, we still don’t have our very own data center!), started offering squirrel mail webmail, and were giving away a free iBook! This was back when people used to read books!

We later had a lot of other giveaway contests.. DreamCasts, Handspring Visors, Game Boy Advances, and even, on the launch of “DH2″, a PS2!

DreamHost 2: September 2000

Pretty much since I graduated from college in May of 1998, we’d been working on “the future of webhosting.” We were going to call it “DreamHost 2000″ in the theme of “Windows 2000″, but by the time we actually got it out and done, the year 2000 didn’t really seem like the future anymore, so it was just “DreamHost 2.0.”

What did DH 2 bring?

Well, mostly the panel as you more-or-less know it today. We also had a system that should have pretty much seamlessly scaled from 3 or 4 servers to 3 or 4 thousand. And I guess it more-or-less did.

We also started doing our “own” domain registrations (through register.com, then joker, then tucows, and finally, many years later, our own ICANN account!) for $30/year.

We dropped the “Archive Boy” plan and created “Sweet Dreams” and upped our storage offerings to 30/75/150/250MB, our bandwidth to 2/4/7/12GB a month, and our mailboxes included to 3/10/20/40!

We also raised our prices to $10/$20/$35/$60 per month and added more extras: discussion lists for $10/month each, SSL access (not including a cert) for $20/month, extra MySQL DBs for $7/month each, and extra bandwidth for the rock-bottom-remainder price of $15/GB!

We also didn’t include any sub-domains on Crazy Domain Insane back then.. or even CGI access! We did however have an official logo! And 31 employees though.

Promo Codes, DreamServers, and Disaster: September 2001

September 2001: did you forget that American flags weren’t just in meatspace? We got on the bandwagon ourselves.

Witness, the humble beginning of promo codes… originally a way for us to give college students a discount! We also started giving away a free registration with hosting, and had a domain checker right on the front page. We created the KBase and launched DreamServers; starting at only $395/month now.. for 40GB of bandwidth, a 10GB disk, and a 600Mhz Celeron with 128MB RAM!

We were now hosting 29,380 domains! And.. we went crazy and way upped disk to 60/300/600/1000MB, and bandwidth a smidge to 2/5/10/30GB, and dropped pricing for overage to $15/$10/$10/$5 per GB.

Even crazier… on the front page, a huge form where you could specify how much of each feature you wanted and how much you were willing to pay, and we would “recommend” a plan for you! The actual point of this feature was market research though.. after months of data collection I would go back through and decide how to best update our plans to maximize revenues!

At this point we had gone through our own little mini dot-bomb.. our head count was only 28 since we decided to stop borrowing money from Sage! Fortunately, our stock art head count had tripled in the same period.

Rapid Growth: We Turn Five!

Really, we started growing faster a little bit before this completely sweet redesign. It started when I analyzed all those “recommendation” requests and realized we really should just be giving more away on our cheapest plan. Sure, less people are “up-sold” on our more expensive plans, but really, those people were just going to our competitors.

So, we gave CGI on CDI, upped our disk to 100/400/900/1500MB, our bandwidth to 2/10/20/30GB, our mailboxes to 20/60/140/300, and gave away unlimited MySQL databases on all plans… I believe an industry first? Oh ho ho!

Of course, we did put a limit on the database usage you could have… the short lived “conueries” metric! 25 times your connections plus your queries! And you got 10M “conueries” per month on CDI!

But still, the redesign was nice too.. we did it thanks to the prodding of a PR company we hired for the still-ridiculous price of $10,000 a month. The biggest thing we learned from that was how easy PR is.. in fact, they even told us they couldn’t have done our press releases (they made us do) any better.

With the fifth birthday, we renamed Code Warrior to Code Monster, upped disk space another 50MB each (of course, all old customers got it as well!), and jumped bandwidth up to 20/25/30/40GB a month. DreamServers was now just DreamHost Dedicated (too many brands the PR company said!), and for $199/mo you got a 1.6Ghz P4, 256MB of RAM, a 30GB drive, and 75GB of bandwidth!

Another critical thing we started here was the ability to cash out your rewards (10% of all payments for people you referred, plus 5% of people they referred), instead of just applying it towards your hosting bill. That was a pretty big deal for our burgeoning affiliate crowd!

We were down to just 24 employees, and $300,000 in the bank!

The Sale Era: September 2003 and 2004

A lot of you reading this probably trace your history with DreamHost back to this period.

Although 777 on our 7th birthday was the culmination, the beginning was actually back on our sixth birthday, in September 2003, while I was actually in Hawaii at a friend’s wedding.

We decided to try, as a lark almost, giving our Strictly Business plan (1.6GB of storage and 40GB of bandwidth plus every other feature) for the price of Crazy Domain Insane, forever.

At that time, we were peaking at about 30 new customers a day. I thought, optimistically, the sale would give us a 50% bump.

The day I turned it on (from Hawaii), we got 300 new customers. The next day, 600! The third day, at which point everbody was screaming for me to turn it off, 1200! In a period of 3 days, we’d provisioned as many accounts as we usually got in 3 months.

It was a hectic time, fo’ sure. The support team hated it. Fortunately, they’re not in charge!

As a result of the incredible demand we witnessed, I was able to convince everybody to allow me to up our offerings a few months later to 500/1000/1600/2300MB of disk and a bit more bandwidth too. We also dropped the price of Code Monster to Sweet Dreams “temporarily.”

The next year, we did essentially the same thing, except we tried just making our cheapest plan SUPER CHEAP.. the 777 sale allowed you to get a year of CDI for just $9.24! Are we Crazy? Insane? Domain?

At that point we’d also already upped disk to 800/1600/2560/3680MB doubled bandwidth to 40/48/64/88GB (and dropped overage to $4/$3/$2/$1 per GB) as well as tripled the number of included mailboxes. We’d also started giving a 20% discount for pre-paying for two years. Pretty much just so we could say our price was $7.95/month!

We also started offering a 91-day money back guarantee (since 1 and 1 had appeared on the scene offering a 90-day!) and allowed opting for a one-time payment of $65 for referring somebody to DreamHost!

Our dedicated servers had a $99.95/month option with a Pentium 4, 512MB of RAM, a 30GB disk, and 500GB of bandwidth.

We had 23 employees for the entire two year period, had paid Sage off the money we’d borrowed (without much interest, which turned out to still be a pretty good return for 1999-2002), and had a cool mil in the bank!

It Gets a Little Ridiculous: September 2005

So, in January 2005 we decide to triple disk and bandwidth to 2.4/4.8/7.6/11GB and 120/144/192/264GB! We had to, man! It was like all you had to do was up those numbers and you got more money!

In the “Spirit of ‘97″ (not at all because others were offering more, nope!) we upped our rewards payout from $65 to $97, as well as our money back guarantee from 91 days to 97. In March we hit 100,000 domains!

And that’s when our power problems began.

I won’t get into it tooooooooooo much right here, but our main data center essentially ran out of power over two years ago and is still out today. We immediately stopped selling any new Dedicated Servers (at that point we were adding about one a day). I wasn’t too heart-broken because my first love had always been shared!

So never mind all that!

We added Ruby on Rails support shortly thereafter, and this blog got started in July with let’s save our environment, truly one of this generation’s great folk hits.

We double disk again, and added the feature where your bandwidth and disk grows every week you stay a customer with us… we’re still the only host who does this that I know of/care about!

Anyway, the 888 promo code only gave you 80% off, and wasn’t nearly as big a deal as 777 (which we’d actually secretly still left working for most of the year!), but we did also up all our plans to finally include unlimited domains and sub-domains, something customers had been asking for for years, which gave us a pretty big boost.

Domain registrations also dropped to $9.95/year and extra bandwidth was now $1-$0.50/GB. We had 30 employees now.. an unsettling trend in my book!

I Just Like This Fat Kid

That was our website in January 2006, when we went completely insane and finally upped our disk FORTY TIMES and our bandwidth TEN TIMES to 20/40/60/90GB and 1/1.2/1.6/2.2TB.

Around now was when we got sick of just losing all those potential dedicated server customers (still no power) and decided to just start linking them over to hosting.com for some affiliate sugar.

Fan Gets Hit With It: September 2006

That summer there’s more power outages and we have TWO FULL MONTHS of pretty darn bad service. It was pretty sucky all around.

We did about the only thing we could do.. made a new site all based on “community” and doubled bandwidth and 10 timesed disk again!

At this point we’re also giving away 3000/6000/12000/24000 mailboxes and 75/175/375/775 shell accounts. We have a 999 promo code which gives $99.99 off (again, it’s no 777!) and take it a little easy.

We’ve got 300,000 domains, 50 employees and a lot of infrastructure stuff to deal with.

Everything Is Wonderful: September 2007

In January of this year, we took a step back. A step away from everything that’s made us who we are, our very essense, and we actually started reducing how much disk and bandwidth we included on our plans.

We had (close to) no promo code sales all year, and never upped those quotas a smidge. It’s been very very very painful for me.

Well… sweet release is finally here!

The Payoff

If you’ve read, or at least scrolled, this far… you deserve something!

And here it is.. for the big One - Oh, DreamHost is now offering only one plan! It’s called “Happy Hosting” (though it doesn’t really need a name when it’s the only one) and it comes with 500GB of disk, 5TB of bandwidth per month, and unlimited users and mailboxes, etc, etc, etc…

Current customers immediately get the unlimited users and mailboxes, and their bandwidth doubled. We’re also doubling your existing disk space, but it will be rolled out incrementally. If you want to switch to the new plan, you can today from our panel!

It’s $10.95/month, but if you prepay for 1 year it’s $9.95/month, 2 years it’s $8.95/month, 3 years it’s $7.95/month, 5 years it’s $6.95/month, and 10 years it’s $5.95/month! There may be a crazy 777-ish promo code too (for new customers) if you look around.

TEN YEARS?

Who would pay for ten years in advance?

I dunno, but at least we’ve finally shown we can do ten years!

Happy Hosting!

P.S. And, when you renew in 2017 you’ll (most definitely) be up to 12.5 PB of storage and bandwidth for $1.95/month!

I’m bored, so for punishment this is going to be a very boring, and serious, and factual blog post.

We’ve now got DreamHost PS MySQL.

Eat THAT, commenters.

DreamHost PS MySQL is completely separate from regular DreamHost PS.

That means:

You can get it without DreamHost PS.

Get it if you just want isolated/protected/guaranteed MySQL resources for your databases.

ALL your MySQL dbs on your account are moved to your very own MySQL instance inside a Linux-VServer virtual private server on dedicated database hardware.

The dedicated database hardware means local, raided, high-performance SAS disks, the same stuff we use for our shared DB servers.

The pricing is the same as for regular DreamHost PS: $1/month per 10MB of RAM.

If you get both DreamHost PS and DreamHost PS MySQL, you’ll get 20% off your monthly fee for both services.

(Okay, those didn’t really follow from the lead-in sentence, but nevertheless, them’s the things you needs to know.)

Once again, DreamHost PS MySQL is invitation only. But, you can get on the wait list at the same place as before: our panel, doi.

These are the facts as we understand them to be.

Well, it wasn’t a TOTAL con.

At least Dallas and I didn’t pay anything to go. He was on a panel about green hosting, and I got free admission by signing myself up as “press”. I guess in a way I’m paying now via this feeling of obligation to blog-post about it though.

Anyways, now I finally understand why we say we don’t go to hosting conferences.

They’re not for us.

Overall, we just got a really “businessy” feel from the whole thing. I mean… we can’t be the only host who’s just doing this until our band makes it big, right? And man, nobody told me to wear a logo collared short-sleeved shirt; the official uniform of hosting cons.

What Happened

Basically, we checked out the display booths (it looks like the new trend is to give away Wiis, iPhones, and Mini Coopers.. sadly, Dallas already has those, and I don’t want a Mini Cooper because I hate the environment), had three meetings, went to three talks, the best of which by far was Dallases panel. And that was just because I interrupted a lot.

We also went to the keynote, which was by some myspace founder guy, and probably the second most-famous person at the con showed up, Carson Daly.

(The most famous? Hmmm… well, I don’t see you reading the Last Call Blog, do I?)

The booths really didn’t do anything for me.. it was almost entirely places offering pre-packaged software (we use only open source or develop our own) and out-sourcing / reselling opportunities (again, we try to be as “vertically integrated” as possible, and don’t outsource anything besides our data centers and network connectivity.. plus, any add-on service we do add we develop (and fully control) ourselves).

We were a little shocked to find out that some fairly sizable hosts just use The Planet for their entire infrastructure… they don’t own any of their servers!

The talks didn’t really do anything for me.. I already knew all the gibberish Dallas was going to say.. so predictable, man!

The next talk, from a Tier1 Research guy, allowed me to self-affirm the seemingly irrational disdain I’ve always held for market research companies. His talk was entitled “Marketing Web Hosting Services in a Rapidly Transforming Market” and basically his message was “I think everybody should partner with Microsoft and other value-added resellers to make more money by offering more junk to your customers.”

Exactly what we don’t want to do.

Oh, and he also threw in for good measure “Just offering lots of disk and bandwidth isn’t going to get you any more customers.” Ah, now that actually sounds like a pretty reasonable assumption, Philbert… if only it weren’t 100% exactly WRONG! “Research” is always easier when you just declare your hypothesis correct rather than bothering to actually test it…

(Ouch, my punches are un-pulled!)

(Oh yeah, and despite what I said before, bad stuff DID happen while we were away. A $64,000 rack of NetApp storage got dropped on the loading dock by the delivery guys! The gentle curving of the rack you see above is not to reduce wind resistance.)

The last talk we went to, before we decided we had to stop for fear of death (and not by boredom actually, but by freezing in the lecture halls!) was by founder of Open Hosting, entitled “Virtual Private Server Hosting with Utility Pricing.”

I had some high hopes for this talk; at least the guy giving it actually runs a web host! Unfortunately, it turned out to pretty much be a bust. I guess there’s just not a lot of insight to be gleaned from a host with 2,000 times fewer customers than you!

Also, it turned out what this guy called “utility pricing” wasn’t anything of the sort. It wasn’t something cool like Amazon … instead, he had regular old (and not very generous) monthly plans with hefty overage fees for excess CPU and memory.

The whole point of “utility pricing” is if you don’t actually USE something, you don’t have to PAY for it! Not to still pay $19.95/month minimum no matter what! This guy has taken the worst from both worlds and combined them.. no “overselling” and yet still a high minimum monthly fee! Where’s the VALUE?

The Open Hosting guy also claimed that they were the only Linux-Vserver-based host in the U.S. Say whuuuuut?

Who Happened

On the bright side, every person we met was very nice… plus I got to taunt lunarpages, as well as eat lunch with the just-a-little-bit-less-cool-than-us Media Temple entourage. I also got to meet all my secret admirers, and let me tell you, THERE WERE A FEW.

Honestly, I guess if there’s any reason for us to ever go back to a hosting convention, apart from avoiding our smelly employees, it’d probably be the chance to try and recruit some decent “human capital”. That’s what it’s known as in the “biz”, which is what the biz is known as in the “biz.”

Oh, before I forget, there was maybe one more tiny thing that that came out of our three days in sunny Chicago. We got an idea for a brand new feature.. and it’s already ready to go!

Perhaps it was the Tier 1 guy yammering on about upselling, or maybe it was the Open Hosting guy’s illuminating discussion of Linux-Vserver, but we’re not here to play the blame game.

Nonetheless, for some reason, we’re now proud to announce our first entirely new product in a lonnnng time: the massively simple, tremendously useful, surprisingly cheap, and enticingly prestigious, currently invite-only DreamHost PS!

(Yep, DreamHost just became one more American host offering Linux-VServer. And Open Hosting just became one American host offering Linux-VServer less special.)

No, not the iPhone.

DreamHost!

That’s right, we’ve sold out!

Let me ’splain:

There are only two ways to grow a company’s revenue:

1. Get more customers.
2. Get more money from each customer.

(or 3… both 1. and 2.)

It’s been our feeling at DreamHost that we’d rather go with the former than the latter (or even both).

But why?

Mainly because you don’t have to be so sleazy when dealing with your customers, always trying to upsell them and nickle and dime them to death for extra features. People appreciate no hidden fees and no bait and switch, and by not going for #2 at all, you help #1.

And, you can always save #2 for later, if you ever reach a point where you can’t grow your existing customer base as fast as you’d like (we ain’t there yet!).. which is basically the old “get big fast and worry about monetizing later” philosophy you hear about so much in Internet start-ups.

We’re getting kinda old for a start-up (Almost a decade!), but I guess we’re still happily in that “get big slow” phase!

Let’s check out the sign up process at godaddy:

On the first page you see this at the top (among the dozens of ads for other services):

On the next step you see:

(heh, I just discovered godaddy doesn’t allow you to register domains with the string “godaddy” in them!)

Ignore everything (even MeUnemployment.com, if you can), click continue… and…

Compare that to our signup process. One page for your hosting info, and one page for your payment info.

It’s a little amazing what people will go through to save a couple of bucks a year!

You know what else is amazing? One of our support guys used to work at godaddy and guess how they measure the effectiveness of their phone support team?

I’ll give you a hint.. it’s not based on number of calls. It’s not based on customer feedback. It’s not based on random monitoring of quality. Nope, none of these things.

It’s based on sales! That’s right.. they actually treat their customer service line as a SALES team. Your job as phone support is not to help a caller with their question or issue.. it’s to upsell them on other services!

I can just imagine a typical godaddy support exchange:

godaddy: Welcome to godaddy, can I interest you in an adjustable rate home equity line of credit?
UGC: Um.. hi. Er, no? I just need some help transferring my domain registration away.
godaddy: No problem! However, it’s important when transferring your domain away that you turn on domain privacy. Would you like me to add that to your account now?
UGC: Uh.. really? I need that? I guess so if it’s required. Wait, how much is it?
godaddy: It’s really cheap, less than the cost of a latte a day! And, if you also get it with the new American Airlines Citibank Mastercard we’re offering, you can get it for even less!
UGC: Well, wait. I’m trying to transfer my domain away from you.. it doesn’t seem like I should be adding new servi..
godaddy: Don’t worry, I’ve already added the privacy for you! But with the credit card and heloc I signed you up for, you’ll actually be saving money, long term! If you can’t afford it right now, don’t worry, I ALSO just signed you up for a payday loan, and you should see your shipment of Viagra in 2-3 business days. Thanks for calling godaddy, goodbye!
UGC: Blam! (Shoots self.)

But they’re not the only one! I registered a domain with Aplus.net once just to check em out.. and look at this screen shot I took of their admin panel:

It’s crazy.. more then half their navigation is dedicated to selling you more services!

But, I can’t really blame these guys. People really do fall for these stunts, and you really do earn more money. It doesn’t hurt that when you start to get big enough, you’re approached almost daily with some amazing new opportunity that needs access to your loyal customer base.

After a while it starts to wear on you and you decide to try just one.

And you find out it really does bring in some extra cash. Not much, but hey, every little bit counts, right? And it’s at no expense to you, so it’s all profit! So you try another, and another, and another, and before you know it your business is starting to remind you of a default Dell desktop!

Long term, all this selling out can be damaging to your brand. Godaddy is huge, but (as far as I’m concerned), pretty much universally reviled. When customers start to lose their naivete, they start to realize they’ve been shelling out for a lot of junk they really didn’t need.. or at least didn’t need to pay extra for. They start to realize they’ve been taken advantage of. And that realization hurts!

Alright already… what’s this about DreamHost selling out?

Okay, enough meandering. What I’m trying to say is, we’ve been approached so many times for things we finally decided to go ahead and let some of them get through. However, we’re approaching it in the most straight-forward, un-annoying, not-invading-your-privacy, open, and passive way we could…

There’s now just one place on our panel where all of these offers will be collected. It’s the new Home > Partner Offers area (thanks for the inspiration, aplus!), and if you never visit that tab, we won’t blame you.

Besides keeping it so confined, we’re also only adding “partners” who:

1. We deem as useful to DreamHost customers specifically.
2. Offer our customers something they couldn’t just get by signing up on their own.
3. Seem at least somewhat non-skeezy to us.

We’re also doing something I bet you’ve never seen anywhere else; we’re fully disclosing whatever kickback we’re getting for each of these offers. I’m not sure how that will affect people’s decisions, if at all, but hey, anything to be different, right?

Finally, we’re launching this new area with a “partner” (imagine the double curly air fingers thing) we’re actually somewhat stoked about teaming up with… Bandwagon! It’s Mac software that costs $24 a year (they have to host your meta data or something?) that allows you to back up and sync your iTunes library with an FTP server.. a la your DreamHost account.

So, the deal is, any DreamHost customer can get a free year of Bandwagon and any Bandwagon customer can get a free year of DreamHost! Seems fair (except wait, isn’t a year of DreamHost worth five times a year of Bandwagon? WE’VE BEEN HAD!) If you want to try Bandwagon out for free for a year.. just go to that new Home > Partner Offers area, and click away!

Just what we need, everybody actually using their storage.

That’s awesome!

But what does it mean?… It means a few things, actually!

It means we’ve calculated our carbon footprint. Our footprint represents the impact of everything that DreamHost uses and leaves behind in the course of our daily work. All of the resources that we use - paper in the office, electricity for our 1300+ servers, even the gas in our cars that bring us to the office - leaves behind some kind of soul-sucking residue in the world.

Electrical power plant emissions clog up our atmosphere. Cars idling in LA’s renowned freeway traffic turn the skyline a muddy brown. Angry profanity-laced faxes are printed on laser printers which generate headache-inducing ozone. (Thanks for those, by the way.)

Putting a price on carbon output is just one way to help make the world a better place. It’s a first step towards true energy sustainability. Organizations large and small are constantly working on reducing their environmental emissions to meet government-imposed (and self-imposed) emissions caps. When they do so a neutral third party then steps in to verify the reduction and issues what are known as “emission reduction credits”.

Companies like DreamHost can purchase these credits which are then immediately retired on their behalf. This effectively takes them off the market and the money goes toward funding further emission reduction projects. We are not currently able to actually power our servers with the wind or the sun, and this is the next best thing!

The market for purchasing voluntary offsets is blowing up right now, and as such it’s a bit of a wild west.

You can find “offsets” for as little as ~$5 at ton all the way up to $30. At the low end you risk purchasing an offset that is not actually doing what it says.

To ensure we got the good stuff we worked with The Green Office to obtain credits with true value that aren’t considered “trash tags”. We learned that our ecological footprint is 30,600 acres and we generate 2725 tons of CO₂ per year. That’s about the same amount as 545 average homes!

We’ve offset that by doing a few things.

We’ve purchased Renewable Energy Credits (RECs) certified by Green-e. RECs don’t offset electricity-related emissions, but combined with the purchase of regular electricity they do help us to avoid them.

We’ve also purchased carbon credits certified by The Gold Standard. These credits come directly from reductions that meet standards established by the Kyoto Protocol and offset emissions from projects that would otherwise not exist.

We’re also looking for more ways to reduce our resource usage around the office and the data center. We’ve switched to using coffee cups made from fully renewable resources. In the last couple of years we’ve also been deploying many more servers with notoriously power-efficient AMD processors. As processor and server technology continues to evolve we’ll follow the path of power efficiency. It just makes sense!

And because Los Angeles is so large we’ve also got two offices on either side of the expanding megalopolis to ensure our employees spend less time sitting in traffic and more time not polluting.

“Big deal, DreamHost! Yahoo’s already carbon neutral! You missed out on the PR train to Happytown.”

Yeah well, Yahoo’s cool and all…but they’re not carbon neutral YET and so their train is broken down between Happytown and Pollutiontown! They’re still brown! Here’s a clown!

It only took us three days to go from “Hey let’s do this” status to “Hey it’s done!” status. Three days and several thousand dollars, but that’s another story. You don’t have to be a tree-hugger to appreciate the value of renewable energy and you don’t have to charge a premium for your services to afford it, either.

Every single DreamHost customer now benefits from our carbon neutral status. If you’re a DreamHost customer trying to reduce your footprint on the environment, rest assured that your hosting is taken care of! You are welcome!

Maybe you want to brag about how green your hosting is. We can hardly blame you! We’ve made some images below to help with that.

If you run into any of these icons in your web travels, click them! You’ll be taken to a page that’ll confirm (or deny) that the site is hosted with us and is therefore green!

If you’re a DreamHost customer and want to add these icons to YOUR site, just visit the “Home > Green Hosting” section of your DreamHost account control panel for linking instructions.

We may not be the first green web host out there but we’re probably the largest. Maybe we can encourage the whole industry to do it. It’s not really that expensive and it’s worth it!

PEACE OUT!

Have you checked out https://panel.dreamhost.com/ in oh, say, the last five minutes?!

After more than six years of the current panel, we thought it was maybe time for a little update..

Really, all that’s changed is:

1. it’s got a hawt new “web 2.0″ look.
2. we’ve css-ified everything.
3. we’ve added a TINY bit of Ajax.

Functionality-wise for you, the Happy DreamHost Customer, not all that much has really changed.

YET!

But, getting all our various panels standardized and modernized was a huge first step in being able to make lots more little improvements across our panel.

More features? Faster load times? More AJAX?

…who knows!

ANYTHING IS POSSIBLE NOW!

And it’s all thanks to Brett!

P.S. Feel free to post comments/critiques/bugs in the comments here, or over at this thread in our forum!