We have always worked hard to provide the best hosting service you can get for the money and we’ve been pioneers in offering many unique features such as Jabber IM, WebDAV, Subversion, Phusion Passenger, and our One-Click Installer for popular web applications.  We love to play with all the newest Internet toys and provide them for you to play with also.  Our attention to all the “shiny new things” can sometimes distract us from the more fundamental aspects of our service, though, and that’s why…

The DreamHost New Year’s Resolution for 2010 is a renewed focus on performance and reliability.  Over the last few years, we have grown in leaps and bounds and our customers’ expectations of our service have grown along with it.  We have not always kept up with those expectations, and we need to put some energy into improving that.

Yesterday, I spent some time looking back at the past few years.  We’ve had our share of ups and downs and many of our loyal customers have stuck with us through it all.  We couldn’t ask for a better group of people to work for than you.  So, looking forward, we’re going to update our methods and systems to provide a more consistently high level of service.

Here’s some of the areas we’ve already improved on, and things we will be doing over the next several months…

• We are now using a more flexible storage system so we can easily keep up with your awesome demand for data storage.
• Additionally, our open-source next generation file system called Ceph is coming along nicely and will provide a great platform for our network storage needs in the future.
• A lot of work has been done over the past year on optimizing and better organizing our network infrastructure.  Everything being done is laying the groundwork for major future improvements in the works.
• By purchasing a major interest in Alchemy, our data center provider, we can rest easy knowing the relationship will be mutually beneficial for a long time.  Our system administrators are very happy to know that we will not be moving data centers again for the forseeable future.  We have plenty of spare data center space available and will be able to grow capacity at a comfortable pace.
• Our network connectivity has always been redundant but we’re going to kick that up another notch this year as we go multi-homed.  Once this is in place we’ll be able to withstand the loss an entire network point of presence without losing connectivity.
• We developed an internal system to help us manage individual server stability by improving communication and reducing wasted effort.
• We have deployed Ksplice Uptrack across our system, making mass server reboots for kernel security holes a thing of the past.  The Linux kernel has been hit with a couple of high profile security exploits in the past few months and we found ourselves scrambling to patch and reboot quickly.  Never again!
• We have a new mantra, “test, test, test”.  Some amount of testing has always been part of our procedures, but we are now making testing a fundamental component.
• Increased focus is being put on optimizing our procedures at all levels to reduce duplicated effort and errors, while also increasing accountability.  This is not something that happens overnight and it will be an ongoing process.

Having now solved our most distracting problems from the recent past, DreamHost is now positioned to tackle the future better than at any time in our history.  It won’t be easy but we have the best team in the business and a few new tricks up our sleeves on top of it.  And with that, we wish you all a very happy new year and a great 2010 (twenty-ten!).

As the end of the decade approaches, it’s a perfect time to look back and reflect on the past.  What has gone wrong and what has gone right?  We have a way of making waves, even if not always in the way we might like.  Here’s some reflections on some of those waves.

Power

Starting about 4 years ago, we were battling a problem of power constraints at our data center.  That led to a general inability to provision data center space the way we wanted to and we had to become very creative to manage our system.  The biggest problems from that time were two unplanned power outages of the entire building followed by an emergency planned outage 8 months later.  Power is one of the lifelines of a data center and not being able to rely on it can be a major distraction.  It was during that period that we established our off network status site (dreamhoststatus.com), which has proven to be a great asset.  Those issues are long behind us now, and looking forward we have plenty of power capacity.

Network

Pretty much in the middle of the power situation we were also hit with a networking problem between our two core routers that was causing serious website slowness.  We had grown so quickly that nobody really understood exactly how the network devices were interacting.  That combined with the distraction of the power situation made it take way too long to resolve the problem.  From that, we learned the hard way what we needed to do to improve and maintain the network.  Things have improved, but we were still recently hit with a couple network outages.  They were caused by human errors and weaknesses in the procedures we had in place.  We have already refined those procedures and future improvements to the networking infrastructure will also help to mitigate the potential for human error to result in outages.  This is still very much a work in progress but huge steps have been taken already and more are still to come.

Data Storage

The next major hurdle we faced involved our data storage infrastructure.  In the early days we had migrated from disks inside each server to network attached storage for the added redundancy and to allow us to better utilize our available storage.  At that time hard drives were only 9 gigabytes and our users were generally only using a hundred megabytes or less of space.  Boy, have things changed!  We now have users with multiple terabytes of data and even everyday websites sometimes have multiple gigabytes of files.  The huge growth of online video and the popularity of digital photography increased the demands on our storage infrastructure by a couple orders of magnitude over the years.  To accommodate that growth, our simple system of file servers had grown to a large network of over 100 individual servers that we had to constantly juggle data between.  The cost per gigabyte for our Network Appliance based system had also not come down nearly as fast as the per-user storage requirements had gone up, but we had become reliant on network storage for things such as backups, rapid recovery from server failures, and seamless sharing of data between separate hosting and email accounts.  We were addicted to network storage, and our next couple of major performance problems came from gloriously failed experiments with other storage products.  One of them was cheap and unreliable, and the other was expensive and unreliable.  We couldn’t win!  (Note that we haven’t used either of those for awhile so both of them may work better now than they did for us a couple of years ago.)

Through those years that our addiction to network storage had developed, a shift had happened that we hadn’t noticed.  First, individual hard drives had dropped like a rock in price and skyrocketed in capacity.  Second, users were consuming data at such a rate that evenly utilizing our available storage was no longer a problem.  Switching back to locally installed storage was the answer we had been looking for!  We started experimenting with the new server architecture and developing a backup strategy.  Then once the pieces were all in place, we started moving forward with it.  So the storage bottleneck was resolved and we had a clear path forward, but we were still throwing out a lot of knowledge we had learned over the preceding decade and were starting over from scratch in a lot of ways.  With every technology shift comes with it a new set of problems and this one was no exception.  It’s been about a year and a half since then and we’re already on our fourth revision of the server configuration (three different RAID cards with two different configs).  The current hardware has been working out quite well but we’re doing some testing to see if it can be optimized further.

Looking into next year, our core technology systems are under control and we’ll be able to focus more on improving the service than we have in years.  The future looks very bright…. but more on that tomorrow!

I’m a big fan of holiday advent calendars where you open up one of the doors each day in December and get some sort of little prize as you count down the days to your favorite winter-time holiday.  In fact, last year I asked my wife to get me one… and she did!  She filled it up with all kinds of chocolatey goodies and I was a happy days-until-my-favorite-winter-time-holiday-counter-downer all through most of the month of December.  So, sort of like that, but not every day and not as chocolatey and not really actually holiday related, we have something new for you! …  Slick integration with the Chartbeat real-time analytics service into the DreamHost Web Control Panel.  Chartbeat provides real-time information about the visitors to your website, including what they are doing, which pages they are visiting, and what parts of the pages they are looking at.  It also goes beyond that and incorporates information from other parts of the web like twitter so you can gain even more insight into where visitors are coming from and what they’re saying about you.  I’ve been using Chartbeat for this blog for the past month or so and it’s actually made boring old web stats fun and interesting again.  It’s a unique look at your visitors that I had never seen before.  The iPhone app with push notifications is slick, too.

Chartbeat is a paid service that you can order straight from the DreamHost Web Control Panel, and there’s a 30 day free trial so you can give it a go and see what you think.  The Chartbeat team would love to hear comments from our users so give it a try even if you don’t think you’d end up using it.  You may be surprised to find you use it more than you thought you would!

For some of you, it’s that time of the year again.. holiday shopping!

I say some of you not because some of you are not Christian, Jewish, African, or employed, but because some of you are not early holiday shoppers.

For those of us already finished with our seasonal purchases, it’s that time of the year again.. holiday sleeping!

Now, when I was young, early holiday shopping meant at least ONE of the big boxes under the tree wasn’t filled with an “I O U: 1 Cobra Terror Drome – SANTA!” But nowadays, thanks to the influences of my lovely spouse, I’ve been able to get all my shopping (apart from gifts for her) done E A R L Y.

What’s my secret?

It’s easy!

She gets everything! Her family, my family, her friends, my friends, everything! The only thing I have to get IS my gifts for her!

And man, she starts EARLY. Like, it’s an all year thing. If she sees something that would be perfect for my brother in May, BOOM she buys it for Christmas. If she finds a sweater on sale in February that would look great on her mom… POW bought and stored. If she comes across an amazing deal on Zhu Zhu Hamsters December 26th.. well, they’ll probably still be popular next holidays too, right?

It’s weird. I mean, compared to how I was brought up (which I’m 100% sure is the norm) it’s down-right un-American. But, I’ve slowly come around. Sadly, I’ve started to find myself buying things in the spring with an eye towards December. And not just me.. DreamHost too!

Remember in the last post when I mentioned how we bought (some of) Alchemy Communications back in May? Well, here it is, December, and that forward thinking is finally starting to pay off… for you!

In honor of our new relationship (and by using ex-DreamHost shared hosting servers) Alchemy has just started offering Dedicated Servers at pretty much the best deal you can get anywhere (we checked!):