Behind the Scenes of a botched Ruby on Rails upgrade

As many of you noticed last night and this morning, our attempted upgrade of Ruby on Rails from 1.0 to 1.1 did not go as well as we had hoped. We have now reverted back to the stable 1.0 release while we spend some time testing. A few different factors had a role in causing this mishap including the ever present ‘human error’.

The Ruby on Rails software is really 6 or so separate pieces of software that all must have the correct versions installed. If one of the pieces is not properly installed the whole thing breaks. That in itself is not much of a problem if you use the recommended ‘gem’ tool to upgrade the software, handling any software dependencies for you along the way. A lot of Ruby software is distributed in these little packages called rubygems (clever, eh?) and the gem tool is used to manage them. The gem and rubygem system is somewhat like an improved version of Perl’s CPAN or PHP’s PEAR, if you are familiar with those. This is all well and good but we don’t rely on the gem and Rubygem system to handle our Ruby software upgrades for us. I’ll step back in time a little to explain why.

Several years back as our fleet of servers started to grow to be somewhat large we realized we could not continue to manage each server independently. To deal with the situation we developed methods and procedures to manage our servers as groups keeping the servers in a group all configured and set up as identically to one another as possible. Debian’s excellent apt-get software package management system was the basis for the software installation side of things, and still is today. We rarely distribute any software to our hosting servers without putting it into a Debian package and anything that will be installed on more than one server must be packaged up first. That includes all perl modules, php software, and (you guessed it!) rubygems.

As things are now those 6 or so individual pieces of software that form Ruby on Rails are all individual Debian packages and they must all be upgraded simultaneously, preserving the proper dependencies. We have coded the dependencies into the Debian packages themselves to make it a little easier, but they are still separate packages that must be upgraded and built separately. In hindsight it may not have been the best decision to put them into separate packages and that is something we will likely change now! So, the human error part came into play when two of the upgraded packages were not built correctly. Human error is the component of any technical problem that’s simultaneously the easiest and hardest to correct. You don’t have to change anything to correct it but you can also never really be sure it won’t happen again. In this case it will take plenty of time to get the egg off of our faces so we’re pretty sure that will cement things in our heads plenty.

Unfortunately, even if we had done a perfect job upgrading the Ruby on Rails software, a lot of websites still would have broken. It seems as if the Rails developers’ claims of an easy upgrade were perhaps a little over-enthusiastic. However, we still can’t just wait around until every Rails user we have updates their code. We have plenty of customers who would rather we do upgrade Rails as soon as possible after every new release even if it might break their website just so they can have access to that ever tempting ‘bleeding edge’. Ruby on Rails is still in a very rapid development cycle with frequent updates with hundreds of changes and it is somewhat likely that the next few upgrades may cause some small problems as well.

Fortunately, there is an easy solution!

As of a couple of versions ago, Ruby on Rails added the very nifty ability to freeze your Rails application at the version of the Ruby on Rails software currently installed system-wide. Once that’s done any changes to the software on the server (ie, changes done by us!) will not affect your application. This is highly recommended for any applications running on a shared hosting platform or any other setup where you do not have control over the Ruby on Rails installation.

Here’s how…
From the application directory, run rake freeze_gems. That will copy all of the Ruby on Rails software (that does not include other rubygems!) into a local area. It must be done for every Rails application independently. When you want to switch back and use the version of the software installed system-wide you just run rake unfreeze_gems.

So, everyone go freeze your applications now!

New Rails

As I write this, the latest version of Ruby on Rails, version 1.1, is being installed on all DreamHost servers.

When I asked Josh if I could post to the blog about this, he asked me, “Is it exciting?” Pfah! Of course it’s exciting!

For one thing, they’re bragging over at the RoR Blog that the update has over 500 “fixes, tweaks, and features.” I call that pretty darn exciting, and I knew our DreamHost customers who love Ruby on Rails would be excited to get their hot little hands on those changes as soon as possible. The big one that I know will get RoR fans all a-quiver is Ajax. I’ve heard some pretty excited talk about Ajax, and now RoR fans can get their Ajax on.

Additionally, another blogger has published a list of lesser-known features that may not be as exciting as the major changes, but are a nice addition to make life a bit easier for RoR developers.

So maybe nobody will be dancing in the streets over this one, but who knows, maybe a beer or two will be tipped.

Like hammer for sidekick.

We go through a lot of sidekicks and some never get returned for one reason or another. We had some fun with one of those un-returned sidekicks the other night.

Sneaky Changes Afoot..

We’re making some changes to the way we do MySQL…

One of the most popular suggestions we have is to be able to have multiple databases per hostname. Along with that, people want to be able to use the same mysql username to access multiple databases. To top it off, they are even so bold as to want one PhpMyAdmin management area for all the databases on their account!

This hasn’t been possible with our system since very near the beginning, because our customer database servers are separate from our web servers. That’s why you can’t use “localhost” to connect to your database. It is not why your database may sometimes seem slow. The reason for that (if it happens) is because the database server itself is overloaded (or maybe dns was messed up). And actually, if it had been the same physical machine as your web server, that would have also meant all the websites on that server would be slow too (not just the ones accessing that database server). SO THERE!

But that’s not really why we kept our database servers separate from our web servers. The main reason is we mount the file system on our web servers over NFS, and that is just noooo good for MySQL performance. The other reason is we do performance tweaks on our MySQL servers that we wouldn’t want to do on Apache servers.

Now, when you’ve got your database servers separated from your web servers, you’ve got to have some way to determine which database server new databases go on.

The simplest thing, I guess, would be to just assign database servers to particular web servers.

If we’d just done that, everything people are asking for now would have always been possible!

But, there is one teeensy drawback with doing it that way. It’s not the most efficient use of hardware.

And when you’re a self-funded, completely independent, low-price, big-feature web host, efficient use of hardware is pretty much the difference between driving a ferrari and driving a limo. Let me explain (not the “driving a limo” part)…

Here are the three essential facts for your consideration:

1. Each database server has a maximum number of databases it can support.
2. Customers continue adding databases gradually for the life of their account with us.
3. Moving databases between servers causes some downtime and is a big pain in the admin’s behind.

So our solution to these three facts was to just make a current “active database server.” Any new MySQL created by a customer, regardless of their home web server, went on it, and when it filled up, we just added a new one.

That works great, and is pretty much the maximum efficiency you can get in terms of hardware use.. at any given time you only have one non-full database server, and it’s in the process of filling up pretty darn fast!

The only thing that doesn’t work great is one person’s databases are most likely spread across multiple servers… which means you need a separate hostname for each one, and most importantly, you can’t do all those things you Happy Customers clamor so loudly for! It also means if one database server poops its chute, a huge swath of customers are affected instead of just the ones on web servers tied to that database machine.

But still, it was the cheapest and easiest way to do things!

However.. we’re rich now and we like a challenge!

So, we’re in the process of changing our system to start assigning each new customer to a database server for life! What this means though, if we don’t want to move databases later (see fact 3 above), is we have to essentially “cut off” a database server before it’s “full’ (see facts 1 + 2). And that means add another database server sooner than we would otherwise have to.

But it also means all your databases will be on the same server, which means soon you WILL be able to manage them all from one hostname (and one username). It will also mean there’ll be less chance of your databases being affected should a random mysql server have problems. In fact, it also also means if you’re a heavy MySQL user and causing problems for your server, you’ll mostly be affecting your own performance and not too many unsuspecting neighbors!

It’s more expensive, but you guys are worth it!

The Four Horsemen of Marketing.

“Visibility, Quality, Fashion, and Price”

I’m not sure why I’m calling this the four horsemen of marketing. Apart from the fact it lent itself to making a silly picture. Which is not really a good reason.

You see, this article is not about horsemen at all, but rather the four critical ingredients to any companies marketing efforts. And those horsemen you see above (Apple, Coke, Honda, and Wal-mart) were chosen not only for the relative roundness of their logos, but because each of them has focused on a different ingredient.

That’s not to say any of them ignored the other three!

When it comes time to make a purchasing decision, customers will always pick the solution that provides the greatest personal value they can afford. In determining this value, all four elements will come into play, each weighing varying amounts based on the specific product and the specific customer. Of course, each of the critical ingredients is just that… critical. If any one of the four is below a mystical level in your customer’s head, your total value has just dropped to ZILCH, and there will be NO SALE.

All four must receive at least passing grades for your company to survive… but in your customers minds you will only ever be associated with one. So let’s break em down:

Visibility: To the layman, (Required legal disclaimer: I have no training in marketing!) the goal of marketing IS visibility, period. Actually, marketing is a lot more than that.. marketing is everything your company does to make your product or service something a customer would actually decide to buy. Visibility is of course critical to this. It doesn’t matter how shiny your surf wax is, if nobody knows it exists, nobody can decide to buy it.

If your company decides to focus on visibility above all else, you’re going the Coca-cola route. You’ll spend a lot of money on advertising, FOREVER, and your brand will be eminently recognizable. What that brand actually represents in people’s minds may not be clear, but hey, at least it’s in their mindsl! And that’s a huge part of the equation.

Quality: It’s not just an engineering problem. If your product does not meet some minimum standard of quality (reliability, ease-of-use, functionality, etc..) for whatever it is your customers expect it to do, no amount of the other three horsemen can save it. You may flounder around for years, keeping your head above water as your Public Relations team hates their lives, until you inevitably beg God herself for the sweet release of bankruptcy.

Now, if your brand is synonymous with quality, you’re most likely a niche player. Because honestly, really high quality is generally not worth the price to the mass market. But don’t worry, you don’t need perfect quality to survive.. you just need enough quality. (See Microsoft.) Honda is an example of a company that is all about quality, and they’ve done well because they’re in a market where customers believe quality is important (or else they die)!

Fashion: Really, fashion shouldn’t be a “critical” ingredient in that spicy Italian meatball that is marketing. And if your customers are robots or Vulcans, it isn’t.

If not, it is. It doesn’t matter how cheap, great, and well-known your product is.. if nobody would be caught dead using it, you’re better off selling matches in a volcano. To the devil. During a heat wave. In Nevada. In August.

Fashion is intriguing though. In some markets, it is the least important aspect (though still necessary!), but in others, it is the very sword you will live or die by! If your company is known above all else as fashionable, you will be able to command a higher price than others of a similar quality and visibility. Which leads to great profits… because fashion, as elusive as it is to attain, once attained, is a relatively inexpensive quality to maintain!

Apple has done a great job in the last few years of making their consumer electronics products not only of a passable quality and price, but also of intense fashion appeal, which has done wonders for their stock! The scary thing about relying on fashion is it’s so ephemeral.

Today’s iPod is tomorrows Member’s Only jackets and baggy pants.

Price: No big tricks here.. if your stuff is too expensive, nobody will buy it. Of course, price is not value! $30,000 for a Ferrari is a better value than $25,000 for a Fiat. You see, by working on the other three ingredients, you can raise the level where the price for your product passes that invisible “no longer worth it” boundary for your market.

If your company decides to compete on price above all else, the conventional wisdom is, you’re screwed! Pick something else! No companies ever make it big competing on price! Nobody like say McDonalds, Wal-Mart, or Southwest Airlines.

Oh, I’m sorry, was that too much IRONY?

It’s true, competing on price is probably the hardest ingredient to compete on. As companies get bigger, it’s easier to offer cheaper and cheaper prices.. and the odds are, if you’re reading this, you’re not a very big company. However, the rewards are also the greatest. You see, most people are poor (Or at least cheap bastards.)

And as I said before, people decide on the GREATEST PERSONAL VALUE THEY CAN AFFORD. So even if you’re not offering that great a value, if your price is low, you’ve opened yourself up to a whole puck-load of people who just can’t afford anything better.

And if your overall value isn’t actually that great, it probably means you’re able to make a pretty healthy profit out of each sale. And since there are soooo many cheap customers in the world, you’ll probably make a lot of sales. You might be universally reviled for your junky products and cut-throat business practices, but you’re universally reviled all the way to the bank.

An exercise for the reader!

So really, every brand out there has some combination of Visibility, Quality, Fashion, and Price that makes them tick. Here are how various brands strike me personally… feel free to discuss and add your own in the comments area (especially your thoughts on DreamHost!

Coke: Visibility, then Fashion, then Price, then Quality.
Honda: Quality, Price, Fashion, Promotion.
Apple: Fashion, Quality, Visibility, Price.
Wal-Mart: Price, Quality, Visibility, Fashion.

DreamHost: Visibility, Quality, Price, Fashion..?

And that’s the way we like it!

What You Get When You Work Here…

…is, as often as not, a whole lot of abuse.

Take, for example, our fearless Tech Support team leader, Ralph.

Every month all of the team leaders go out to lunch and goof off talk about important company stuff, leaving the rest of us poor plebs in our respective offices to work. Lately, though, the queue has been pretty well managed (thanks to all of our stunningly efficient new hires who toil away as the old guard takes a nap supervises). You know what they say about idle hands being the devil’s tools, right? Well, despite the fact that Ralph is a pretty nice guy, some brave souls in our Brea location took it upon themselves to make some, erm, changes to the front of Ralph’s office.

I, of course, had absolutely nothing to do with it.

This isn’t the first time such weirdness has been afoot. I made the mistake of taking a two week vacation once only to find that the six foot tall shelving unit in my office had been flipped over. Another prank involved an unsolicited modification to the blinds in my office window. I caused a bit of annoyance in my would-be torturers by not bothering to touch my blinds for several days (unbeknownst to me, if I had they would have all come off and fallen to the ground). Then, of course, there’s the time that Brett forged a fake $500 ticket and affixed it to my car’s windshield - I was grabbing my keys to go raise hell at the court house before I was told it was a fake.

If you ever get a chance to work here - well, I won’t lie - DreamHost is a ton of fun to work for. However, be prepared to fall victim to all sorts of wacky hijinks.

BTW: For those of you who have followed the exploits of our friend Ted, there is a new development. Ted now has what we like to call “empathy caps” - little orange caps that go on either end in order to “soften the blow” for lesser offenses. Of course, we leave those off for those who really should know better, like Javier.

Avis Bandwidth Rental

It’s just too cold here in L.A.

Which is why I just got back from vacation in Hawaii.

The weather was nice (though wet), and ironically, I drove more in a week in Hawaii than I do in an average month here in L.A.

And, every rental car place we used always had three “options” for paying for gas.

#1. Pre pay for a tank. Pay only $2.19 / gallon and bring it back empty!

#2. You fill it up. Bring your tank back full (Average gas price on the island: $2.89).

#3. Pay us to fill it up. When you return, pay $5.09 / gallon for any used gas.

There were three things that struck me about this.

Firstly, #2 and #3 are the same frickin thing! All they really need to say is “Bring it back full or pay us $5.09 / gallon, fool.”

Secondly, do ALL rental car companies really need to bend their customers over so bad? It’d be nice if they all just had one option: “Bring it back full or pay us $2.89 / gallon.”

Finally, and this is the big one, you only save money with option one if you bring the tank back less than 1/4 full.

Because otherwise you’ve paid more than $2.89 / gallon for the 75% of the tank you used. And considering that they put the value of themselves filling a tank up about $2.20/gallon (they charge $5.09 for it), they must make a profit of about $2.20/gallon for every tank they have to fill up under option one. Which implies the average person who chooses option one comes back with a tank only 43% empty… and are effectively paying $5.09 per gallon anyway! SUCKERS!

Of course, if YOU know you’ll definitely come back with less than a quarter tank (or you don’t like the hassle of filling up your tank before returning to the airport) option one is fine for you.

DreamHost likes option one!

You see, upstream providers are a lot like rental car companies. (Don’t get the insurance!)

Right now, we’ve got an offer on the table to pay $26/mbps a month for an unmetered gigabit line (i.e. “pre-filling the tank”) or $37/mbps a month for actual usage.

For the $26,000/month option to make sense, we’ve obviously got to be able to average over 700mbps on the gigabit line. Which wouldn’t be too hard.. except for the fact things start to get grungey if you start pushing above 850mbps.

Lucky us, right now we’ve got almost exactly 1.7 gbps in over-commit (that is, we’re doing 1.7gbps more than the minimum amount we pay for no matter what) with our current providers. So the plan is to take 2 full gig lines from the new provider, and immediately put 850mbps on each of them!

Option one it is!

But when it comes to Avis, I’m an option two guy till I die!