DreamHost Blog » Behind the Scenes of a botched Ruby on Rails upgrade

Behind the Scenes of a botched Ruby on Rails upgrade

March 29, 2006 on 3:03 pm | In Foobars, Insider View by Dallas Kashuba |

Rails Logo As many of you noticed last night and this morning, our attempted upgrade of Ruby on Rails from 1.0 to 1.1 did not go as well as we had hoped. We have now reverted back to the stable 1.0 release while we spend some time testing. A few different factors had a role in causing this mishap including the ever present ‘human error’.

The Ruby on Rails software is really 6 or so separate pieces of software that all must have the correct versions installed. If one of the pieces is not properly installed the whole thing breaks. That in itself is not much of a problem if you use the recommended ‘gem’ tool to upgrade the software, handling any software dependencies for you along the way. A lot of Ruby software is distributed in these little packages called rubygems (clever, eh?) and the gem tool is used to manage them. The gem and rubygem system is somewhat like an improved version of Perl’s CPAN or PHP’s PEAR, if you are familiar with those. This is all well and good but we don’t rely on the gem and Rubygem system to handle our Ruby software upgrades for us. I’ll step back in time a little to explain why.

Several years back as our fleet of servers started to grow to be somewhat large we realized we could not continue to manage each server independently. To deal with the situation we developed methods and procedures to manage our servers as groups keeping the servers in a group all configured and set up as identically to one another as possible. Debian’s excellent apt-get software package management system was the basis for the software installation side of things, and still is today. We rarely distribute any software to our hosting servers without putting it into a Debian package and anything that will be installed on more than one server must be packaged up first. That includes all perl modules, php software, and (you guessed it!) rubygems.

As things are now those 6 or so individual pieces of software that form Ruby on Rails are all individual Debian packages and they must all be upgraded simultaneously, preserving the proper dependencies. We have coded the dependencies into the Debian packages themselves to make it a little easier, but they are still separate packages that must be upgraded and built separately. In hindsight it may not have been the best decision to put them into separate packages and that is something we will likely change now! So, the human error part came into play when two of the upgraded packages were not built correctly. Human error is the component of any technical problem that’s simultaneously the easiest and hardest to correct. You don’t have to change anything to correct it but you can also never really be sure it won’t happen again. In this case it will take plenty of time to get the egg off of our faces so we’re pretty sure that will cement things in our heads plenty.

Unfortunately, even if we had done a perfect job upgrading the Ruby on Rails software, a lot of websites still would have broken. It seems as if the Rails developers’ claims of an easy upgrade were perhaps a little over-enthusiastic. However, we still can’t just wait around until every Rails user we have updates their code. We have plenty of customers who would rather we do upgrade Rails as soon as possible after every new release even if it might break their website just so they can have access to that ever tempting ‘bleeding edge’. Ruby on Rails is still in a very rapid development cycle with frequent updates with hundreds of changes and it is somewhat likely that the next few upgrades may cause some small problems as well.

Fortunately, there is an easy solution!

As of a couple of versions ago, Ruby on Rails added the very nifty ability to freeze your Rails application at the version of the Ruby on Rails software currently installed system-wide. Once that’s done any changes to the software on the server (ie, changes done by us!) will not affect your application. This is highly recommended for any applications running on a shared hosting platform or any other setup where you do not have control over the Ruby on Rails installation.

Here’s how…
From the application directory, run rake freeze_gems. That will copy all of the Ruby on Rails software (that does not include other rubygems!) into a local area. It must be done for every Rails application independently. When you want to switch back and use the version of the software installed system-wide you just run rake unfreeze_gems.

So, everyone go freeze your applications now!

28 Comments

1

Thanks for the full explanation.

Any idea when real upgrade will happen, now?

Comment by Rob Sanheim — March 29, 2006 #
2

So, can I recommend that you *test* this stuff before you try pushing it out? Seems like even a minimal test upgrade on your spare server would’ve revealed these problems. And what about posting this freeze_gems routine to the blog a week *before* you upgrade, to give people time to stabilize their sites?

I don’t mean to come down hard on you for enthusiastically keeping up with the cutting edge of Rails — I think that’s great and it’s one of the reasons I stick with Dreamhost. Nobody’s expecting you to do extensive regression testing and your own 6-month security study. But at least give us the basics — a bit of testing and advance notice with minimal information on how to manage the upgrade with regards to our own applications. I would appreciate it!

Comment by Joe — March 29, 2006 #
3

rake freeze_gems is failing on my install of Typo 2.6.0 — is this because Typo is old or a problem with my setup?
This is what I get when I run rake in the root directory of Typo:
[zebes]$ rake –version
rake, version 0.7.0
[zebes]$ rake freeze_gems
(in /home/.jeno/turnlav/typo)
rake aborted!
Don’t know how to build task ‘freeze_gems’

Comment by Alex — March 29, 2006 #
4

[...] Updated again: DreamHost has the full explanation of the problems yesterday in their blog. [...]

Pingback by Unofficial DreamHost Blog » Blog Archive » Rails upgraded to 1.1 — March 29, 2006 #
5

[...] Ruh Roh George .. Dreamhost seem to cop alot of crap but apart from the usual niggly things here and there, I’ve had no reason to complain, and in fact regularly recommend them to friends. [...]

Pingback by oli young » Blog Archive » DreamHost Blog — March 29, 2006 #
6

Dreamhost, I have nothing but respect for you posting this blog and explaining what happened. I was fornuate enough to have read an article posted by the Rails developers about how important it is to “freeze” your application when on a shared host.

http://weblog.rubyonrails.org/articles/2005/12/22/freezing-your-rails-when-you-deploy-shared

So, I never experienced any problems when Rails was upgraded to version 1.1. :)

I do want to also say thank you for being so proactive in software upgrades, this is one of the things I love most about DreamHost.

The only thing I could ask more from DreamHost is to start offering dedicated servers again. I have 3 seperate shared hosting accounts (1 Code Monster & 2 Crazy’s) at DreamHost and desperately need a dedicated server. Please oh please start offering dedicated servers soon (and if you could upgrade the server specs, that would be AWESOME).

Thanks DreamHost

Comment by George — March 30, 2006 #
7

The day before I was thinking, how long would it take Dreamhost to upgrade RoR? I figured a month or so. Not a couple days. This was a good lesson for us all. I had a client site affected, but I loaded a version freeze and in a matter of minutes and the app was working fine. I knew for a while it was something I should be doing, particularily in a shared environment, but was to lazy to do.

My apps are now be upgrade proof, so you can procede. ;-) Kidding of course. Do you have an upgrade timeline? I look forward to it. This new version has some kick ass features.

And thank you for not posting a video of tossing eggs off your building roof before explaining this.

Clint

Comment by Clint Pidlubny — March 30, 2006 #
8

I completely agree with George about Rails and especially dedicated servers.

I am in desperate need to get 4 dedicated servers and would love DreamHost to be my provider. I’m not sure how much longer I can wait before I have to go with another web host :(

And as George said, when you do start offering the dedicated servers again - it would be great if you could bump up the specs of the dedicated servers (CPU, RAM, and disk).

Comment by Henry — March 30, 2006 #
9

My view is that DH should upgrade RoR soon after releases (albeit properly, of course), and not worry too much about users’ stuff breaking. If their apps have any features that are tied to a specific version of Rails, they should just make sure that version is in their /vendor/rails directory. It is, after all, their responsibility.

In short, I hope DH won’t use this as an excuse to wait several ages to upgrade their stuff in future.

PS: when will we see ruby 1.8.4 ?

Comment by telos — March 30, 2006 #
10

I’m also glad for the quick upgrade and hope that DH keeps with that practice. It may be worth them testing against a stock install of Typo as that’s about the only Rails app that is widely used.

The question came up to me that if we all freeze gems (or a similar step) then what is the meaning of DH ’supporting’ Rails? Well, it’s the FastCGI primarily, but also you can use the stock Rails for quick little apps that aren’t mission-critical (nobody’s lunch is depending on them) which is convenient.

At any rate, the lesson learned is valuable.

Comment by Tom Wilcoxen — March 30, 2006 #
11

Same here.. I’m forced to go with another host provider now for dedicated, shared hosting isn’t enough for me.. I need dedicated.. do you have any suggestions? DH please have a priority list for dedicated hosting!
[quote]I completely agree with George about Rails and especially dedicated servers.

I am in desperate need to get 4 dedicated servers and would love DreamHost to be my provider. I’m not sure how much longer I can wait before I have to go with another web host :(

And as George said, when you do start offering the dedicated servers again - it would be great if you could bump up the specs of the dedicated servers (CPU, RAM, and disk). [/quote]

Comment by peter — March 30, 2006 #
12

@Peter

I with you and the others, DH pleaaaaaaaaaaaaaaaaaaaase bring back dedicated servers SOON. I needed 2 dedicated servers really bad and as the others said, upgrading the RAM and disk would be a huge benefit too.

Comment by Jay — March 30, 2006 #
13

Me too! I would like Dedicated Hosting please! Make that 2… Pronto you Head Honcho :) Q3 is wayyy too late…

Comment by Fred — March 30, 2006 #
14

I tried running ‘rake freeze_gems’ but I got an error. When I ran ‘rake -T’ the rake task isn’t even available. Has anyone been able to do this?

Comment by Brian Sweeting — March 31, 2006 #
15

Do you folks have a dedicated QA Tester on staff? I’ve noticed you’ve had a lot of problems rolling out new software or making mail infrastructure changes. We have a small independent testing team our self; it has saved us a lot of grief and loss of customers. I know some smaller and less mature organizations resist having a tester but it certainly pays off.

Comment by Alex — April 1, 2006 #
16

I can only hope that these comments about leasing dedicated servers gets passed along within DreamHost to the appropriate people.

It sounds like many people are loyal DH customers but need a dedicated server now, like me, and are having to look to other web host providers :(

If you could start leasing dedicated servers in the next 2-3 weeks, I could hold off. But I can’t wait any longer than that.

For what it’s worth, the company I work for needs 6 dedicated servers and I would love to give DH that business (also long as the server specs also increased) if DH starting offering leasing by the end of the month.

Comment by Sara — April 2, 2006 #
17

Ruby on Snails

Comment by Jon — April 2, 2006 #
18

[...] Trying to use Typo after my host updated RoR (after a lot of angry people complained, Dreamhost reverted back to the previous version) [...]

Pingback by Mesén Labs » Blog Archive » Living with Typo and Ruby on Rails — April 2, 2006 #
19

Netpoint…

Selline uus koduleht avati, vist täna või kunagi hiljuti. Sümpaatne firma, kelle kodulehel veetsin rohkem kui ühelgi teisel .ee saidil (v.a uudistes surfamine) end hiljuti mäletan. Kuigi nad ei kasuta tekstis kuskil sõna “blogi”, on s…

Trackback by Jaanuse asjad — April 5, 2006 #
20

Not a comment on this post. I just wanted to say how much the new web-based FTP thingie totally ROCKS!

right on dudes. :)

Comment by the running blogfather — April 6, 2006 #
21

http://weblog.rubyonrails.com/articles/2006/04/06/rails-1-1-1-fixing-a-slew-of-minors-but-you-must-still-freeze-typo

“This is the release we recommend that hosting companies upgrade to. If you still haven’t frozen your application and it for some reason doesn’t work with Rails 1.1.1, don’t run crying to them. We screwed up in the release notes of the last release by not telling people that Typo would break, but now that this information is spread far and wide, it’ll rest on your shoulders to make sure you’re frozen and stay cool.”

Comment by Vince — April 6, 2006 #
22

I can’t stress enough how much I need a dedicated server!

Comment by Bryon — April 6, 2006 #
23

So when are you going to install Rails 1.1.1? Now that everyone knows to freeze their apps, there shouldn’t be a fire drill this time.

Comment by Sean — April 7, 2006 #
24

Oh boy, it looks like Rails developers are going to push more freq. releases. Now version 1.1.2 is out.

Comment by Yvonne — April 9, 2006 #
25

I am having trouble deploying my application because my dev machine has rails fully upgraded but DH is out of date…

Comment by Markb — April 17, 2006 #
26

DreamHost Upgrades Rails Monday…

In case you haven’t read it on the new DreamHost Status Page: DreamHost will start to roll out Ruby on Rails 1.1 early next week.
When they initially upgraded Rails to 1.1 three weeks ago a lot of application broke since Rails 1.1 isn’t 100…

Trackback by Unofficial DreamHost Blog — April 23, 2006 #
27

Does this mean Rails 1.1 is now live?
‘gem list’ says otherwise:

rails (1.0.0, 0.14.3)

Comment by Doug — April 29, 2006 #
28

[...] The reply was that dreamhost offers terrible support, oversells their servers, and their sites frequently go down. I already knew all this. You know why, because dreamhost told me. He told me that I might be saving money but that I would be losing support and performance. [...]

Pingback by from the gut » SRLNet Has Separation Issues — November 3, 2006 #

Sorry, the comment form is closed at this time.

Behind the Scenes of a botched Ruby on Rails upgrade

28 Comments

Categories:

Archives:

Meta:

Behind the Scenes of a botched Ruby on Rails upgrade

28 Comments

Categories:

Archives:

Other Sites

Meta: