Gemfile

gem ‘rails’, ‘3.0.0.rc’
gem ‘haml’, ‘3.0.14′

Obviously you have to tell your application to use the 3.0.0.rc. Also, it seems as if haml 3.0.13, which we were previously using, had some incompatible changes with Rails 3.0.0.rc

Rakefile

Change:

Rails::Application.load_tasks

to:

YourApplicationName::Application.load_tasks

config/environments/test.rb

Add:

config.active_support.deprecation = :stderr

config/routes.rb

Change:

YourApplicationName::Application.routes.draw do |map|

to:

YourApplicationName::Application.routes.draw do

Rails 3 routing in great detail: The Lowdown on Routes in Rails 3.

Before we jump into setup, first a bit about our pre-Rails3 environment:

• Mac OS X 10.5.8
• ruby 1.8.6 (2009-06-08 patchlevel 369) [universal-darwin9.0] (stock)
• RubyGems 1.3.7
• Various older versions of Rails (2.3.8, 2.3.5, 2.3.4, 2.3.2, 2.2.2, 2.1.1, 2.1.0, 2.0.2, 1.2.6, 1.2.3)

We will be installing the latest release of Rails which, as of this writing, is Rails 3 beta4.
We will also complete setup using the latest version of Ruby 1.9.
The minimal version of Ruby 1.9 required for this Rails release is 1.9.2.
To complete setup with Ruby 1.8.7, consult the Rails 3 release notes for the minimal version requirements.

The basic installation steps are as follows:

1. Install RVM (Ruby Version Manger)
2. Install Ruby 1.9.2
3. Install Rails3 beta4
4. Profit!

To install, issue the following commands in a terminal window:

1. bash < <( curl http://rvm.beginrescueend.com/releases/rvm-install-head )
   • This is the preferred RVM installation method as described in the RVM installation instructions.
2. rvm install 1.9.2-head
3. rvm 1.9.2-head
4. gem install rails --pre

Installation of the rails gem should also install it’s respective dependencies:

    [05:16:50][tquackenbush@TQuackenbush ~]$ gem list

    *** LOCAL GEMS ***
never
    abstract (1.0.0)
    actionmailer (3.0.0.beta4)
    actionpack (3.0.0.beta4)
    activemodel (3.0.0.beta4)
    activerecord (3.0.0.beta4)
    activeresource (3.0.0.beta4)
    activesupport (3.0.0.beta4)
    arel (0.4.0)
    builder (2.1.2)
    bundler (0.9.26)
    erubis (2.6.5)
    i18n (0.4.1)
    mail (2.2.5)
    mime-types (1.16)
    polyglot (0.3.1)
    rack (1.1.0)
    rack-mount (0.6.4)
    rack-test (0.5.4)
    rails (3.0.0.beta4)
    railties (3.0.0.beta4)
    rake (0.8.7)
    rdoc (2.5.8)
    thor (0.13.6)
    treetop (1.4.8)
    tzinfo (0.3.22)

To test out your new installation, try creating a new bare bones Rails 3 application like so:

1. rails new test_app
2. cd test_app
3. bundle install
   • Bundler is the new default dependency manager in Rails 3, and will install any missing gems required by the project.
   • In my case, this was ’sqlite3-ruby (1.3.0)’
4. rails server

As in Rails 2, this should launch a WEBrick server instance listening on localhost port 3000 with output similar to:

    => Booting WEBrick
    => Rails 3.0.0.beta4 application starting in development on http://0.0.0.0:3000
    => Call with -d to detach
    => Ctrl-C to shutdown server
    [2010-06-23 16:54:06] INFO  WEBrick 1.3.1
    [2010-06-23 16:54:06] INFO  ruby 1.9.2 (2010-06-22) [i386-darwin9.8.0]
    [2010-06-23 16:54:06] INFO  WEBrick::HTTPServer#start: pid=6529 port=3000

And that’s it! You’re all ready to go with Rails 3!

Update: June 25th, 2010 (David Czarnecki)

I ran into an issue on one system where rvm and Ruby 1.9.2 were correctly installed on the system, but Rails 3 would not install. The installation would go as follows.

machine-name:~ dczarnecki$ gem install rails --pre
WARNING:  RubyGems 1.2+ index not found for:

RubyGems will revert to legacy indexes degrading performance.

I blew away the ~/.gemrc file and Rails 3 installed successfully. YMMV.

Update: June 29th, 2010 (Joshua Childs)
Ran into two issues with with dependencies while getting setup on my Ubuntu workstation.

First was while problem I ran into was while following the installation steps.

# Command:
josh@jagar-tharn:~$ rvm install 1.9.2-head

# Response:
fail: bison is not available in your path. Please ensure it exists before compiling from head.

# Solution:
sudo apt-get install bison

And the second problem I ran into was while using bundle to setup my dependencies. My workstation did not have sqlite3 installed.

# Command:
josh@jagar-tharn:~/projects/test_app$ bundle install

# Response:
...
Using rails (3.0.0.beta4) from bundler gems
Installing sqlite3-ruby (1.3.0) from rubygems repository at http://rubygems.org/ with native extensions /home/josh/.rvm/rubies/ruby-1.9.2-head/lib/ruby/1.9.1/rubygems/ext/builder.rb:46: warning: Insecure world writable dir /usr/local/libevent/ in PATH, mode 040777
/home/josh/.rvm/rubies/ruby-1.9.2-head/lib/ruby/1.9.1/rubygems/installer.rb:483:in `rescue in block in build_extensions': ERROR: Failed to build gem native extension. (Gem::Installer::ExtensionBuildError)

/home/josh/.rvm/rubies/ruby-1.9.2-head/bin/ruby extconf.rb
checking for sqlite3.h... no
sqlite3.h is missing. Try 'port install sqlite3 +universal' or 'yum install sqlite3-devel'
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers.  Check the mkmf.log file for more
details.  You may need configuration options.

Provided configuration options:
	--with-opt-dir
	--without-opt-dir
	--with-opt-include
	--without-opt-include=${opt-dir}/include
	--with-opt-lib
	--without-opt-lib=${opt-dir}/lib
	--with-make-prog
	--without-make-prog
	--srcdir=.
	--curdir
	--ruby=/home/josh/.rvm/rubies/ruby-1.9.2-head/bin/ruby
	--with-sqlite3-dir
	--without-sqlite3-dir
	--with-sqlite3-include
	--without-sqlite3-include=${sqlite3-dir}/include
	--with-sqlite3-lib
	--without-sqlite3-lib=${sqlite3-dir}/lib
...

# Solution:
josh@jagar-tharn:~/projects/test_app$ sudo apt-get install sqlite3
josh@jagar-tharn:~/projects/test_app$ sudo apt-get install libsqlite3-dev

There are a few rake tasks that I’ve been using more and more now that all of our applications are running in a shared, virtual enviroment. They are:

* rake -T     # -T, –tasks [PATTERN]            Display the tasks (matching optional PATTERN) with descriptions, then exit.

Everyone should know and use this task at least once in their Rails-development life.

* rake rails:freeze:gems   # Lock this application to the current gems (by unpacking them into vendor/rails)
* rake gems:unpack   # Unpacks all required gems into vendor/gems.
* rake gems:unpack:dependencies   # Unpacks all required gems and their dependencies into vendor/gems.

So, once I’ve created my project with “rails [project name]“, then next thing I do is “rake rails:freeze:gems” to freeze the Rails gems. I’ll enumerate the application’s dependencies in the environment.rb file and then run “rake gems:unpack” to make sure those dependencies exist with the application. Example:

config.gem 'will_paginate', :version => '2.3.11', :source => 'http://gemcutter.org'
config.gem 'factory_girl', :version => '1.2.4', :source => 'http://gemcutter.org'
config.gem 'fakeweb', :version => '1.2.8', :source => 'http://gemcutter.org'

Why do I like this approach? In a shared environment, it means we don’t have to have our systems folks install any dependencies for our application to run making the application self-contained. This is mostly fine for gems that don’t have an explicit native component. Also, in being explicit about versions of the gems that an application is using, we do not run the risk of chasing a moving target. Again, in a shared environment, if someone updates a gem on the system, it’s probably not an issue until the one time it is and you’re being called at 3 AM that the application is down because of a gem incompatibility change.

That’s it. Any other packaging dos and don’t?

I have a requirement that certain data in an application that I’m developing will probably have to be sharded because, if you’ll excuse my English, there will a “shit ton” of data. This only affects one model out of the few models I have in the application. I don’t have a requirement that the data will be replicated (which is another feature supported in data_fabric), so I’m not going into that here. In any event, here is a rundown of how I got started developing and testing with data_fabric.

- Configure the data_fabric gem in your config/environment.rb file.

config.gem 'data_fabric'

- In your model(s), decide on which column or how the data is going to be shared.

data_fabric :replicated => false, :shard_by => :initial_code

In this case, inital_code is a method that looks at a piece of the model’s data and gives me the correct shard.

- Setup the database shards in your config/database.yml file. I actually setup only one shard for development and testing environments to make things easier. I’m just including the one for the test environment here. You can read on the data_fabric site about the naming convention for sharded database connections.

test:
adapter: mysql
encoding: utf8
reconnect: false
database: myapp_test
pool: 5
username: root
password:

# This is the database shard
initial_code_testenv_test:
adapter: mysql
encoding: utf8
reconnect: false
database: myapp_test_testenv
pool: 5
username: root
password:

- In config/initializers/my_app_model.rb, I actually stub out the initial_code method to return a single value for the development and test environments. This is merely convenience so I don’t have to include every single database shard for development and testing.

require 'mocha'

if 'development'.eql?(RAILS_ENV)
PromotionCode.stubs(:initial_code).returns('devenv')
end

if 'test'.eql?(RAILS_ENV)
PromotionCode.stubs(:initial_code).returns('testenv')
end

- I copied part of the Rakefile from the data_fabric gem to actually be able to migrate the database for the sharded database connections. This was definitely missing from the data_fabric README.

require 'fileutils'
include FileUtils::Verbose

namespace :db do
task :migrate do
require 'erb'
require 'logger'
require 'active_record'
reference = YAML::load(ERB.new(IO.read("config/database.yml")).result)
env = RAILS_ENV = ENV['RAILS_ENV'] || 'development'
ActiveRecord::Base.logger = Logger.new(STDOUT)
ActiveRecord::Base.logger.level = Logger::WARN
ActiveRecord::Base.configurations = reference.dup
old_config = reference[env]
reference.each_key do |name|
next unless name.include? env
next if name.include? 'slave' # Replicated databases should not be touched directly

puts "Migrating #{name}"
ActiveRecord::Base.clear_active_connections!
ActiveRecord::Base.configurations[env] = reference[name]
ActiveRecord::Base.establish_connection RAILS_ENV
ActiveRecord::Migration.verbose = ENV["VERBOSE"] ? ENV["VERBOSE"] == "true" : true
ActiveRecord::Migrator.migrate("db/migrate/", ENV["VERSION"] ? ENV["VERSION"].to_i : nil)
end
end
end

- In my test classes that use the sharded model, I have setup and teardown methods that activate and deactivate the shard.

def setup
DataFabric.activate_shard(:initial_code => 'testenv')
end

def teardown
MyAppModel.delete_all
DataFabric.deactivate_shard(:initial_code => 'testenv')
end

I did find that I needed to delete all the objects in the database for the sharded model. I’m still digging into why that’s the case. My ActiveRecord_fu isn’t that strong I guess.

All in all, sharding is relatively easy with data_fabric. Pimping, however, “ain’t easy.” But that’s for another blog post I guess.

In a series of blog posts I want to cover how we have approached scaling out various parts of our Ruby and Rails infrastructure here at Agora Games using real-world examples on very high-traffic sites such as the Guitar Hero and Call of Duty community sites.

Here I’ll cover the “Deep Dive”. I originally came from BigCo. and there we used a concept called the “Deep Dive”, which involved taking a specific requirement in combination with an approach or technology and following a thread of execution that would take you through the entire technology stack, or a “deep dive” through the system. At the end you would either prove or disprove the technology or approach. But it was done in the context of a real set of requirements.

The following is the e-mail (project/features names changed to protect the innocent … the important concept here is the Deep Dive, not the project/features) I sent around to our engineering team in September of last year after doing a Deep Dive on a queue system.

_________________

From: David Czarnecki

To: Engineering

Clearly Defined Requirement(s)

Ultimately, to do a deep dive correctly, you need clearly defined requirements to evaluate your technology or approach against. In the case of PROJECT X, with the use of a queue, we had the following:

Setup queue
Decide event(s)
Send to queue
Aggregate from/to queue
Put into message creation
Send back to the app

Narrowing the Field

I spent a day looking at various queue packages in Ruby and other languages to understand:

Features – What features do we get out of the package?
API – How easy is it to setup/create/interact with the queue from actual code?
Aliveness – Is this an ongoing effort or was it thrown on RubyForge and ultimately abandoned?
Community – Where is this package being used? How many developers or contributors commit to the project?
Language – Are we expanding our technology stack by introducing a queue written in one language with an interface in another language?

Pork, aka The Other Other Requirements

And don’t forget about the other “unspoken” requirements.

Ease of setup
Speed
Failsafe
Scaling

At the end of the day, whichever package is picked, you want some guarantee that the package you’ve chosen is “good” or at least “good enough”. But what if the Guarantee Fairy’s a crazy glue sniffer? Next thing you know there’s change missing from your dresser and your daughter’s knocked up. I’ve seen it a hundred times. Although you’ve got a set of requirements that define how you’re going to use a technology or approach operationally, there are still requirements that need to be addressed, even if there isn’t anything formally specified.

Let’s Get Ready To Rumble

I chose Sparrow and Rabbit/AMQP since these passed the “ease of setup” requirements with flying colors.

http://code.google.com/p/sparrow/
- Pure Ruby
http://hopper.squarespace.com/blog/2008/7/22/simple-amqp-library-for-ruby.html
- Erlang Queue Server/Ruby interface to Queue

Next up it was time to prove out the feasibility of the two technologies looking at the “soft” requirements in the context of the “hard” requirements. This meant setting up the two systems to:

Setup queue
Send event(s) to queue
Aggregate from/to queue

The other “hard” requirements would be addressed based on the outcome of this initial sanity check.

2 Queues Enter, 1 Queue Leaves … Wait, what?

Although I wanted to use this to prove out “FEATURE X”, I also wanted to address its use in “FEATURE Y”. “FEATURE Y” involves converting a song file into an MP3. So I setup a test to evaluate the two systems which was:

1k, 8k, 16k, 32k, 64k messages do
25.times do
10000 messages do
publish message to queue
read message from queue
end
end
end

In other words, publish 10000 messages to the queue (in one process) and read those messages from the queue (in another process), noting how long it took to publish and read. Do this 25 times to get a min/max/average time for each of the different message sizes.

I have attached the spreadsheet of the results which show: the larger the message, the longer it takes to publish and read from the queue. However, it also shows that Sparrow could handle the 64k messages while Rabbit/AMQP could not. Sparrow got slower to process those 10000 64k items from the queue, but it never failed as with Rabbit/AMQP. Ultimately, the deep dive was not about fixing a broken AMQP adapter.

The Devil is in the Details

One benefit of using Sparrow is that persistence is built into the server. If you take down Sparrow and there are messages on the queue, it will write those out to an SQLite3 database. Ultimately this lead me to look at the size of the field it was using for queue data which would need to be patched from its current 255 characters.

Conclusion

So, I’ve now got a queue server that I feel comfortable setting up and using and that can probably handle the load of data we’re going to throw at it come launch. The queue server/queues were integrated into PROJECT X in the context of the “FEATURE X” to prove its feasibility in addressing that feature in a future sprint.

And one more thing …

There are tests for the various bits that make up “FEATURE X”. I’m most happy with the integration test which fires up a Sparrow server, fires up a foo, creates a bar, runs the aggregator, and checks to see that a baz was created for the account (oh and then cleaning up the queue server and the subscriber). 14 LOC, but there’s a lot of code that it exercises behind the scenes. And yes, it passes :)

_________________

So, there you go. Hopefully you have enough information to do your own Deep Dive.

Ultimately for FEATURE X and FEATURE Y, Sparrow more than met our needs. Advances and changes to AMQP and its associated libraries have been made which I’m sure make it a more than viable candidate. At the time however, with just getting the system to work for a day to prove out the Deep Dive, it just didn’t meet our needs. Again, the point of this blog post is to talk about the Deep Dive in the larger context of its use in Scaling Ruby and Rails.

That said, I’ve got some free moments, so I’m putting up some reference materials.

Anyway, for those who didn’t see it, my talk was about the cloud infrastructure we built to power the Call of Duty stats site, and the tools we built to support that. The slides from the talk are available here, though they probably don’t mean too much without the audio behind them.

The puzzle pieces I talked about were:

• Virtualization, which allowed us to not have to worry about hardware failures (and gave us a lot of flexibility unrelated to scaling, such as being able to instantly clone a test environment, do some testing, and shut it back down, for virtually no cost). Terremark is our awesome hosting provider.
• NFS, which allowed us to do away with complicated tools like Capistrano and avoid managing server installations separately. It’s been around for 25 years, so it’s stable, and it’s also dead simple. Unfortunately, it is not suitable for tasks requiring heavy IO or a very large number of files, so it must be used with care.
• Monit, which allowed us to monitor our hosts and automatically fix certain problems (such as application failures) without requiring human intervention. As mentioned in the slides, if you want to pull XML from Monit (which we do to aggregate data from all of our hosts), the URL or doing so is “/_status?format=xml”. This behavior is not documented.
• Overlord, a simple internal tool that distributes server configuration and aggregates monitoring information from each Monit instance. Overlord is currently proprietary, but we’re considering releasing it once it’s ready (read: once it’s properly documented). But it’s really simple. It basically says what files should be on which servers (all of which are just simple text files), and they’re placed there on boot. Any files put into a particular directory are run as scripts. After that, it’s just a big cron job to pull XML from all of our servers and make pretty graphs.
• RRDTool, which has many uses, but specifically makes the aforementioned pretty graphs. These are vital to determine trends and validate results. Also, they’re really pretty. Double also, RRDTool is extremely well constructed and totally awesome.
• Using shell scripts everywhere. Thanks to using NFS, Monit, and Overlord, our distribution needs are already taken care of, so we can do a lot via simple shell scripts, which makes our infrastructure self-documenting and easy to work with. The simplest example of this is that we switched to deploying our applications with shell scripts, instead of using Capistrano or other network tools.

The end result is a scalable system that, while it still has a few warts we’re working through, is very simple, self-documenting, and easy for even a non-sys-admin to diagnose and solve problems on–there’s very little magic here, just some elegant abstractions.

Some people were interested in getting some more information on our deploy scripts, so I’ve made them available online. They come in two parts: “core.sh” defines the core set of functionality, for logging, rollbacks, and some basic “here’s how you deploy Rails 101″ functions. “deploy.sh” actually performs the deploy itself, and is the script that should be executed.

All told, we had a blast, and are looking forward to future RailsConfs!

Occasionally we need to do something more fancy. These instances usually come up when we’re splitting one query into two because rails doesn’t support :force_index or :adapter_specific_find_options on ActiveRecord::Base.find. We understand this motivation but really hate ActiveRecord::Base.connection#find_by_sql or ActiveRecord::Base.connection#execute. These are not rational hatreds.

So when we get into a situation where we’re going to be caching manually it’s usually in the controller and we almost always end up with a pattern of:

@object = Rails.cache.read('really/complicated/and/stinky/key')
if @object.nil?
@object = what_should_my_object_be?
end

That’s fine in a contrived example but we were doing this in about 10 different places and it looked like a good candidate for drying up.

Here’s the solution we use:

module ActiveSupport
module Cache
class Store
def read_and_write_if_nil(key, options = {})
object = read(key)
if object.nil?
object = yield
write(key, object, options)
end
object
end
end
end
end

And the production example looks like this:

account_ids = Rails.cache.read_and_write_if_nil("member_ids_for_clan_#{@clan.id}", :expires_in => 5.minutes) do
@clan.members.find(:all, :order => 'groupies DESC', :select => 'accounts.id').collect(&:id)
end

Let me preface this little article by saying I did not write the app I am now porting to our new network. I know noting about its specific intricacies, and furthermore, I know nothing specifically about gettext, file_column, or attachment_fu. So, I dove into this project with a sort of wanton abandon that is fairly characteristic of much of the work I do. Any obviously stupid mistakes below thus really happened.

Let me also preface this with the fact that this Rails app is old. Version 1.2.3 old. So, as always, YMMV.

Finally, I really wish attachment_fu had proper documentation.

Installing GetText 1.10.0

We don’t really like GetText around here, but, at the time, it was the only option. Getting it running off of a local copy (so I wouldn’t have to pollute other projects with it) was less difficult than I expected. Go to your vendor/gems directory and issue:

gem unpack gettext

This assumes you have gettext 1.10.0 installed locally. I think adding a VERSION=1.10.0 will handle it if you have multiple gettext versions laying around. After that, I merely needed to add the following to config/environment.rb to make sure the gem loaded:

GETTEXT_GEM_DIR = "#{File.expand_path(RAILS_ROOT)}/vendor/gems/gettext-1.10.0"
$LOAD_PATH.reject { |path| path =~ /gettext/ }
$LOAD_PATH << GETTEXT_GEM_DIR
$LOAD_PATH << File.join(GETTEXT_GEM_DIR, 'lib')
$LOAD_PATH << File.join(GETTEXT_GEM_DIR, 'ext', 'gettext')
require File.join(GETTEXT_GEM_DIR, 'lib', 'gettext.rb')

Ta-da!

Replacing File_Column With Attachment_Fu

After doing the above and booting up a test server, I get the following fun error:

no such file to load -- RMagick

There is no way I am ever installing RMagick onto our nice, new, beautiful infrastructure. I want things done right, here. I also don’t want to have to be paranoid about memory leaks, especially for legacy applications. So, we have to get rid of it.

This problem is caused by our use of file_column, an old-but-effective way of easily adding file uploading support into a Rails app. The modern de facto standard for this seems to be attachment_fu, which uses the much-better-designed ImageScience instead of RMagick.

So, let’s gut this fish.

Remove File_Column.

$ svn rm vendor/plugins/file_column

Install ImageScience.

$ sudo apt-get -y install libfreeimage3 libfreeimage-dev
$ sudo gem install image_science

Install Attachment_Fu.

$ script/plugin install http://svn.techno-weenie.net/projects/plugins/attachment_fu/

After installing attachment_fu, I powered back up script/server to check the next fun error:

undefined method `file_column' for Clan:Class

This is expected: file_column is gone, so the app can’t possibly know what a file_column is. Now, we have to get into the nitty-gritty of tweaking the app.

Adapting the Application to Attachment_Fu.

Let’s start in app/models/clan.rb, to which I was directed by the aforementioned error:

$ grep file_column app/models/clan.rb

No output. The words file_column don’t appear in that class. Not what I expected.

In the time-honored tradition of killing a cockroach with a shotgun, I then turned my sights on the entire app directory:

$ grep -R file_column app

The only results that turn up happen to be in app/views, leaving me perplexed. A quick conversation with the project lead informs me that this application uses “skeletons”, or appable_plugins, which was an absolutely horrible idea that we thankfully no longer use. In essence, it allows you to wrap pieces of functionality into a self-contained plugin, which is duck-punched onto your application when it starts. It is great from a code-reuse perspective, but horrible from a KISS and maintenance standpoint, which is why we no longer use it.

Except that every now and again, it comes back to bite you in the ass when you least expect it.

Anyway, checking the skeletons quickly turns up what I am looking for:

$ grep -R file_column vendor/skeletons/clan_skeleton
vendor/skeletons/clan_skeleton/app/models/clan.rb:  file_column :avatar_image, :magick => { :size => "64x64!"}

So, there is where I have to make the first incision. I changed it as follows:

# file_column :avatar_image, :magick => { :size => "64x64!"}
# validates_filesize_of :avatar_image, :in => 0..40.kilobytes

has_attachment :content_type => :image,
               :storage => :file_system,
               :max_size => 40.kilobytes,
               :resize_to => '64x64!'

validates_as_attachment

Time to restart the server and try again. (I don’t honestly expect it to work: there are (at least!) a lot of views that use file_column methods. It is easiest to just hack at them one at a time until everything works, though!)

I spent the next couple hours digging through the innards of attachment_fu, only to discover that I had written :filesystem, above, instead of :file_system (note the underscore). Make sure you spell things correctly, people, or you’ll be in for a world of self-induced hurt. (The above snippet shows things the correct way, so you, my beloved internet, won’t inadvertently copy-and-paste yourself into hell.)

Once I had properly spelled everything, though, I met with success: the front page of the site loaded! Clicking on just about anything yielded a failure, as anticipated:

undefined method `url_for_file_column' for #<#<Class:0x7f7d7f3006c0>:0x7f7d7f300670>

The views didn’t work, but the fix wasn’t horrible. Generally speaking, I simply searched and replaced the following:

url_for_file_column(X, 'avatar_image')

with:

X.public_filename

Also, along with this, I modified the clan table’s SQL schema to accomodate attachment_fu:

ALTER TABLE clans CHANGE avatar_image filename VARCHAR(255) DEFAULT NULL;
ALTER TABLE clans ADD (content_type VARCHAR(255) DEFAULT NULL,
                       size INT DEFAULT NULL,
                       width INT DEFAULT NULL,
                       height INT DEFAULT NULL);

Thankfully, this was pretty easy to do, since the clans table was very small for this particular title. If it were larger, I would create a separate table and do a

SELECT * FROM clans INSERT INTO clans_temp

sort of thing. (Yes, I know that it is very evil to hack the SQL directly instead of using migrations. This app is sufficiently old that I cannot guarantee that its migrations even work; I’m already spending a lot of time trying to port this app, I don’t want to spend much more trying to make it perfect. If it weren’t a legacy application, however, I would do things a bit more cleanly.)

At this point, virtually everything on the site worked. Avatars (once moved to the proper new directories) showed up just fine. The only remaining error came from trying to upload a new avatar image. Solving this essentially boiled down to a complex series of hacks to blend attachment_fu into the remainder of the existing codebase; none of which were particularly complex or interesting (generally involving changing the names a form parameters from the previous :avatar_image to the new :uploaded_data, like this:)

<%#= file_column_field 'clan', 'avatar_image' %>
<%= file_field_tag :uploaded_data %>

Doing all this left me with one final application error, received when actually performing the file upload:

Content type is not included in the list
content type can't be blank

Validations are complaining. After tinkering with attachment_fu’s code for a while, I discover that there is a method that attachment_fu adds to your ActiveRecords (uploaded_data=), which sets content_type and other miscellaneous variables. And, clearly, this wasn’t getting called.

Turns out I was getting bitten by multipart forms. By tossing a quick

puts params.inspect

into the code, I quickly saw where my problem lay. Notice the following bit from our clans controller’s update method:

@clan.update_attributes(params[:clan])

Unfortunately for me, the uploaded data is in params[:uploaded_data], and not in params[:clan][:uploaded_data]. While there’s a cleaner fix, the following hack suits our purposes fine and allows me to finish up this 20-hour hackfest:

params[:clan][:uploaded_data] = params[:uploaded_data]

So, after all of this code mangling, we finally had our app ready to move to a new, lighter-weight infrastructure, ultimately saving us money. A very frustrating few days, but they will pay for themselves. All that remains is integrating into a new infrastructure, and switching over the DNS.

AgoraGames has been doing community sites for several years now, which means that some of them are pretty old and some of them have particularly unruly codebases, especially one such site. This site (which shall remain nameless to protect the guilty) ran perfectly fine at launch, but over the course of a year-and-a-half has slowly gone from speed-demon to slug-stuck-in-molasses.

(I apologize for the lack of accompanying images, etc. as I worked through this… next time, I’ll keep posterity in mind as I work through an interesting problem!)

I’m the System Administrator here, so it’s my job to make sure that everything stays running smoothly from the OS’s standpoint; thanks to our religious use of monit, the cause of the site’s slowness was pretty obvious: the mongrels running the site would balloon to consuming a gigabyte of memory within minutes, forcing monit to kill-and-restart the servers. This is a Bad Thing (TM).

Finding and fixing memory leaks in Rails isn’t the easiest task in the world (as other bloggers have been apt to mention), but in this case, the path to fixing it was pretty straightforward.

First task was to set up a separate bank of mongrels for the forums: since the site is so old, it’s the only part of the site that gets much traffic these days. I told NGINX to proxy all requests starting with /forums to those. It worked just fine, and the forums were immediately quite quick. From looking at monit, it was pretty apparent that the forum mongrels didn’t have the memory leak, since they ran without problems.

So, to start digging into the main portion of the site, I booted up a local server in production mode and curled the homepage. It took a whopping 222 seconds (and, from looking at top, consumed 800MB of memory!). Looking at the debug logs, virtually all of that time was spend doing two queries: loading forum posts with associations, and loading tournaments with associations. Sounds a bit sketchy, no?

The home page’s controller was laid out very logically, so mapping the SQL queries in the log to method invokations in the controller was simple. There were two methods that were killing us, and they both looked something like this:

class ForumPost < ActiveRecord::Base
  ...
  def self.most_recent
    find(:first, :order => "forum_posts.created_at desc",
                 :include => [:account, :forum_topic, :forum])
  end
  ...
end

Apparently, the find is loading up the (tens of thousands) of forum posts, with all the associated objects! Ruby doesn’t play nice when you have thousands of object in memory, and loading up hundreds of thousands of objects really makes things go south. Also makes sense why it didn’t become a problem for a while — it’s no big deal to load 100 object into memory (during testing and launch); but after a year and the site’s forum starts to fill…

But why? Isn’t this doing a :first? Shouldn’t it only be loading one object with associations? My guess is that Rails is loading all objects (with their associations) into an array and shifting off the first element. This isn’t smart behavior, but would explain what we’re seeing. (It’s also worth noting that this is using a fairly old version of Rails; I believe this silliness is fixed in more recent versions.)

Fixing this was actually trivial. Slap a :limit => 1 on there, or replace the :include with a :join. (Or, in the particular case above, remove the :include entirely. It’s not being used for anything relevant.)

I grepped the codebase for any other :includes, and fixed the ones that were broken. The site blazes along now and doesn’t leak, just like it did in its glory days.

Moral of the story: don’t use includes lightly. Make sure you know what your finds are actually doing. In our case, it translated into memory leaks, but it could just as easily translate into high database load, which is just as bad.