And so I did just that, http://github.com/agoragames/brightcove

sudo gem install brightcove-api

>> require 'brightcove-api'
=> true
>> brightcove = Brightcove::API.new('0Z2dtxTdJAxtbZ-d0U7Bhio2V1Rhr5Iafl5FFtDPY8E.')
=> #<brightcove ::API:0x114dbc8 @token="0Z2dtxTdJAxtbZ-d0U7Bhio2V1Rhr5Iafl5FFtDPY8E.", @api_url="http://api.brightcove.com/services/library">
>> response = brightcove.get('find_all_videos', {:page_size => 3, :video_fields => 'id,name,linkURL,linkText'})
=> {"items"=>[{"name"=>"Documentarian Skydiving", "id"=>496518762, "linkText"=>nil, "linkURL"=>nil}, {"name"=>"Surface Tricks", "id"=>496518763, "linkText"=>nil, "linkURL"=>nil}, {"name"=>"Free Skiing", "id"=>496518765, "linkText"=>nil, "linkURL"=>nil}], "page_number"=>0, "page_size"=>3, "total_count"=>-1}

Before we jump into setup, first a bit about our pre-Rails3 environment:

• Mac OS X 10.5.8
• ruby 1.8.6 (2009-06-08 patchlevel 369) [universal-darwin9.0] (stock)
• RubyGems 1.3.7
• Various older versions of Rails (2.3.8, 2.3.5, 2.3.4, 2.3.2, 2.2.2, 2.1.1, 2.1.0, 2.0.2, 1.2.6, 1.2.3)

We will be installing the latest release of Rails which, as of this writing, is Rails 3 beta4.
We will also complete setup using the latest version of Ruby 1.9.
The minimal version of Ruby 1.9 required for this Rails release is 1.9.2.
To complete setup with Ruby 1.8.7, consult the Rails 3 release notes for the minimal version requirements.

The basic installation steps are as follows:

1. Install RVM (Ruby Version Manger)
2. Install Ruby 1.9.2
3. Install Rails3 beta4
4. Profit!

To install, issue the following commands in a terminal window:

1. bash < <( curl http://rvm.beginrescueend.com/releases/rvm-install-head )
   • This is the preferred RVM installation method as described in the RVM installation instructions.
2. rvm install 1.9.2-head
3. rvm 1.9.2-head
4. gem install rails --pre

Installation of the rails gem should also install it’s respective dependencies:

    [05:16:50][tquackenbush@TQuackenbush ~]$ gem list

    *** LOCAL GEMS ***
never
    abstract (1.0.0)
    actionmailer (3.0.0.beta4)
    actionpack (3.0.0.beta4)
    activemodel (3.0.0.beta4)
    activerecord (3.0.0.beta4)
    activeresource (3.0.0.beta4)
    activesupport (3.0.0.beta4)
    arel (0.4.0)
    builder (2.1.2)
    bundler (0.9.26)
    erubis (2.6.5)
    i18n (0.4.1)
    mail (2.2.5)
    mime-types (1.16)
    polyglot (0.3.1)
    rack (1.1.0)
    rack-mount (0.6.4)
    rack-test (0.5.4)
    rails (3.0.0.beta4)
    railties (3.0.0.beta4)
    rake (0.8.7)
    rdoc (2.5.8)
    thor (0.13.6)
    treetop (1.4.8)
    tzinfo (0.3.22)

To test out your new installation, try creating a new bare bones Rails 3 application like so:

1. rails new test_app
2. cd test_app
3. bundle install
   • Bundler is the new default dependency manager in Rails 3, and will install any missing gems required by the project.
   • In my case, this was ’sqlite3-ruby (1.3.0)’
4. rails server

As in Rails 2, this should launch a WEBrick server instance listening on localhost port 3000 with output similar to:

    => Booting WEBrick
    => Rails 3.0.0.beta4 application starting in development on http://0.0.0.0:3000
    => Call with -d to detach
    => Ctrl-C to shutdown server
    [2010-06-23 16:54:06] INFO  WEBrick 1.3.1
    [2010-06-23 16:54:06] INFO  ruby 1.9.2 (2010-06-22) [i386-darwin9.8.0]
    [2010-06-23 16:54:06] INFO  WEBrick::HTTPServer#start: pid=6529 port=3000

And that’s it! You’re all ready to go with Rails 3!

Update: June 25th, 2010 (David Czarnecki)

I ran into an issue on one system where rvm and Ruby 1.9.2 were correctly installed on the system, but Rails 3 would not install. The installation would go as follows.

machine-name:~ dczarnecki$ gem install rails --pre
WARNING:  RubyGems 1.2+ index not found for:

RubyGems will revert to legacy indexes degrading performance.

I blew away the ~/.gemrc file and Rails 3 installed successfully. YMMV.

Update: June 29th, 2010 (Joshua Childs)
Ran into two issues with with dependencies while getting setup on my Ubuntu workstation.

First was while problem I ran into was while following the installation steps.

# Command:
josh@jagar-tharn:~$ rvm install 1.9.2-head

# Response:
fail: bison is not available in your path. Please ensure it exists before compiling from head.

# Solution:
sudo apt-get install bison

And the second problem I ran into was while using bundle to setup my dependencies. My workstation did not have sqlite3 installed.

# Command:
josh@jagar-tharn:~/projects/test_app$ bundle install

# Response:
...
Using rails (3.0.0.beta4) from bundler gems
Installing sqlite3-ruby (1.3.0) from rubygems repository at http://rubygems.org/ with native extensions /home/josh/.rvm/rubies/ruby-1.9.2-head/lib/ruby/1.9.1/rubygems/ext/builder.rb:46: warning: Insecure world writable dir /usr/local/libevent/ in PATH, mode 040777
/home/josh/.rvm/rubies/ruby-1.9.2-head/lib/ruby/1.9.1/rubygems/installer.rb:483:in `rescue in block in build_extensions': ERROR: Failed to build gem native extension. (Gem::Installer::ExtensionBuildError)

/home/josh/.rvm/rubies/ruby-1.9.2-head/bin/ruby extconf.rb
checking for sqlite3.h... no
sqlite3.h is missing. Try 'port install sqlite3 +universal' or 'yum install sqlite3-devel'
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers.  Check the mkmf.log file for more
details.  You may need configuration options.

Provided configuration options:
	--with-opt-dir
	--without-opt-dir
	--with-opt-include
	--without-opt-include=${opt-dir}/include
	--with-opt-lib
	--without-opt-lib=${opt-dir}/lib
	--with-make-prog
	--without-make-prog
	--srcdir=.
	--curdir
	--ruby=/home/josh/.rvm/rubies/ruby-1.9.2-head/bin/ruby
	--with-sqlite3-dir
	--without-sqlite3-dir
	--with-sqlite3-include
	--without-sqlite3-include=${sqlite3-dir}/include
	--with-sqlite3-lib
	--without-sqlite3-lib=${sqlite3-dir}/lib
...

# Solution:
josh@jagar-tharn:~/projects/test_app$ sudo apt-get install sqlite3
josh@jagar-tharn:~/projects/test_app$ sudo apt-get install libsqlite3-dev

We recently started a number of projects, each of which will require the use of a common library. So, here’s how it went down:

* Create the common library as a gem using the jeweler tool.

* Install Gem in a Box to allow the gem to be downloaded internally by our developers using gem install.

* Upload the common library gem.

BLAMMO! Now every developer can pull my gem ;)

But seriously, “Gem in a Box” is awesome and a great way to share gems internally at your office.

Right, Redis.

“Redis is an advanced key-value store. It is similar to memcached but the dataset is not volatile, and values can be strings, exactly like in memcached, but also lists, sets, and ordered sets. All this data types can be manipulated with atomic operations to push/pop elements, add/remove elements, perform server side union, intersection, difference between sets, and so forth. Redis supports different kind of sorting abilities.”

I was particularly interested in the support in Redis for lists, which would allow me to have one application push stuff into the datastore and allow for another application to pull stuff from the datastore. A datastore that could operate as a queue? I guess this is as close as we’re going to get to flying cars in 2010. Redis is also supposed to be wicked fast and there are a number of libraries available in your programming language of choice with which to communicate with Redis.

Redis setup was trivial and “just worked”.

After starting the Redis server, I could just prototype in script/console.

Here we go:

>> redis_queue = Redis.new
=> #<Redis:0x222df9c @thread_safe=nil, @logger=nil, @password=nil, @timeout=5, @db=0, @sock=nil, @host="127.0.0.1", @port=6379>
>> redis_queue.push_tail 'strings', 'string 1'
=> "OK"
>> redis_queue.push_tail 'strings', 'string 2'
>> "OK"
>> redis_queue.list_length('strings')
=> 2
>> redis_queue.list_range('strings', 0, -1)
=> ["string 1", "string 2"]
>> some_string = redis_queue.pop_head('strings')
=> "string 1"
>> r.list_range('strings', 0, -1)
=> ["string 2"]
>> r.list_length('strings')
=> 1
>> quit

There are quite a few commands that make Redis even more awesome, but for now I’m sold. (If using an entire sentence as a link is wrong then I don’t want to be right)

Also, I know of Resque, “a Redis-backed library for creating background jobs, placing those jobs on multiple queues, and processing them later.” If I ever feel like driving the Mercedes Benz of flying cars, I’ll use it.

In a series of blog posts I want to cover how we have approached scaling out various parts of our Ruby and Rails infrastructure here at Agora Games using real-world examples on very high-traffic sites such as the Guitar Hero and Call of Duty community sites.

Here I’ll cover the “Deep Dive”. I originally came from BigCo. and there we used a concept called the “Deep Dive”, which involved taking a specific requirement in combination with an approach or technology and following a thread of execution that would take you through the entire technology stack, or a “deep dive” through the system. At the end you would either prove or disprove the technology or approach. But it was done in the context of a real set of requirements.

The following is the e-mail (project/features names changed to protect the innocent … the important concept here is the Deep Dive, not the project/features) I sent around to our engineering team in September of last year after doing a Deep Dive on a queue system.

_________________

From: David Czarnecki

To: Engineering

Clearly Defined Requirement(s)

Ultimately, to do a deep dive correctly, you need clearly defined requirements to evaluate your technology or approach against. In the case of PROJECT X, with the use of a queue, we had the following:

Setup queue
Decide event(s)
Send to queue
Aggregate from/to queue
Put into message creation
Send back to the app

Narrowing the Field

I spent a day looking at various queue packages in Ruby and other languages to understand:

Features – What features do we get out of the package?
API – How easy is it to setup/create/interact with the queue from actual code?
Aliveness – Is this an ongoing effort or was it thrown on RubyForge and ultimately abandoned?
Community – Where is this package being used? How many developers or contributors commit to the project?
Language – Are we expanding our technology stack by introducing a queue written in one language with an interface in another language?

Pork, aka The Other Other Requirements

And don’t forget about the other “unspoken” requirements.

Ease of setup
Speed
Failsafe
Scaling

At the end of the day, whichever package is picked, you want some guarantee that the package you’ve chosen is “good” or at least “good enough”. But what if the Guarantee Fairy’s a crazy glue sniffer? Next thing you know there’s change missing from your dresser and your daughter’s knocked up. I’ve seen it a hundred times. Although you’ve got a set of requirements that define how you’re going to use a technology or approach operationally, there are still requirements that need to be addressed, even if there isn’t anything formally specified.

Let’s Get Ready To Rumble

I chose Sparrow and Rabbit/AMQP since these passed the “ease of setup” requirements with flying colors.

http://code.google.com/p/sparrow/
- Pure Ruby
http://hopper.squarespace.com/blog/2008/7/22/simple-amqp-library-for-ruby.html
- Erlang Queue Server/Ruby interface to Queue

Next up it was time to prove out the feasibility of the two technologies looking at the “soft” requirements in the context of the “hard” requirements. This meant setting up the two systems to:

Setup queue
Send event(s) to queue
Aggregate from/to queue

The other “hard” requirements would be addressed based on the outcome of this initial sanity check.

2 Queues Enter, 1 Queue Leaves … Wait, what?

Although I wanted to use this to prove out “FEATURE X”, I also wanted to address its use in “FEATURE Y”. “FEATURE Y” involves converting a song file into an MP3. So I setup a test to evaluate the two systems which was:

1k, 8k, 16k, 32k, 64k messages do
25.times do
10000 messages do
publish message to queue
read message from queue
end
end
end

In other words, publish 10000 messages to the queue (in one process) and read those messages from the queue (in another process), noting how long it took to publish and read. Do this 25 times to get a min/max/average time for each of the different message sizes.

I have attached the spreadsheet of the results which show: the larger the message, the longer it takes to publish and read from the queue. However, it also shows that Sparrow could handle the 64k messages while Rabbit/AMQP could not. Sparrow got slower to process those 10000 64k items from the queue, but it never failed as with Rabbit/AMQP. Ultimately, the deep dive was not about fixing a broken AMQP adapter.

The Devil is in the Details

One benefit of using Sparrow is that persistence is built into the server. If you take down Sparrow and there are messages on the queue, it will write those out to an SQLite3 database. Ultimately this lead me to look at the size of the field it was using for queue data which would need to be patched from its current 255 characters.

Conclusion

So, I’ve now got a queue server that I feel comfortable setting up and using and that can probably handle the load of data we’re going to throw at it come launch. The queue server/queues were integrated into PROJECT X in the context of the “FEATURE X” to prove its feasibility in addressing that feature in a future sprint.

And one more thing …

There are tests for the various bits that make up “FEATURE X”. I’m most happy with the integration test which fires up a Sparrow server, fires up a foo, creates a bar, runs the aggregator, and checks to see that a baz was created for the account (oh and then cleaning up the queue server and the subscriber). 14 LOC, but there’s a lot of code that it exercises behind the scenes. And yes, it passes :)

_________________

So, there you go. Hopefully you have enough information to do your own Deep Dive.

Ultimately for FEATURE X and FEATURE Y, Sparrow more than met our needs. Advances and changes to AMQP and its associated libraries have been made which I’m sure make it a more than viable candidate. At the time however, with just getting the system to work for a day to prove out the Deep Dive, it just didn’t meet our needs. Again, the point of this blog post is to talk about the Deep Dive in the larger context of its use in Scaling Ruby and Rails.

You will need to download and install the Lighthouse API gem and the Net::TOC gem.

Run this in an irb session.

require 'net/toc'
require 'lighthouse'
require 'sanitize'
include Lighthouse

Lighthouse.account = 'shaft'
Lighthouse.token = 'hesonebadmotherfushutyourmouth'

Net::TOC.new("someaimaccount", "someaimpassword") do |msg, buddy|
project = Project.find(Sanitize.clean(msg).to_i)
buddy.send_im("Here is some information about your project: #{project.name}")
end

You should then be able to add ’someaimaccount’ to your AIM buddy list and send a project ID and have it return the project name.

And boom goes the dynamite!

You’ll of course need to change the Lighthouse account and token as appropriate (or use username and password for logging in). msg is a little weird and although it’s a string, it’s HTML, so you need to strip out any tags before doing anything. And of course you’d want to add in some way of parsing the input from the user into some DSL (Domain Specific Language) for interacting with your fancy new Lighthouse AIM bot.

Happy hacking!

def some_method_that_uses_a_soap_web_service
storage = SOAP::WSDLDriverFactory.new(STORAGE_WSDL).create_rpc_driver
...
storage.wiredump_dev = STDERR
response = storage.DoSomething(...)
end

When the DoSomething web service method is called, the Ruby SOAP library will print out the request and response to STDERR.

It is invaluable. Trust me!

Follow-up (8/13/2009): A friend pointed me at Handsoap, a library for creating SOAP clients in Ruby, which is an alternative to SOAP4R. It apparently does not suck. YMMV.