And so I did just that, http://github.com/agoragames/brightcove

sudo gem install brightcove-api

>> require 'brightcove-api'
=> true
>> brightcove = Brightcove::API.new('0Z2dtxTdJAxtbZ-d0U7Bhio2V1Rhr5Iafl5FFtDPY8E.')
=> #<brightcove ::API:0x114dbc8 @token="0Z2dtxTdJAxtbZ-d0U7Bhio2V1Rhr5Iafl5FFtDPY8E.", @api_url="http://api.brightcove.com/services/library">
>> response = brightcove.get('find_all_videos', {:page_size => 3, :video_fields => 'id,name,linkURL,linkText'})
=> {"items"=>[{"name"=>"Documentarian Skydiving", "id"=>496518762, "linkText"=>nil, "linkURL"=>nil}, {"name"=>"Surface Tricks", "id"=>496518763, "linkText"=>nil, "linkURL"=>nil}, {"name"=>"Free Skiing", "id"=>496518765, "linkText"=>nil, "linkURL"=>nil}], "page_number"=>0, "page_size"=>3, "total_count"=>-1}

Today is Misha’s last day.  We wished him luck and toasted him off with some local microbrews from the Troy Pub and Brewery.

#/usr/bin/ruby
require 'openssl'
require "base64"
require 'uri'

# First lets encrypt the string!
plaintext = 'Super secret message'

# Create the cipher
cipher = OpenSSL::Cipher::Cipher.new("aes-256-cbc")
cipher.encrypt # Tell OpenSSL to operate in encrypt mode
puts "Cipher wants a key that is #{cipher.key_len}"
key = '01234567890123456789012345678901'
cipher.key = key

puts "Cipher wants an initialization vector that is #{cipher.iv_len}"
cipher.iv = iv = cipher.random_iv # Create and set a random initialization vector

# Encrypted
encrypted = cipher.update(plaintext) + cipher.final
encrypted = iv + encrypted # Send along the IV

# Lets pretty up the encrypted string
encrypted = Base64.encode64(encrypted)
#encrypted = URI.escape(encrypted, Regexp.new("[^#{URI::PATTERN::UNRESERVED}]"))
encrypted = URI.escape(encrypted)

# Now lets unencrypt it, first start with a new cipher
cipher = OpenSSL::Cipher::Cipher.new("aes-256-cbc")
cipher.decrypt # Use SSL in decrypt mode
cipher.key = key
encrypted = URI.unescape(encrypted)
encrypted = Base64.decode64(encrypted)
cipher.iv = encrypted.slice!(0,16) # Remove the IV from the encrypted data
decrypted = cipher.update(encrypted) + cipher.final 

# Test
puts 'The original was '+ plaintext
puts 'Encrypted that was ' + encrypted
puts 'Decrypted we have ' + decrypted

We recently started a number of projects, each of which will require the use of a common library. So, here’s how it went down:

* Create the common library as a gem using the jeweler tool.

* Install Gem in a Box to allow the gem to be downloaded internally by our developers using gem install.

* Upload the common library gem.

BLAMMO! Now every developer can pull my gem ;)

But seriously, “Gem in a Box” is awesome and a great way to share gems internally at your office.

There are a few rake tasks that I’ve been using more and more now that all of our applications are running in a shared, virtual enviroment. They are:

* rake -T     # -T, –tasks [PATTERN]            Display the tasks (matching optional PATTERN) with descriptions, then exit.

Everyone should know and use this task at least once in their Rails-development life.

* rake rails:freeze:gems   # Lock this application to the current gems (by unpacking them into vendor/rails)
* rake gems:unpack   # Unpacks all required gems into vendor/gems.
* rake gems:unpack:dependencies   # Unpacks all required gems and their dependencies into vendor/gems.

So, once I’ve created my project with “rails [project name]“, then next thing I do is “rake rails:freeze:gems” to freeze the Rails gems. I’ll enumerate the application’s dependencies in the environment.rb file and then run “rake gems:unpack” to make sure those dependencies exist with the application. Example:

config.gem 'will_paginate', :version => '2.3.11', :source => 'http://gemcutter.org'
config.gem 'factory_girl', :version => '1.2.4', :source => 'http://gemcutter.org'
config.gem 'fakeweb', :version => '1.2.8', :source => 'http://gemcutter.org'

Why do I like this approach? In a shared environment, it means we don’t have to have our systems folks install any dependencies for our application to run making the application self-contained. This is mostly fine for gems that don’t have an explicit native component. Also, in being explicit about versions of the gems that an application is using, we do not run the risk of chasing a moving target. Again, in a shared environment, if someone updates a gem on the system, it’s probably not an issue until the one time it is and you’re being called at 3 AM that the application is down because of a gem incompatibility change.

That’s it. Any other packaging dos and don’t?

I was adding another project to our Continuous Integration (CI) server that runs the test suite for a project after code is checked in and started noticing the time it takes to build certain projects. A few interesting statistics for you to noodle on.

Average build time: ~32 seconds

Max build time: 1 hour and 48 minutes

There are approximately 18 projects that get built through CI and many other projects that have not been setup for CI.

What does this tell us? Testing is not an impediment to development because it takes too long to run tests. As most test suites run in less than a minute, not running the test suite is not an option for our developers. And if you don’t, CI sends an e-mail to the team letting everyone know you broke the build. And since our post-commit hooks for SVN and git notify our Campfire room each time code is committed, chances are you’re going to get an earful in Campfire telling you to fix the build because your last commit broke the build.

I’m glad nearly all of our projects have test suites that run quickly, but I also have a practical view on test suites. Above all, if nothing else, I want a test suite to exist. I want it to be there so that when I’m adding code to the repository, the test suite is looking at me asking for more.

About that project that takes 1 hour and 48 minutes to build? Right. Guitar Hero. It’s a BIG project. It’s over 3 years old. There is a LOT of code. We test A LOT of systems, e.g. accounts, clans, tournaments, leaderboards, game configurations, game integration processing, etc. I’m OK with it taking that long to run the test suite. We have development practices that we follow for running sub-sections of the test suite when we touch the application to test our new code.

To summarize:

• Ensure your projects have a test suite
• Run a Continuous Integration (CI) server for your projects for when developers forget to run the test suite before committing code
• Ensure your projects have a test suite
• Ensure your projects have a test suite
• Ensure your projects have a test suite
• Make sure developers know how to run part of the test suite for long-running test suites
• Ensure your projects have a test suite

We recently came across a bottleneck in one of our applications that grabs data from a content repository. A large part of the bottleneck had to do with caching of data from the content repository, or lack thereof. A small part of the bottleneck has to do with parsing of XML data as the application grabs XML feeds from the content repository and parses it to display data within the application. Novel idea using XML as a communication mechanism between 2 applications? You betcha.

In any event, boring machine name aside, I decided to benchmark the data parsing via JSON, Hpricot, and Simple-RSS.

David-Czarneckis-iMac:application-playground dczarnecki$ ruby parsing-benchmarking.rb
"Benchmarking JSON"
"    JSON document: /Users/dczarnecki/projects/parsing-benchmarking/sample_json_document.json, Size: 161574"
"    JSON::Ext::ParserJSON::Ext::Generator"
"    Benchmarked JSON parse for 20x (average): 0.0121459603309631"
"Benchmarking Hpricot"
"    XML document: /Users/dczarnecki/projects/parsing-benchmarking/sample_photo_rss_document.rss, Size: 162644"
"    Benchmarked XML parse for 20x (average): 0.00232400894165039"
"Benchmarking Simple-RSS"
"    XML document: /Users/dczarnecki/projects/parsing-benchmarking/sample_photo_rss_document.rss, Size: 162644"
"    Benchmarked XML parse for 20x (average): 1.78787769079208"
David-Czarneckis-iMac:application-playground dczarnecki$

The numbers are interesting. Here is my opinion on the results:

- Hpricot is the fastest at parsing a big XML document. You have to use XPath to get at the elements in the document. XPath is, for me, non-intuitive and you sacrifice a bit in terms of readability of the code.

- JSON is the second fastest at parsing a big JSON document. Also, you have a four letter acronym technology integrated into your application, and let’s finally admit it, size matters. For me, you gain intuitive access to the underlying data and more readability of the code.

- Simple-RSS is slow.

- XML output from our content repository doesn’t give us paged access to the content, but the JSON API in the content repository does give us paged access to the content. With the XML output, we have to suck down a firehose and slice it up appropriately leading to a bunch of flackery with caching the parsed document.

If you’re looking for a better read, then check out the following book, “Paint It Black: A Guide to Gothic Homemaking”.

Right, Redis.

“Redis is an advanced key-value store. It is similar to memcached but the dataset is not volatile, and values can be strings, exactly like in memcached, but also lists, sets, and ordered sets. All this data types can be manipulated with atomic operations to push/pop elements, add/remove elements, perform server side union, intersection, difference between sets, and so forth. Redis supports different kind of sorting abilities.”

I was particularly interested in the support in Redis for lists, which would allow me to have one application push stuff into the datastore and allow for another application to pull stuff from the datastore. A datastore that could operate as a queue? I guess this is as close as we’re going to get to flying cars in 2010. Redis is also supposed to be wicked fast and there are a number of libraries available in your programming language of choice with which to communicate with Redis.

Redis setup was trivial and “just worked”.

After starting the Redis server, I could just prototype in script/console.

Here we go:

>> redis_queue = Redis.new
=> #<Redis:0x222df9c @thread_safe=nil, @logger=nil, @password=nil, @timeout=5, @db=0, @sock=nil, @host="127.0.0.1", @port=6379>
>> redis_queue.push_tail 'strings', 'string 1'
=> "OK"
>> redis_queue.push_tail 'strings', 'string 2'
>> "OK"
>> redis_queue.list_length('strings')
=> 2
>> redis_queue.list_range('strings', 0, -1)
=> ["string 1", "string 2"]
>> some_string = redis_queue.pop_head('strings')
=> "string 1"
>> r.list_range('strings', 0, -1)
=> ["string 2"]
>> r.list_length('strings')
=> 1
>> quit

There are quite a few commands that make Redis even more awesome, but for now I’m sold. (If using an entire sentence as a link is wrong then I don’t want to be right)

Also, I know of Resque, “a Redis-backed library for creating background jobs, placing those jobs on multiple queues, and processing them later.” If I ever feel like driving the Mercedes Benz of flying cars, I’ll use it.

I have a requirement that certain data in an application that I’m developing will probably have to be sharded because, if you’ll excuse my English, there will a “shit ton” of data. This only affects one model out of the few models I have in the application. I don’t have a requirement that the data will be replicated (which is another feature supported in data_fabric), so I’m not going into that here. In any event, here is a rundown of how I got started developing and testing with data_fabric.

- Configure the data_fabric gem in your config/environment.rb file.

config.gem 'data_fabric'

- In your model(s), decide on which column or how the data is going to be shared.

data_fabric :replicated => false, :shard_by => :initial_code

In this case, inital_code is a method that looks at a piece of the model’s data and gives me the correct shard.

- Setup the database shards in your config/database.yml file. I actually setup only one shard for development and testing environments to make things easier. I’m just including the one for the test environment here. You can read on the data_fabric site about the naming convention for sharded database connections.

test:
adapter: mysql
encoding: utf8
reconnect: false
database: myapp_test
pool: 5
username: root
password:

# This is the database shard
initial_code_testenv_test:
adapter: mysql
encoding: utf8
reconnect: false
database: myapp_test_testenv
pool: 5
username: root
password:

- In config/initializers/my_app_model.rb, I actually stub out the initial_code method to return a single value for the development and test environments. This is merely convenience so I don’t have to include every single database shard for development and testing.

require 'mocha'

if 'development'.eql?(RAILS_ENV)
PromotionCode.stubs(:initial_code).returns('devenv')
end

if 'test'.eql?(RAILS_ENV)
PromotionCode.stubs(:initial_code).returns('testenv')
end

- I copied part of the Rakefile from the data_fabric gem to actually be able to migrate the database for the sharded database connections. This was definitely missing from the data_fabric README.

require 'fileutils'
include FileUtils::Verbose

namespace :db do
task :migrate do
require 'erb'
require 'logger'
require 'active_record'
reference = YAML::load(ERB.new(IO.read("config/database.yml")).result)
env = RAILS_ENV = ENV['RAILS_ENV'] || 'development'
ActiveRecord::Base.logger = Logger.new(STDOUT)
ActiveRecord::Base.logger.level = Logger::WARN
ActiveRecord::Base.configurations = reference.dup
old_config = reference[env]
reference.each_key do |name|
next unless name.include? env
next if name.include? 'slave' # Replicated databases should not be touched directly

puts "Migrating #{name}"
ActiveRecord::Base.clear_active_connections!
ActiveRecord::Base.configurations[env] = reference[name]
ActiveRecord::Base.establish_connection RAILS_ENV
ActiveRecord::Migration.verbose = ENV["VERBOSE"] ? ENV["VERBOSE"] == "true" : true
ActiveRecord::Migrator.migrate("db/migrate/", ENV["VERSION"] ? ENV["VERSION"].to_i : nil)
end
end
end

- In my test classes that use the sharded model, I have setup and teardown methods that activate and deactivate the shard.

def setup
DataFabric.activate_shard(:initial_code => 'testenv')
end

def teardown
MyAppModel.delete_all
DataFabric.deactivate_shard(:initial_code => 'testenv')
end

I did find that I needed to delete all the objects in the database for the sharded model. I’m still digging into why that’s the case. My ActiveRecord_fu isn’t that strong I guess.

All in all, sharding is relatively easy with data_fabric. Pimping, however, “ain’t easy.” But that’s for another blog post I guess.

Subversion: 4

Git: 7

Here is what I found personally about my Git transition experience.

• * If you look at the simple examples or cursory blog post introductions of using Git as a version control system, you’re probably not going to switch. I didn’t find those examples or Git blog posts enlightening at all. I just thought to myself, “Great, Git can track changes to files just like Subversion, so why should I switch?”.
• * Git is something I can use independent of a service like GitHub locally to implement version control on projects that might never make it off of my machine.
• * Git can be taken to the extreme where every “change” can be separated from the main branch of development and then merged at a later point. At Agora, we’ve taken a more balanced approach where major features go into a new branch and then are reviewed and merged back into the main branch, after which the new branch can be safely removed (e.g. replacing an authentication system).
• * Although tools like SmartGit exist, I needed to get comfortable by using Git from the command-line.
• * There are a lot of Git commands and capabilities I haven’t used yet, and that’s OK.
• * I love the idea of the Git stash, where you can scurry away local changes and revert to a clean working directory, but then recover those changes later.

Git is just something you need to try. I’m no expert in Git (yet). Git’s barrier to entry feels very minimal when compared to other version control systems.

P.S. I realize this blog post falls under the “cursory blog post introductions of using Git as a version control system” category. Whatever.

:)