AGORA GAMES

Yesterday I started looking at ways to do inter-application communication. In a number of projects we’ve done here at Agora Games, we’ve used queues to make that happen. Redis has been on my radar for awhile now, but yesterday I drove my Chevy to the levee and guess what? The levee is NOT dry people. I mean, who drinks rye anyway these days? Old people.

(more…)

The data_fabric gem “provides flexible database connection switching for ActiveRecord”. If you’re not concerned with database sharding, you might want to skip this blog post. Or not. Either way, I’m not going to be offended.

I have a requirement that certain data in an application that I’m developing will probably have to be sharded because, if you’ll excuse my English, there will a “shit ton” of data. This only affects one model out of the few models I have in the application. I don’t have a requirement that the data will be replicated (which is another feature supported in data_fabric), so I’m not going into that here. In any event, here is a rundown of how I got started developing and testing with data_fabric.

- Configure the data_fabric gem in your config/environment.rb file.

config.gem 'data_fabric'

- In your model(s), decide on which column or how the data is going to be shared.

data_fabric :replicated => false, :shard_by => :initial_code

In this case, inital_code is a method that looks at a piece of the model’s data and gives me the correct shard.

- Setup the database shards in your config/database.yml file. I actually setup only one shard for development and testing environments to make things easier. I’m just including the one for the test environment here. You can read on the data_fabric site about the naming convention for sharded database connections.

test:
adapter: mysql
encoding: utf8
reconnect: false
database: myapp_test
pool: 5
username: root
password:

# This is the database shard
initial_code_testenv_test:
adapter: mysql
encoding: utf8
reconnect: false
database: myapp_test_testenv
pool: 5
username: root
password:

- In config/initializers/my_app_model.rb, I actually stub out the initial_code method to return a single value for the development and test environments. This is merely convenience so I don’t have to include every single database shard for development and testing.

require 'mocha'

if 'development'.eql?(RAILS_ENV)
PromotionCode.stubs(:initial_code).returns('devenv')
end

if 'test'.eql?(RAILS_ENV)
PromotionCode.stubs(:initial_code).returns('testenv')
end

- I copied part of the Rakefile from the data_fabric gem to actually be able to migrate the database for the sharded database connections. This was definitely missing from the data_fabric README.

require 'fileutils'
include FileUtils::Verbose

namespace :db do
task :migrate do
require 'erb'
require 'logger'
require 'active_record'
reference = YAML::load(ERB.new(IO.read("config/database.yml")).result)
env = RAILS_ENV = ENV['RAILS_ENV'] || 'development'
ActiveRecord::Base.logger = Logger.new(STDOUT)
ActiveRecord::Base.logger.level = Logger::WARN
ActiveRecord::Base.configurations = reference.dup
old_config = reference[env]
reference.each_key do |name|
next unless name.include? env
next if name.include? 'slave' # Replicated databases should not be touched directly

puts "Migrating #{name}"
ActiveRecord::Base.clear_active_connections!
ActiveRecord::Base.configurations[env] = reference[name]
ActiveRecord::Base.establish_connection RAILS_ENV
ActiveRecord::Migration.verbose = ENV["VERBOSE"] ? ENV["VERBOSE"] == "true" : true
ActiveRecord::Migrator.migrate("db/migrate/", ENV["VERSION"] ? ENV["VERSION"].to_i : nil)
end
end
end

- In my test classes that use the sharded model, I have setup and teardown methods that activate and deactivate the shard.

def setup
DataFabric.activate_shard(:initial_code => 'testenv')
end

def teardown
MyAppModel.delete_all
DataFabric.deactivate_shard(:initial_code => 'testenv')
end

I did find that I needed to delete all the objects in the database for the sharded model. I’m still digging into why that’s the case. My ActiveRecord_fu isn’t that strong I guess.

All in all, sharding is relatively easy with data_fabric. Pimping, however, “ain’t easy.” But that’s for another blog post I guess.

It’s amazing how a few months can change your mindset around the version control system you use. Ever since I joined Agora Games in May 2008, we have used Subversion (SVN). Subversion is a fine version control system. We have one new project using Subversion and we will probably have a few legacy projects that will always use Subversion. However, last year, one of our project teams made the switch to Git and ever since then, new projects have been using Git. Looking at CruiseControl, here’s the breakdown of Subversion and Git projects:

Subversion: 4

Git: 7

Here is what I found personally about my Git transition experience.

• * If you look at the simple examples or cursory blog post introductions of using Git as a version control system, you’re probably not going to switch. I didn’t find those examples or Git blog posts enlightening at all. I just thought to myself, “Great, Git can track changes to files just like Subversion, so why should I switch?”.
• * Git is something I can use independent of a service like GitHub locally to implement version control on projects that might never make it off of my machine.
• * Git can be taken to the extreme where every “change” can be separated from the main branch of development and then merged at a later point. At Agora, we’ve taken a more balanced approach where major features go into a new branch and then are reviewed and merged back into the main branch, after which the new branch can be safely removed (e.g. replacing an authentication system).
• * Although tools like SmartGit exist, I needed to get comfortable by using Git from the command-line.
• * There are a lot of Git commands and capabilities I haven’t used yet, and that’s OK.
• * I love the idea of the Git stash, where you can scurry away local changes and revert to a clean working directory, but then recover those changes later.

Git is just something you need to try. I’m no expert in Git (yet). Git’s barrier to entry feels very minimal when compared to other version control systems.

P.S. I realize this blog post falls under the “cursory blog post introductions of using Git as a version control system” category. Whatever.

:)

I ran into an interesting issue today when deploying a crappy password change app I wrote as an exercise in rails. It turns out that by default, NGINX has a gzip buffer size of 4 * 4k/8k, with the bit size depending on what platform the service is running on.

As a result,

4 * 4k = 16384 bytes (16KB)
4 * 8k = 32768 (32KB)

After gzip, when converted to be plain text, the amount of data returned to the browser nearly doubles (about 52KB in the case of larger). The default javascript library included by rails for AJAX (prototype.js) is 127k, which led me to find this limit.

And so, getting to the point, the limit applied to the gzip buffer needs to be increased within NGINX’s config files for each site that will use a file larger than these pre-set sizes. You can do so using the following:

gzip_buffers 16 8k

This will set the buffer size to 16 * 8k, or 128KB — the uncompressed size of the library, eliminating the issue.

In the future if you find NGINX truncating large files for no apparent reason, this is likely why.

*EDIT* — In NGINX versions post 0.7.28, the default limit of gzip_buffers has been increased to 32 * 4 or 16 * 8 (rounding out to 128K, as noted above) depending on platform. It may be wise to double check any configs you may have to ensure you are not explicitly setting a lesser value, or upgrade NGINX if needed.

The Introduction to the Lighthouse API mentions something you might do with the Lighthouse API is “Accessing and creating tickets through your Instant Messenger client.”. How easy is it to do this? Surprisingly easy.

You will need to download and install the Lighthouse API gem and the Net::TOC gem.

Run this in an irb session.

require 'net/toc'
require 'lighthouse'
require 'sanitize'
include Lighthouse

Lighthouse.account = 'shaft'
Lighthouse.token = 'hesonebadmotherfushutyourmouth'

Net::TOC.new("someaimaccount", "someaimpassword") do |msg, buddy|
project = Project.find(Sanitize.clean(msg).to_i)
buddy.send_im("Here is some information about your project: #{project.name}")
end

You should then be able to add ’someaimaccount’ to your AIM buddy list and send a project ID and have it return the project name.

And boom goes the dynamite!

You’ll of course need to change the Lighthouse account and token as appropriate (or use username and password for logging in). msg is a little weird and although it’s a string, it’s HTML, so you need to strip out any tags before doing anything. And of course you’d want to add in some way of parsing the input from the user into some DSL (Domain Specific Language) for interacting with your fancy new Lighthouse AIM bot.

Happy hacking!