The Blog: Infrastructure

(more…)

It’s no secret that Agorian systems folk favor NGINX for our web serving needs. We’ve written about it a lot before. Therefore, it should be no surprise that we end up making a lot of things designed to work on Apache work on NGINX. (We’ve also written about that before, come to think of it…)

One example is vBulletin. A number of Agora’s sites are powered by the forum software, which comes with rewriting rules for Apache’s mod_rewrite and IIS… but not NGINX.

So, if you’re interested in setting up vBulletin behind NGINX (and are using the advanced URL rewriting, like we are), you can find a sample configuration for doing so here.

Let us know if you have any questions!

When I joined Agora, one of the first things I did was talk up git and how it’ll cure cancer, AIDS, and solve world peace. All at once. What that means for me is I’ve basically been tasked with the job of migrating anything that’s not git to git.

For some things these kinds of migration are first class citizens. Conveniently SVN, our old VCS is one of those. One of my new migrations was, less conveniently, Bazaar. Now we have nothing against Bazaar at Agora. I mean my main personal open source project, Exaile, uses Bazaar. But we agreed we would rather only have one VCS in house.

After looking around and trying some fancy tools that didn’t work (read: tailor), I stumbled on a really quick solution that seems like it does everything necessary. Both Git and Bazaar (via plugins) support the fast-import/export format. I’m not sure about the mystic ways of how this format works but I do know it made my Bazaar repository a Git repository, and that makes me pleased.

Getting the bzr plugin

The first step would be to get the fast-import plugin for Bazaar from the launchpad mirror.

mkdir -p ~/.bazaar/plugins
cd ~/.bazaar/plugins
bzr clone lp:bzr-fastimport fastimport

You can make sure it installed properly using a bzr fast-export --help and ensure that it doesn’t complain.

Copy the repository

Now that we have all the tools, time to copy it over

mkdir ~/project.git
cd ~/project.git
git init
bzr fast-export --plain ~/path/to/bzr/branch | git fast-import
git checkout master  # only needed for a non-bare repository, like I made above

Wait a little while (or a long while if you’re testing the above code on a netbook for some reason like me). And that should be it.

I’m not sure how well this works with multiple Bazaar branches. There may be some crazy --flags on each side to make it work but running the code I put above on a full repo makes fast-export complain that I’m not pointing it to a valid branch. Please give us your comments if you know how to do this :).

Update: Found out it was .bazaar not .bzr. My bad.

As a SysAdmin, my job more-or-less exists by knowing miscellaneous arcana that most software engineers aren’t aware of.

When a particular co-worker here at Agora has a problem with his Linux machine, I provide advice and show him how to fix it. Some months after he had joined Agora, I discovered that after each troubleshooting session, he copy-and-pastes the entire text terminal log into a text file that he keeps on his desktop. He has dozens of transcripts at this point; I bet if I were to look through it, it would read something like the Tao te Ching or Bhagavad Gita, only concerning UNIX instead of right living.

(Of course, there’s not much of a difference between UNIX and right living, but that’s a topic for another day.)

In the spirit of allowing you to build your own little collection, here is a simple trick that came in handy to me yesterday.

(more…)

Yesterday I started looking at ways to do inter-application communication. In a number of projects we’ve done here at Agora Games, we’ve used queues to make that happen. Redis has been on my radar for awhile now, but yesterday I drove my Chevy to the levee and guess what? The levee is NOT dry people. I mean, who drinks rye anyway these days? Old people.

(more…)

I had occasion tonight to give a quick demo of CouchDB.  This is so simple its almost not worth blogging about, but hey, good software is supposed to be simple.

Install
On your Mac:

sudo port install couchdb
sudo launchctl load -w /Library/LaunchDaemons/org.apache.couchdb.plist

On Ubuntu:

sudo apt-get install couchdb

You’re done; now to play.

Futon Admin Interface
This allows you to create a DB, create records, etc.

1. Open http://localhost:5984/_utils/
2. …
3. Prosper

Using our old friend curl

curl -X PUT http://localhost:5984/mlg/ #create a db
curl -X GET http://localhost:5984/mlg/ #get a whole bunch of info about the db
curl -X POST http://localhost:5984/mlg/ -H "Content-Type:application/json" -d '{"body": "Here is a paragraph"}' #create a record

I ran into an interesting issue today when deploying a crappy password change app I wrote as an exercise in rails. It turns out that by default, NGINX has a gzip buffer size of 4 * 4k/8k, with the bit size depending on what platform the service is running on.

(more…)

At Agora we’re experimenting with Splunk as a error collection and reporting tool.  The idea is that all our services will spit out error information to syslog which will be picked up and indexed by Splunk.  Splunk will be configured to display info about recent errors, email summaries of those errors and generally be the starting point for discovering problems with our systems.  In our previous infrastructure error collection and reporting was handled on a per-project basis.  As we’re now moving to a more service oriented approach that involves a small number of generic but highly configurable services deployed over a large number nodes we needed something different.  Splunk seems ideal for this.  It accepts a number of different data sources ranging from syslog to raw log files to inputs from random UDP and TCP ports so we can easily integrate with systems we’ve written and, more importantly, those we haven’t.  It also has the summary and reporting capabilities we were looking for so we can chart errors over time periods, dig for trends and see which services are acting up.

(more…)

Those of you in the Rails community who use NGINX may have come across this before: Ezra Zygmuntowicz’s NGINX config. It bears some similarity to the ones we use at Agora. Specifically, the section relating to serving page-cached files. The relevant lines (pulled from one of our configs) looks like this:

if (-f $request_filename.html) { rewrite (.*) $1.html break; }
if (-f $request_filename/index.html) { rewrite (.*) $1/index.html break; }
if (!-f $request_filename) { proxy_pass http://some-proxy; }

This generally works well, for simple cases. Unfortunately, we’ve run into some more complex ones where it didn’t cut it. I’ll outline two of those cases below and show what we did to fix it.

(more…)

Well, we’re back from Vegas! And have been, for a couple weeks… I’ve been meaning to put up some follow-up resources for my talk (PWN Your Infrastructure: Behind Call of Duty: World at War), but there was just so much work to do when I got back… such is the life of a system administrator!

That said, I’ve got some free moments, so I’m putting up some reference materials.

(more…)