The application, and cluster deployment in general, is hamstrung by Erlang’s standard approach to cookies. The cookie, typically .erlang.cookie in the HOME directory of an application, must have only user read permissions. For a Linux RabbitMQ install, that would look like this:

bofh@rabbit_host1:~$ ls -al /var/lib/rabbitmq/
total 333
drwxr-xr-x  3 rabbitmq rabbitmq    160 2010-07-12 17:49 .
drwxr-xr-x 37 root     root       1008 2010-07-02 14:29 ..
-r--------  1 rabbitmq rabbitmq     20 2010-05-04 20:44 .erlang.cookie
drwxr-xr-x  3 rabbitmq rabbitmq     72 2010-05-04 20:49 mnesia
-rw-r--r--  1 rabbitmq rabbitmq     27 2010-07-12 17:49 pids

Any variation on these permissions and Erlang will refuse to start. The permissions requirement is embedded into Erlang itself, making it more or less impossible to work around. This creates the following problems:

• You must copy this file (or its contents) to all hosts in the cluster
• All users of rabbitmqctl must run it as sudo
• All monitoring tools must also run as root or have sudo capability
• You must have the cookie file present on the host running rabbitmqctl

Given how powerful rabbitmqctl is, you will likely still want to limit access to it, but this can be readily accomplished with standard Unix permissions. By expanding who can use it and where, your system administrator will be thrilled to reduce need for sudo access and your customers will be happy with the additional tools you can deploy to monitor your cluster and its clients to ensure that all is well, 24/7.

Thankfully, RabbitMQ’s scripts accept a lot of environment variables which can be passed into the Erlang runtime, and Erlang kindly provides a way to bypass the cookie file through a command line literal. The scripts are all located in the scripts directory inside your RabbitMQ installation.

bofh@rabbit_host:~$ ls -ald `erl -eval 'io:format("~s~n", [code:lib_dir()])' -s init stop -noshell`/rabbitmq-server*/scripts
drwxr-xr-x 2 root root 520 2010-07-21 14:47 /usr/local/lib/erlang/lib/rabbitmq-server-1.7.0/scripts
drwxr-xr-x 2 root root 520 2010-07-21 16:15 /usr/local/lib/erlang/lib/rabbitmq-server-1.7.2/scripts

The three standard scripts, rabbitmq-server, rabbitmq-multi and rabbitmqctl all source an environment script, rabbitmq-env, which in turn will source the file /etc/rabbitmq/rabbitmq.conf if it exists. This is the file that you can edit to take control of your RabbitMQ cluster.

SERVER_START_ARGS="+K true +A300 +P512000 -setcookie NOMNOMNOMYUMYUM -kernel inet_default_listen_options [{nodelay,true},{sndbuf,32768},{recbuf,32768}]"
MULTI_START_ARGS="-setcookie NOMNOMNOMYUMYUM"
CTL_ERL_ARGS="-setcookie NOMNOMNOMYUMYUM"

The value of each environment variable will be passed verbatim to the Erlang runtime. The SERVER_START_ARGS are passed to each node started by rabbitmq-multi, and directly affect the performance of a RabbitMQ instance. In this example, we have increased the number of native threads available to Erlang and allowed it to spin up numerous (Erlang) processes. We have also instructed the kernel to increase its TCP buffer sizes and disable Nagle's algorithm.

You can now run rabbitmqctl without sudo on any host which has this same configuration deployed. This solution still requires that you synchronize /etc/rabbitmq across your cluster, though there are many ways of solving that problem.

We have taken this a step further and enabled rabbitmqctl on any host in our network. All of our core software is deployed from source to /usr/local over NFS. We have deployed the rabbitmq.conf file there too, and have a small shell script wrapping rabbitmqctl.

#!/bin/bash
ERL_PATH=`erl -eval 'io:format("~s~n", [code:lib_dir()])' -s init stop -noshell`
LATEST_VERSION=`ls $ERL_PATH | grep rabbitmq | sort | tail -1`
RABBIT_PATH="$ERL_PATH/$LATEST_VERSION"

set -f
[ -f /usr/local/etc/rabbitmq/rabbitmq.conf ] && . /usr/local/etc/rabbitmq/rabbitmq.conf
export CTL_ERL_ARGS
HOME=/var/lib/rabbitmq $RABBIT_PATH/scripts/rabbitmqctl $@

We can now connect to and monitor any RabbitMQ node from any host on our network

bofh@gateway:~$ rabbitmqctl -n rabbit@rabbit_host1 status
Status of node rabbit@rabbit_host1 ...
[{running_applications,[{rabbit,"RabbitMQ","1.7.2"},
                                {mnesia,"MNESIA  CXC 138 12","4.4.10"},
                                {os_mon,"CPO  CXC 138 46","2.2.2"},
                                {sasl,"SASL  CXC 138 11","2.1.6"},
                                {stdlib,"ERTS  CXC 138 10","1.16.2"},
                                {kernel,"ERTS  CXC 138 10","2.13.2"}]},
 {nodes,[rabbit@rabbit_host1,rabbit@rabbit_host2]},
 {running_nodes,[rabbit@rabbit_host1,rabbit@rabbit_host2]}]
...done.

One example is vBulletin. A number of Agora’s sites are powered by the forum software, which comes with rewriting rules for Apache’s mod_rewrite and IIS… but not NGINX.

So, if you’re interested in setting up vBulletin behind NGINX (and are using the advanced URL rewriting, like we are), you can find a sample configuration for doing so here.

Let us know if you have any questions!

For some things these kinds of migration are first class citizens. Conveniently SVN, our old VCS is one of those. One of my new migrations was, less conveniently, Bazaar. Now we have nothing against Bazaar at Agora. I mean my main personal open source project, Exaile, uses Bazaar. But we agreed we would rather only have one VCS in house.

After looking around and trying some fancy tools that didn’t work (read: tailor), I stumbled on a really quick solution that seems like it does everything necessary. Both Git and Bazaar (via plugins) support the fast-import/export format. I’m not sure about the mystic ways of how this format works but I do know it made my Bazaar repository a Git repository, and that makes me pleased.

Getting the bzr plugin

The first step would be to get the fast-import plugin for Bazaar from the launchpad mirror.

mkdir -p ~/.bazaar/plugins
cd ~/.bazaar/plugins
bzr clone lp:bzr-fastimport fastimport

You can make sure it installed properly using a bzr fast-export --help and ensure that it doesn’t complain.

Copy the repository

Now that we have all the tools, time to copy it over

mkdir ~/project.git
cd ~/project.git
git init
bzr fast-export --plain ~/path/to/bzr/branch | git fast-import
git checkout master  # only needed for a non-bare repository, like I made above

Wait a little while (or a long while if you’re testing the above code on a netbook for some reason like me). And that should be it.

I’m not sure how well this works with multiple Bazaar branches. There may be some crazy --flags on each side to make it work but running the code I put above on a full repo makes fast-export complain that I’m not pointing it to a valid branch. Please give us your comments if you know how to do this :).

Update: Found out it was .bazaar not .bzr. My bad.

When a particular co-worker here at Agora has a problem with his Linux machine, I provide advice and show him how to fix it. Some months after he had joined Agora, I discovered that after each troubleshooting session, he copy-and-pastes the entire text terminal log into a text file that he keeps on his desktop. He has dozens of transcripts at this point; I bet if I were to look through it, it would read something like the Tao te Ching or Bhagavad Gita, only concerning UNIX instead of right living.

(Of course, there’s not much of a difference between UNIX and right living, but that’s a topic for another day.)

In the spirit of allowing you to build your own little collection, here is a simple trick that came in handy to me yesterday.

Last night, the cron jobs powering one of our managed sites stopped running. I had stopped them, cleaned up, and restarted them, but I wanted to watch the system and see if they had started running again so I could monitor that they were doing their job.

UNIX has a well-known command called “top,” which shows superlative processes running on your system: which ones are consuming the most CPU time, which ones are consuming the most RAM, which ones have been running the longest, and so on. It’s a very useful tool, but if you’re trying to track down processes over time that don’t necessarily consume very many resources, then it’s not the tool you’re looking for.

However, our Linux distribution (and, indeed, most others) provide a useful graphical tool for mapping the current state of the system: “pstree.” This command will give you a list of processes (much like the “ps” command), except that instead of generating a list, it will draw a little tree showing the state of the system. This is perfect for seeing exactly what is going on: which daemons are running, who is on the system and what they’re executing, and so on. However, it doesn’t show you what is happening over time; it merely lists what is happening at this moment.

However, our Linux distribution (and, indeed, most others) provides yet another useful tool: “watch.” It runs a program every 2 seconds (by default) and displays the result to your screen. This allows you to make a ghetto clock by writing something like “watch date;” in our case, we could make a ghetto graphical “top” by writing the simple “watch pstree.” You can now watch your virtual trees rustle in the wind.

It kept us on top of the cron job, and things were back to normal in no time.

For more information, RTFM. :)

Right, Redis.

“Redis is an advanced key-value store. It is similar to memcached but the dataset is not volatile, and values can be strings, exactly like in memcached, but also lists, sets, and ordered sets. All this data types can be manipulated with atomic operations to push/pop elements, add/remove elements, perform server side union, intersection, difference between sets, and so forth. Redis supports different kind of sorting abilities.”

I was particularly interested in the support in Redis for lists, which would allow me to have one application push stuff into the datastore and allow for another application to pull stuff from the datastore. A datastore that could operate as a queue? I guess this is as close as we’re going to get to flying cars in 2010. Redis is also supposed to be wicked fast and there are a number of libraries available in your programming language of choice with which to communicate with Redis.

Redis setup was trivial and “just worked”.

After starting the Redis server, I could just prototype in script/console.

Here we go:

>> redis_queue = Redis.new
=> #<Redis:0x222df9c @thread_safe=nil, @logger=nil, @password=nil, @timeout=5, @db=0, @sock=nil, @host="127.0.0.1", @port=6379>
>> redis_queue.push_tail 'strings', 'string 1'
=> "OK"
>> redis_queue.push_tail 'strings', 'string 2'
>> "OK"
>> redis_queue.list_length('strings')
=> 2
>> redis_queue.list_range('strings', 0, -1)
=> ["string 1", "string 2"]
>> some_string = redis_queue.pop_head('strings')
=> "string 1"
>> r.list_range('strings', 0, -1)
=> ["string 2"]
>> r.list_length('strings')
=> 1
>> quit

There are quite a few commands that make Redis even more awesome, but for now I’m sold. (If using an entire sentence as a link is wrong then I don’t want to be right)

Also, I know of Resque, “a Redis-backed library for creating background jobs, placing those jobs on multiple queues, and processing them later.” If I ever feel like driving the Mercedes Benz of flying cars, I’ll use it.

Install
On your Mac:

sudo port install couchdb
sudo launchctl load -w /Library/LaunchDaemons/org.apache.couchdb.plist

On Ubuntu:

sudo apt-get install couchdb

You’re done; now to play.

Futon Admin Interface
This allows you to create a DB, create records, etc.

1. Open http://localhost:5984/_utils/
2. …
3. Prosper

Using our old friend curl

curl -X PUT http://localhost:5984/mlg/ #create a db
curl -X GET http://localhost:5984/mlg/ #get a whole bunch of info about the db
curl -X POST http://localhost:5984/mlg/ -H "Content-Type:application/json" -d '{"body": "Here is a paragraph"}' #create a record

As a result,

4 * 4k = 16384 bytes (16KB)
4 * 8k = 32768 (32KB)

After gzip, when converted to be plain text, the amount of data returned to the browser nearly doubles (about 52KB in the case of larger). The default javascript library included by rails for AJAX (prototype.js) is 127k, which led me to find this limit.

And so, getting to the point, the limit applied to the gzip buffer needs to be increased within NGINX’s config files for each site that will use a file larger than these pre-set sizes. You can do so using the following:

gzip_buffers 16 8k

This will set the buffer size to 16 * 8k, or 128KB — the uncompressed size of the library, eliminating the issue.

In the future if you find NGINX truncating large files for no apparent reason, this is likely why.

*EDIT* — In NGINX versions post 0.7.28, the default limit of gzip_buffers has been increased to 32 * 4 or 16 * 8 (rounding out to 128K, as noted above) depending on platform. It may be wise to double check any configs you may have to ensure you are not explicitly setting a lesser value, or upgrade NGINX if needed.

The biggest problem I’ve had so far while working with Splunk is the documenation.  There are a lot of options and the documentation for those options is spread out over a number of distinct manuals.  Some options are poorly documented or not at all and often it’s not clear, even once you’ve found the proper option, where/how the option should be used.  The following is an attempt to lay out some of those problems I ran into and what the solutions are so you don’t have to go looking yourself.  This list will likely continue to grow over the coming weeks and months.

1. Extracting fields from search results

   Splunk automatically extracts certain fields from your search results.  This is all well and good but what if you want to pull out your own values?  As an example we stick a field into our syslog output that indicates what service the log entry came from.  This would be extremely handy since we’d then be able to filter out entries for a particular services, build charts with color coding based around the services and so on.  It turns out this is pretty easy.  Although this functionality is mentioned in the ‘User Manual’ most of the ral documentation for the commands used can be found in the ‘Search Manual’ (grrrr).  The portion in the User Manual can be found by search for ‘Extract fields with search commands’.  For the command information look for ‘rex’ in the ‘Search Commands’ section of the Search Reference.Anywho, the trick is in the ‘rex’ command.  This can be chained onto your search request (something else that was not obvious) like many other Splunk commands:

   sourcetype="syslog" foo | rex field=_raw "[0-9a-z]{2} err \[(?<service>.*?)\]"

   The rex command takes a regular expression and produces fields based on matches.  The field option specifies which of Splunk’s default fields you want to match against (search in the Splunk manuals for ‘Use default and internal fields’ for a list of all available fields).  The regex in this case matches two alphanumeric characters followed by the string ‘err’ followed by a service name in brackets.  The ?<service> notation in parentheses indicates a match that should be saved in the ’service’ field.

   Once your search has been executed you should see that you can now filter on whatever fields you extracted, create charts with them, etc.

2. Extracting only those results that have a particular field

   So you’ve created a regex to extract certain fields from your result set but the results that don’t match your regex are still in the result set.  How do you filter those out?  The solution is actually pretty simple:

   <your_original_search> | rex <your regex> | search your_field="*"

   Likewise you can find those that didn’t match by negating the lookup:

   <your_original_search> | rex <your regex> | search NOT your_field="*"

3. Changing indices for search

   By default your searches will only hit the ‘main’ index.  If you’ve created your own index it won’t be searched by default.  To use your index in a particular search just add it to the search string:

   index=my_index foo NOT bar

   To add your index to the default list so you don’t have to modify every search you’ll need to modify your roles a bit.  Visit the Roles page in the Manager, select your role (probably user or admin if you haven’t made any modifications here, and scroll down to the ‘Default indexes’ list.

4. Displaying the body of a message in an event table

   In our main dashboard we wanted to show a list of the 5 most recent errors.  This actually turned out to be more difficult than you might have expected.  The date, host, and a few other pieces of data showed up fine but for the life of me I couldn’t figure out how to get the body of the log message to show up in the table.  There is a special field called _raw for each result but adding that to the table had no effect.  In the end my solution was to create a regexp against my search results that extracted a parameter called ‘body’.

   Search:

   sourcetype="syslog" err | rex field=_raw "[0-9a-z]{2} err \[(?<service>.*?)\] (?<body>.*)"

   Event Table:

   <table>
   <title>Most recent service errors</title>
   <searchName>Service errors with body</searchName>
   <option name="count">5</option>
   <fields>_time,host,body</fields>
   </table>

The events table:


5. Changing the default app

   With a normal install the launcher is the default app.  This gets kind of annoying after a while.  To change it to your own app you can adjust the default_namespace option in $SPLUNK_HOME/etc/apps/user-prefs/local/user-prefx.conf.

   default_namespace = my_app

6. Email alerts

   Email alerts are built around scheduled searches.  When saving or editing a search you’re given the option to provide a schedule for it.  Change this option so that the search is run on some interval and additional options will appear including one to send an email alert.  Additionally you can modify the search to only include a certain time range.  These features are very handy when used in combination.  Schedule your search to run every 30 minutes and adjust the start time to -30m and you’ll get alerts about all errors that occurred during that time.

   Mail settings themselves can be found in Manager -> Email alert settings.

7. Stacked time-based charts

   Took me a while to sort out this one.  The first thing you need to do is group your data into larger time periods.  If you have one column for each second your graph is going to be pretty useless.  Instead you should consolidate your data into large groups such as one per minute or one per hour.  I’ve found that dividing your time range by roughly 30 gives the best graphs.  To actually perform this grouping you need to add the ’span’ option to your search:

   sourcetype="syslog" foo NOT bar | timechart span=1h count

   This will produce a count of all the items containing foo and not bar for syslog in intervals of one hour.

   Next you need to stack the resulting groups.  Annoyingly this is done at the view layer.  In my case I was adding it to my dashboard.  The relevant lines looked similar to the following:

   <chart>
   <searchName>Service errors in the last 24 hours</searchName>
   <title>Service errors in the last 24 hours</title>
   <option name="charting.chart">column</option>
   <option name="charting.chart.stackMode">stacked</option>
   </chart>

   There is a reference for all these options in the Splunk docs.  Search for ‘Custom charting configurations’.

   The end result looks something like this:
   

8. Edit everything manually

   The Splunk Manager interface is rather clunky (not entirely the developer’s fault, there are a lot of interdependent options) so after a while you’ll likely get sick of it.  That’s perfectly fine.  All the options are written out to config files per app.  All your changes are written out to $SPLUNK_HOME/etc/apps/<your_app>/local and override those options in $SPLUNK_HOME/etc/apps/<your_app>/default.  The default directory can be used as a handy reference for available options as well.  Note that some system wide options such as indices and roles are written out to $SPLUNK_HOME/etc/system/local.

9. Don’t delete the ‘user’ role

   If you do you’ll be sad as it breaks everything.  I’d attempted to remove the role and replace it with our own modified version.  The result was that every page I visited gave me a “Client not authorized” error.  To resolve the problem delete the ‘disabled = 1′ line from $SPLUNK_HOME/etc/system/local/authorize.conf.

10. Adding output to syslog(-ng)

    As mentioned previously our Splunk setup is largely based on syslog but because we want to have raw syslog files in addition to having them indexed by Splunk we allow syslog to write the logs to disk and then have Splunk watch the created files.  Some information such as facility and log level get lost during this process with most default syslog configurations.  To resolve this we did a little bit of reconfiguring to our syslog installs.First, it’s important to note that we actually use syslog-ng rather than vanilla syslog.  Syslog-ng has some handy options for templating output as of 1.5.3.  For each destination you can specify a template function:

    destination df_syslog { file("/var/log/syslog" template("$FULLDATE $FULLHOST $TAG $LEVEL $MESSAGE\n") template_escape(no)); };

    FULLDATE includes the year in the timestamp.  FULLHOST provides the hostname.  TAG is a hex encoding of the facility and level.  $LEVEL is a text representation of the level.  $MESSAGE is obviously just the message.  The template_escape(no) directive says that quotes should not be escaped.  Also note the newline at the end of the template.  Without it your output will be rather unreadable.

    And for reference the default appears to be similar to:

    "$DATE $FULLHOST $MESSAGE\n" template_escape(no)

if (-f $request_filename.html) { rewrite (.*) $1.html break; }
if (-f $request_filename/index.html) { rewrite (.*) $1/index.html break; }
if (!-f $request_filename) { proxy_pass http://some-proxy; }

This generally works well, for simple cases. Unfortunately, we’ve run into some more complex ones where it didn’t cut it. I’ll outline two of those cases below and show what we did to fix it.

The first problem came up when we were page-caching both HTML and XML files (that is, there might be a slots/show/47.html and a slots/show/47.xml. (You can’t do this by default in Rails. We had to patch it.) The problem was that when you request /slots/show/47, the NGINX config only checks for and serves the HTML file, even when what you really wanted was XML. Of course, checking for the XML file gives the reverse problem: if you’re in a web browser, you get XML instead of pretty HTML. Not so hot.

My solution was to assume that the user wants XML unless their browser says otherwise (via the HTTP Accepts: header), in which case we check for HTML first instead. If we can’t find the relevant file, we try with the other extension (in case we guessed wrong), and finally we kick the request to the proxy if we can’t satisfy it. Thus, it looks like this:

# figure out the order we should search the cache
set $preferred_format xml;
set $secondary_format html;

if ($http_accept ~* text/html) {
  set $preferred_format html;
  set $secondary_format xml;
}

# check the cache for cached files
if (-f $request_filename.$preferred_format) {
  rewrite (.*) $1.$preferred_format break;
}

if (-f $request_filename/index.$preferred_format) {
  rewrite (.*) $1/index.$preferred_format break;
}

if (-f $request_filename.$secondary_format) {
  rewrite (.*) $1.$secondary_format break;
}

if (-f $request_filename/index.$secondary_format) {
  rewrite (.*) $1/index.$secondary_format break;
}

# if the file doesn't exist, let the application sort everything out
if (!-f $request_filename) { proxy_pass http://some-proxy; }

The second case was a bit more subtle, but is similar in theory: one of our sites uses proper REST (that is, we support the HTTP GET, PUT, DELETE, etc. methods). When a user performed a GET request, the page gets cached; if they perform a DELETE sometime after that, NGINX (which is not configured to handle REST by default) simply returns the cached file happily instead of forwarding the DELETE request to the application. (As you can imagine, tracking down the source of this problem was not the easiest thing in the world.)

The solution in this case was to disable page caching for GET requests. Unfortunately, NGINX’s support for if statements isn’t so hot, so we had to do something fairly hacky to get it to work. (For the record, while I’ve not been enthused with the if statements, I tend to like NGINX in most other areas, so a couple silly warts like this are excusable for me.)

# Look for "/foo.zmog" unless we're asking for a GET request, in which case
# we look for "/foo.xml". Since "/foo.zmog" will never exist, this means
# that we will only check for pagecached files on a GET request. (Yes, this
# is totally ass-backwards. NGINX is awesome except in this case.)
set $cache_suffix "zmog";
if ($request_method = GET) {
  set $cache_suffix "xml";
}

# Caching rewrites
if (-f $request_filename.$cache_suffix) {
  rewrite (.*) $1.$cache_suffix break;
}

if (-f $request_filename/index.$cache_suffix) {
  rewrite (.*) $1/index.$cache_suffix break;
}

# More metrics logging for cached files
if (!-f $request_filename) { proxy_pass http://switchblade_proxy; }

As you can see, sometimes, making NGINX play nice requires some lateral thinking, but you can generally always get it to do what you need to.