Note: This editorial does not express the feelings of Agora Games as a company, and is solely the opinion of the author.

Art, by definition is the expression or application of human creative skill and imagination typically in a visual form. A video game by definition is a game played by electronically manipulating images produced by a computer program on a television screen or other display screen. By both definitions of each word, video games may be considered a form of art. The computer program which electronically manipulates images on a display screen must be created by a human. This human, must apply creative skill to create such images as well as the manipulation of images, images being a visual form. The definition of art clearly states that a the expression of human creativity is typically in a visual form, which video games by definition clearly do. Therefore, technically disregarding all personal opinions, culture, and other variable elements video games by definition are art. 

I strongly believe that video games are a form of art. Video games in fact include almost every single element of art there is. Let us analyze one of the most “accepted” forms of art in society which is included in video games, literature. Many video games these days staff writers to create a story for the particular game at hand. In fact, many game journalists include the quality of story within their review of a game. How much different is that then the New York Times reviewing a new novel? In fact, with games currently averaging about 20+ hours of gameplay it is safe to say that a well developed short story, or even a novel may be included within a video game. 

While one might argue that the quality of story is rather poor does this not also hold true to several novels? Sure one might say Gears of War had very poor dialogue and a rather poor story progression however, other titles provide players with a very well developed story and dialogue. An example of such a game in my opinion would be Mass Effect, in which a character Lorik Qui’in after you the player say, “I will let you go.” states the following, “Let me go? Do humans consider conversation a form of imprisonment? That would explain why so few of you are willing to sit down and talk…” This dialogue being directed at my character, therefore an extension of myself since I am controlling the character. The communication is not passive in nature, it is directed at myself and therefore directly impacts my thoughts and feelings on a personal level since it is I who am controlling this character, I who am a living breathing human being. The character being addressed is not print on a paper, it is my character, my choices which insinuated this response from Lorik. 

This method of immersion can not be achieved through a novel. Koster, touches on this subject by saying, “…games are very fundamental and powerful learning tools. It’s one thing to read in a book that “the map is not the territory” and another to have your armies rolled over by your opponent in a game. When the latter happens, you’re gonna get the point even if the actual armies aren’t marching into your suburban home.” Furthermore, video games are an accessible medium to any given audience. That is to say, while a certain skill is required to play a video game, both a novice gamer and an experience gamer will both be able to direct their in game experience into the subconscious, intuitive mind. However, in the art of literature, an experienced reader will gain a better experience into their subconscious, intuitive mind rather than a novice reader. 

Koster, in fact hits on this topic by comparing literature to games. “Consider the key differences between something like a book and different kinds of games. A book can do the logical conscious part of the brain pretty well. And really good readers have an ability to slurp that info directly into the subconscious, intuitive mind. But what a book will never be able to do is accelerate the grokking process to the degree that games do, because you cannot practice a pattern and run permutations on it with a book.” In essence, within a book you do not have the option to experience a scenario or event which you caused and observe the various outcomes of your actions. 

In addition to providing players with an immersive experience through literature, video games also provide creative visual elements which allow players to continue their immersion in a video game. Furthermore, combining both visual and literary elements provide the player with a unique experience which can not be simulated by any other form of art. Artists, art directors, and creative directors work in unison to give players a certain “feeling” or evoke a certain “emotion” while playing the game. 

The complexity and infancy of video games make the legitimacy of video games as a medium of art rather difficult to swallow. However, many other forms of art when first introduced went through the same life cycle. From the 1900s to the 1940s on screen films went from short black and white movies to full length hour long movies fully dressed with color and sound. As video games begin to mature and designers, programmers, artists, and musicians begin to become more experienced within this form of art I can only hope that the social stigma and media literacy problem evolves as well. One day I truly believe we will be walking through a museum watching interactive displays featuring some of the greatest forms of art, video games. 

Boston is a city in Massachusetts that was founded a whole buncha years ago.  Back in the day, the guys pushing wagons around, and mules laden with baggage didn’t care about paths that they plodded into the ground.  All they cared about was getting their goods to where ever they needed to be.  If there was a ginormous rock in the way, they could just path around it.  After a while, cars started getting pretty cool.  Cars need roads, so all the roads were built right on top of those paths.

At that time, it was easier to just build a road on top of the path.  No one knew the windy mess of wagon-path converted roads would cause tons of lost tourists and traffic accidents today.  It might have been better to just nuke the ginormous rock out of the way, and make the road straight.  Unfortunately, that ginormous rock is some type of historical monument now, so it’s not possible anymore.

Anything is possible in software (with limit to it’s own scope), the only constraint is time.  Time is the key element here, it’s something you can’t get back.  Every decision that is made in software has an effect on how long something else will take.  Every second saved by hacking around something today, is going to cost you two seconds tomorrow.

“Tomorrow is so far away though, and I need this to work now.”  When the pressure is on, and time is short, this is a common approach to decision making.  It’s in these essential minutes during an applications life span, that hacks start to leak into every crevice.  To make things worse, instead of refactoring the hacks out, time is spent building more new features.  Eventually those crevices become a major fault line, and it causes changes to the code base to take three times as long.  By this time, new developers have no chance of contributing, and the existing developers hate working on the application because everything they do causes an earthquake.

A major problem with software development is the existing solution.

Things change, standards form, upgrades happen.  The best idea you ever had yesterday, can probably be done better today.  Don’t let your legacy code stink take over, get rid of it, refactor it, and upgrade it.  There is always a better way to approach a problem in software and in most every case the quickest solution is the worst solution.

      That’s right, myself and three others flew over to Anaheim, California to the Anaheim Convention Center for Blizzard’s 3rd Annual BlizzCon!  This year Blizzard featured all three of its games at the Con: World of Warcraft, Starcraft and Diablo. I decided that despite the embarrassment which I might endure, I would dress up as a Blood Elf, featured in Blizzard’s World of Warcraft: The Burning Crusade. I was a bit nervous at first. While approaching the convention center I saw no one dressed up in costume in sight, and there I was, sporting these ridiculous elf ears.  Once I entered the convention center I was astounded.  About 40% of the 15,000 people who showed up for the convention were dressed up in the most elaborate outfits.  From bulked up Horde warlocks to Alliance elves skimpy bikinis, it was all there.  Our goodie bags were amazing too!  Filled to the brim with fun swag, including a blow up Frostmourne, keychains, Starcraft “Zerg Creep”, Diablo III stress balls, BlizzCon hand sanitizer, mints, Warcraft trading card game starter kit, bottle opener, n00b tissues, paper WoW masks, Starcraft II beta key, polar bear mount for WoW, mouse pads, and these nifty Blizzard Digipass Authenticator by Vasco!

      Tickets to Blizzcon were about $100, which was pretty good considering everything at the Con other than food was free.  They had free pictures with a background of the game of your choice. You can view more of these photo’s at www.politeinpublic.com  Someone actually proposed in one of the photos - and don’t worry, she said yes!

      During many of the developer panels - where they explained the new game World of Warcraft: Wrath of the Lich King - I was texting our lead QA pro Dave Loya of all of the awesome new details for the new unveiled class, The Death Knight.  The Con also had several computer set-ups for participants to play Starcraft II, World of Warcraft: Wrath of the Lich King and Diablo III.  I ended up demoing Diablo III, and it truly looks fantastic!  They have some more work to do on the game, but they did unveil a new class as well, the Wizard! 

      The Con itself was a sight to see.  There were meet and greets galore! Participants had the opportunity to meet with Blizzard employees and obtain free swag such as professional pictures, figure prints, trading cards and so much more!  We all waited in line for about two hours to get plush Murocs, which Jinx was all sold out by 6 p.m the first evening.  I tried to get pictures of everyone that were dressed up,  I even found a few fellow Blood Elves!  My crew was very recognizable due to the fact that we were all wearing red matching Tabards.  They are a exact replica of our guild tabards in game which are The Bloodrocutioners.  Chelsea (Designer, 1st Playable Productions) made all of the Tabards  worn at the Con all on her own with some sewing help from our other friends who came along for the trip.  The painstaking work totally paid off as people were shouting “Look!  Guildies!” and snapping pictures in our faces.  I think we averaged about 150 photos taken of us by random people per day.  By 10p.m. both nights I was seeing spots.  I couldn’t believe that if you were dressed up, you were photographed like a celebrity.  I even had one kid ask us to sign his Warcraft book with our names, realm name and guild name.  There were also many gals paid by Blizzard to model official WoW costumes! Also, My crew and myself were interviewed my two different websites!

      The final night had a great concert by Level 80 Elite Tauren Chieftain, which is Blizzards metal band. Even the CEO was in the band!  The Orchestra concluded the evening with powerful music from Warcraft, Starcraft and Diablo.  The theme music from World of Warcraft I must say, did give me goosebumps. I encourage anyone who plays any of the Blizzard games to check out the next Con in 2009, because I know I will be there!  It was an amazing time and so surreal with the ambience of gaming and fantasy combined into two fun filled days which was BlizzCon 2008!

Guild Interview

http://www.gamasutra.com/view/feature/3828/building_social_communities_for_.php

Sometimes, things go horribly, horribly wrong, and there’s nothing you can do to stop the slow-motion train wreck unfolding before your very eyes. (Like that one senior prom where you spilled your dinner all over your date’s dress. On the upside, you didn’t need that $200 anyway.) When things go that awry, there’s no way to fix it. (She never did return your phone calls, did she?) The only thing you can do with a mistake like that is to make sure it never, ever happens again.

For most of the sites we produce, we use MySQL as a backend. While it’s extremely rare for the MySQL server to catastrophically fail, it happens, and the more MySQL servers you have, the more likely it is to happen at least once. To keep this from causing problems for us, we replicate our databases (that is, maintain a live copy of them). This provides us with two benefits:

1. If the MySQL master (primary database) blows up, melts down, or gets eaten by vermin, we can “promote” the slave database (the clone) to fulfill the master’s role while we figure out what happened and how to fix it, keeping the site up and preventing any data from being lost.
2. Since the MySQL slave isn’t used for anything other than covering our backsides, we can take it offline periodically to archive all the data in the database without impacting anything. (This is important when your site has literally four million users pouring statistics into it all day, every day, and doing a live backup will slow things down.) Archives are good in case you want to do statistical analysis on your data or in case you accidentally delete everything in the database after working a twelve hour day trying to figure out how vermin ate your master database. (Both of these have happened before. Sort of.)

So, let me walk you through setting up a database slave from scratch.

I am starting with two identical machines that are running identical instances of MySQL. The first of these, Dragon, is going to be the master. The second of these, Unicorn, is going to be the slave. (So I name my servers after mythical beasts. What’s wrong with that?) I am assuming a reasonable level of knowledge with the UNIX shell. If you lack familiarity, you really should spend some time learning. It will make you a better programmer, system administrator, friend, spouse, and human being.

1. First, you should have the following items set in your MySQL configuration (/etc/mysql/my.cnf). If you don’t, add them: they can go anywhere under the [mysqld] heading.

server-id = 1
log-bin = /var/lib/mysql/binlog/mysql-bin
expire_logs_days = 1
max_binlog_size = 100M
relay_log = /var/lib/mysql/binlog/mysqld-relay-bin
relay_log_index = /var/lib/mysql/binlog/mysqld-relay-bin.index

The server-id should be some unique number, not shared with any other machine on your network. Replication occurs by copying around binary log files; you should set expire_logs_days so that you don’t end up with a huge pileup of data clogging up your disk drives (as they will eventually eat up your hard disk). We set binary logs to expire after 1 day, since our disks have a tendency to fill up on us quickly (for example, one of our Guitar Hero master databases has a 6GB binary log directory, even though it expires data after a single day!) Be advised that if you are worried about your slaves falling more than a day behind, you should keep logs around for a greater duration. (However, from our experience, if your slaves fall more than a day behind, you have bigger problems.) Finally, you can put datadir, relay_log, and relay_log_index wherever you please; above are just where we use. (Make sure, however, that they exist and are writable by the mysql user!)

You should restart the server after changing the settings. On Debian and Ubuntu, you can do this via “/etc/init.d/mysql restart”.

2. Your slave should have the same options defined as above in its MySQL configuration. The server-id option, however, must be different from the master (in our case, setting it to “2″ will do). All the other options can be whatever you please.

Also restart this server when it’s set as you like.

3. You should grant permission for the slave to read data from the master. This can be done from the MySQL console on the master (mysql -h dragon -u root). Assuming the user name “foo” and the password “bar”:

mysql> GRANT REPLICATION SLAVE ON *.* TO 'foo'@'unicorn' IDENTIFIED BY 'bar';

You can, of course, restrict which databases and tables you grant permissions with. We don’t in this particular case.

4. You now need to dump all of the data on the master, copy it over to the slave, and import it there; this is necessary even if your databases are empty, as it will initialize the slave regardless.

On the master (this will require your MySQL root password):

mysqldump -uroot -p --all-databases --single-transaction --master-data=1 | gzip >master.sql.gz

If you don’t have a root password (which is generally a bad idea, but fine if your MySQL server is on a private network), you should omit the `-p` above.

You can copy this file over to the slave server using `scp`:

scp master.sql.gz unicorn:

Finally, import the dump on the slave:

zcat master.sql.gz | mysql -uroot

5. Tell the slave to mirror the master.

mysql> CHANGE MASTER TO MASTER_HOST='dragon', MASTER_USER='foo', MASTER_PASSWORD='bar';

6. Start the slave!

mysql> START SLAVE;

You can now monitor the progress of your slave using `SHOW SLAVE STATUS`. (If you use a `\G` instead of a semicolon to terminate the line, it will print out the data in a much more readable format.)

mysql> SHOW SLAVE STATUS\G

There could be a number of errors, which are beyond the scope of fixing here. (In my case, we ran into issues with binary logs and the binary log index files.) These can generally be troubleshooted (troubleshot?) by reading the output of syslog (`tail -f /var/log/syslog`) and fixing the problem one piece at a time; it helps to have a lot of UNIX experience, though.) If all goes well, though, `SHOW SLAVE STATUS` will have lines something like the following:

       Slave_IO_State: Waiting for master to send event
          Master_Host: dragon
          Master_Port: 3306
     Slave_IO_Running: Yes
    Slave_SQL_Running: Yes
Seconds_Behind_Master: 12887

This indicates that the slave is running and in the process of catching up to the master, and in time will be synchronized with it.

And, that’s (hopefully!) all there is to it! Now, with your new slave, you can go forth and conquer the world! (Or, less ambitiously, cover your backside when you accidentally delete something.)

Obligatory CTO Note: Make sure you keep, and regularly verify your daily backups too!

The odds are against a business that is trying to build a branded online community (not unlike an online gaming community). According to a study performed by Deloitte and Beeline Labs, 35% of branded online communities have less than 100 members, and only 9% have more than 10,000 members. These businesses are investing heavily in these online communities, with 43% of the online communities operating with an annual budget of more than $50,000, and 6% with more than $1 million per year. These businesses realize that a successful online community has an incredibly high ROI, but the chances of failure are also high.

Agora Games, Inc. has successfully developed over a dozen successful online gaming communities, with total community members exceeding 1.2 million. Our most recent gaming communities have hundreds of thousands of community members. While complex software engineering is needed to efficiently transfer terabytes of in-game data to and from the web, we believe that half of the success of an online gaming community is due to the community design and management of the site.

Community design focuses on the creative process of “getting into gamers’ heads” to design web and in-game features that add value to the online community and video game. Before an engineer begins programming or a graphic designer starts drawing, our community designers develop a deep understanding of how the game will be played, and who will play it. The design process creates a framework for the online community experience that will help software engineers visualize how features should be implemented.

Community management includes forum and content moderation, hosting and promotion of tournaments, email communication, YouTube video production, and many other proactive actions. Community managers are on the front-lines interacting with community members. The community managers and volunteer community members are integral to building a successful online community because they are the game fanatics that jumpstart and guide the community through its lifecycle.

According to the Deloitte online community study, 34% of the communities only have part-time community managers, and only 16% have more than 6 full-time community managers. Directly after the launch of an online community, Agora dedicates multiple community managers exclusively to a community, which may scale-back over time. However, community managers are always needed for moderating the site, creating new tournaments, disseminating communications, and reporting community activity back to clients.

Developing a successful online gaming community requires so much more than just the engineering of the underlying software. Half of the effort should be expended on software development, and the other half should focus on the creative process and engaging early community adopters. Too many companies think “if we build it, they will come.” Gamers are especially sophisticated, and desire quirky sites that connect with their identity. The end goal is to have an active site with engaged community members that extends game life and enhances game play.

AgoraGames has been doing community sites for several years now, which means that some of them are pretty old and some of them have particularly unruly codebases, especially one such site. This site (which shall remain nameless to protect the guilty) ran perfectly fine at launch, but over the course of a year-and-a-half has slowly gone from speed-demon to slug-stuck-in-molasses.

(I apologize for the lack of accompanying images, etc. as I worked through this… next time, I’ll keep posterity in mind as I work through an interesting problem!)

I’m the System Administrator here, so it’s my job to make sure that everything stays running smoothly from the OS’s standpoint; thanks to our religious use of monit, the cause of the site’s slowness was pretty obvious: the mongrels running the site would balloon to consuming a gigabyte of memory within minutes, forcing monit to kill-and-restart the servers. This is a Bad Thing (TM).

Finding and fixing memory leaks in Rails isn’t the easiest task in the world (as other bloggers have been apt to mention), but in this case, the path to fixing it was pretty straightforward.

First task was to set up a separate bank of mongrels for the forums: since the site is so old, it’s the only part of the site that gets much traffic these days. I told NGINX to proxy all requests starting with /forums to those. It worked just fine, and the forums were immediately quite quick. From looking at monit, it was pretty apparent that the forum mongrels didn’t have the memory leak, since they ran without problems.

So, to start digging into the main portion of the site, I booted up a local server in production mode and curled the homepage. It took a whopping 222 seconds (and, from looking at top, consumed 800MB of memory!). Looking at the debug logs, virtually all of that time was spend doing two queries: loading forum posts with associations, and loading tournaments with associations. Sounds a bit sketchy, no?

The home page’s controller was laid out very logically, so mapping the SQL queries in the log to method invokations in the controller was simple. There were two methods that were killing us, and they both looked something like this:

class ForumPost < ActiveRecord::Base
  ...
  def self.most_recent
    find(:first, :order => "forum_posts.created_at desc",
                 :include => [:account, :forum_topic, :forum])
  end
  ...
end

Apparently, the find is loading up the (tens of thousands) of forum posts, with all the associated objects! Ruby doesn’t play nice when you have thousands of object in memory, and loading up hundreds of thousands of objects really makes things go south. Also makes sense why it didn’t become a problem for a while — it’s no big deal to load 100 object into memory (during testing and launch); but after a year and the site’s forum starts to fill…

But why? Isn’t this doing a :first? Shouldn’t it only be loading one object with associations? My guess is that Rails is loading all objects (with their associations) into an array and shifting off the first element. This isn’t smart behavior, but would explain what we’re seeing. (It’s also worth noting that this is using a fairly old version of Rails; I believe this silliness is fixed in more recent versions.)

Fixing this was actually trivial. Slap a :limit => 1 on there, or replace the :include with a :join. (Or, in the particular case above, remove the :include entirely. It’s not being used for anything relevant.)

I grepped the codebase for any other :includes, and fixed the ones that were broken. The site blazes along now and doesn’t leak, just like it did in its glory days.

Moral of the story: don’t use includes lightly. Make sure you know what your finds are actually doing. In our case, it translated into memory leaks, but it could just as easily translate into high database load, which is just as bad.

This past Thursday, we hosted Rock To A Cure in partnership with Bombers Burrito Bar. The verdict, a total success. The final count of money raised was over $1,400! It was night which will be remembered - or be a blurry vision for those who had one too many Agora Gamers - by all. While the dinner crowd chowed down some burritos all of us were busy hyping up the crowd, raising awareness for the Juvenile Diabetes Research Foundation (JDRF), and just being amiable hosts. Once the dinner crowd was finished with their burritos they were still hungry, hungry for Guitar Hero. Everyone started joining in to play Guitar Hero and Nicole was practically drowning in raffles. Luckily CZ was there to give a hand and rescue Nicole from the raffle abyss.

If things weren’t crazy enough already Mike DelPrete decided to “kick things up a notch” by suggesting some good ol’ fashion arm wrestling. The burly Dave Loya took on contestants from the floor and went undefeated until the last round where one of the members of the crowd brought him down hard.

The event went well past 10PM ending close to 11PM where everyone was eagerly awaiting the raffle of the XBox 360 which was taken home by a fine gentleman who was there the entire night and bought over $100 in raffle tickets! Definitely well deserved. With Rock To A Cure being such a success this is only the beginning to many more charity events to come.

CZ gets ready for the cake while Eric looms in the background.

Today, August 26th marks the birthday of David “My Last Name Is Too Long” CZ. Happy Birthday CZ, hope it’s a good one!

He is holding Frostmourne (srsly, there are people who own this sword and he is one of them).

Daily standup meetings just got interesting :)