In September at the Austin Game Conference I attended a lecture on MMO metric collection using the Emergent Metrics Element. This was a similar talk to the one that Larry Mellon gave at Game Developer’s Conference 2004 on metric collection in The Sims Online. After that talk we implemented something similar to the system he described. It was able to capture sample data points on as many probes as we wanted and logged a few things about the actors involved whenever those things happened. After the Austin conference I sat down with the designers here to find out what kind of data they want to be able to collect from the beta (and soon from the live game) and found something interesting: none of the queries they wanted could be performed with our existing metric system. In almost every case they wanted more context than was actually available in the data.

The big trouble with the queries they wanted to perform was the sheer number of different pieces of information they wanted to know surrounding each event. They asked for (in various cases) where the player was, what level they were, what nation they were, what kind of ship they were using, what ammo they had equipped, what outfitting they had installed, and what skills they had active. All of these are perfectly reasonable things to want to know, but keeping track of that data as a snapshot every time someone fires their guns is unrealistic. The metric system (as outlined in the lecture and implemented by us) depends on the ability to roll up samples to an extreme degree when their parameters match. The more parameters you add to the metric samples, the less likely they are to ever match, and the less you are able to summarize data. Summarizing data is something you have to do to keep the amount of data you are tracking from ballooning out of control.

Just to give you a taste of what I’m talking about here: In the past seven days players in the beta have fired a total of 155,138 volleys. That is for one cluster that is not even 5% loaded. If each cluster records that number of volleys and there are 10 fully loaded clusters, we would need to record just over 31 million records per week. That isn’t so bad if we are just tracking the number of times each size of gun with each kind of ammo because we can summarize that data into a few hundred records per day. But if we need to track the character stats above, we can’t make those summaries. Most of these records are going to have at least something that can’t be rolled up. If recording all of the data above requires 1,000 bytes of data (which is a low estimate) we would have to find someplace to stick 31 gigabytes of data every week.

So if we can’t track all the character state we want with every event, what can we do? Well my solution was to track the character state CHANGES and to keep them small enough that they can be kept forever. If we associate each event with a character ID and can look backwards in the records to find the most recent level up event prior to the event we’re interested in, we can find out what level that character was when the event took place. This is obviously far more difficult to query, but the data is there, so at least it’s possible.

So I set the metric system aside and started working on the Formal Logger, or FLogger for short. The flogger is a system that allows Flying Lab programmers to define any number of event types and the parameters for each one. They add the flog definition to the flog type XML file like this:

<flsFlogType><br />    <name>mission_stage_use_door</name><br />    <fieldNames><br />        <igStringRefList><br />            <e>stage</e><br />            <e>level</e><br />        </igStringRefList><br />    </fieldNames><br /></flsFlogType>

This flog type can then be used in the code wherever the appropriate event occurs:

flsFlogger::flogf("mission_stage_use_door", iCharacterId, "%I64d, %d", pMission->getTemplate()->getId(), getLevel());

Once I had the flogger itself working, we brainstormed a list of 50 or so flog types and then went through the code adding them. The first set covered quite a number of different parts of the game, including login, money, inventory, the economy, combat, skills, advancement, and missions. We have since added more flogs for just about everything else. Isildur spent a few hours putting together a viewer for the data in PHP and this is the result

One thing that was immediately apparent was how useful this could be to a customer support rep. trying to figure out how to help a player. CSRs are able to call up the entire history of a character just by clicking on them in-game. This lets them see where the player was before they started having trouble, and what they did that led up to the trouble itself. We expect our CSRs to use the feature on many of our more tricky customer issues.

The other early benefit we gained from the flogs was rapid detection of exploits. The first time we fired it up on actual data from beta we found a character who levelled 5 times in 5 minutes (from 35 to 40, no less.) Looking at that player’s activity put a spotlight on the culprit: we had a mission that required you to walk across town and rewarded you with about 1/5th of a level worth of XP! And it was repeatable! And the XP was set to scale with your level so it kept working even after you started gaining levels! We closed the loophole with a hotfix and have been keeping an eye on the levelling rate ever since.

Of course the primary reason we have flogs (as opposed to metrics) is to track how well balanced the game is, primarily in the economy and combat systems. I’ll leave discussion of how they’re using the data up to the designers themselves, but I did track down some interesting data while writing this DevLog (DLog?).

This is a graph of the top 10 skills used in the game:

This shows that a large number of player are using the toggle skills “Accuracy” and “Damage”. In fact, they’re using them more than they are boarding skips, which is something that EVERYONE has to do from time to time. That probably means they are a little TOO effective, and need to be retuned. Three of the top ten skills (Speed Increase, Crew Focus: Sailing, and Travel) directly affect the top speed of your ship.

This could mean that speed is very important in combat (which it is) or maybe that there are too many speed skills. The big surprise on this list, to me anyway, is the Slice skill. That’s one of the avatar combat skills that is only available to beta testers in one avatar combat test environment. This probably means that a large number of beta testers are trying out avatar combat, even though they don’t currently get anything out of it (no XP, no treasure, etc.) A little more digging would tell us more of why these skills are used so much, and help the designers to tune them.

The economy is also an interesting thing to graph. This is a graph of the top 20 recipes used in the beta in the past week:

Oak is used to build ship hulls, so it isn’t a surprise that a lot of it is being harvested. In fact, all of the top 20 recipes are either raw materials or intermediate components that are used in ship-building. This actually meets Isildur’s goal of having ship construction be the main focus of the economy. I was curious about what people were actually crafting, so I dug a little more deeply in this one. This is the set of recipes used by the top 5 crafters (in terms of total number of recipes used.) I’ve reduced the names to initials to make it a little tougher to guess who these folks are.

All 5 of these people are building ship components. None of them appear to have shipyards (or at least didn’t use them this week). They aren’t building anything that can be used directly, but are instead building the components that go into ships built by other people. For kicks I checked the flogs to see who the top sellers were this week. Here’s what I found:

CharacterTotal
TS340,511
SD338,850
RH291,002
MK283,954
VK232,851
CS194,270
EG191,215
EB175,300
ST124,600
MC120,000
Grand Total2,292,553

Only MK appears in the top 5 in both lists (which won’t surprise you at all if you’re in the beta and know who that is.) EG managed to stay in the top 10, but the others are all new. That seems to indicate that making ship components isn’t necessarily the most lucrative trade, even though it is the most popular.

This data gives us visibility into how the the game is running for all players. We don’t have to rely on forum posts or bug reports to find out what’s going on when we can see it for ourselves. We have already fixed bugs and closed exploits with the data and it has been helpful for tuning ship combat and the economy. The flogger will only get more useful as our viewing tools improve. When combined with surveys and out of game feedback, flogs will be tremendously helpful in keeping the game challenging and fun for everyone.