01/30/2008    |    Devlog    |    Misha    |    Discuss

In some ways, having 24×7 GM support during our Beta prepared us for Customer Service on a live game and in some ways it did not. I thought I’d share with you the steps we took to get ready and the areas in which we need improvement. I’ve also got some tips for filing bugs that you might appreciate.

During the Beta, we established which tool we would use to receive issues from players (RightNow) and what data we wanted them to supply. We had GMs hired and trained on using the tool and on processing incidents. Basic processing involves trying to gather all the necessary info from a player and from our server tools to repro the problem or confirm the issue and then, if applicable, ensure a bug report is created in our internal bug database so the Devs will fix the problem. We established some procedures for general issues like what to tell someone who can’t run the game due to a problem on their end (insufficient hardware, bad network connection, etc.). We established some procedures for dealing with exploits and the players who use them.

We had staff to cover shifts 24×7 and even on holidays and we had established our internal procedures for what happens when someone is sick and can’t cover their shift. We also had established our procedures for communicating larger issues like server crashes how the GMs would reach the Ops team and when to call in Dev, Design, the Community team, or anyone else who normally works weekdays but whose expertise might be needed if an emergency arose in the middle of a Saturday night.

And we even had some emergencies arise in the middle of the night during the Beta so that we got some practice at the process. It’s one thing to think it all through and write it all down; quite another to actually go through it. For example, our process for deploying builds went through about 25 iterations. We added steps when we realized we might want to work deployments around port contention battles. We removed steps when we got tools in place to automate certain bits. We added steps when some deployments didn’t go well and we realized we needed to test the servers a little more before going live than we had been. And so on.

We knew things would be a bit different once we had paying customers. For one thing, we knew there would be more attempts to hack or exploit the game with fewer people reporting these things. So we have ways to look for such behaviors on our own. Obviously, these means aren’t infallible, but they’re a start. And pretty much every time someone finds an exploit, we think of a new way to detect them. For another, we’ve suddenly gone from confusion about one type of 20-character code (the key that let folks play in the Beta) to about 10 different kinds of 20-character codes: Pre-Boarding Party keys, Landing Party keys, retail keys, buddy keys, and so on. So I knew we’d hit some stumbling blocks early but figured we had the framework in place to adapt and adjust and deal. Well, we do but we’ve hit a higher volume of challenges than I expected.

We ran the Beta entirely in English. We had many international Beta Testers, but we expected them to report bugs and work with us on technical issues in English. We didn’t get our multi-lingual staff hired and trained as quickly as I’d hoped. I tried desperately to get them plugged in during Open Beta, which was late but not too late, but many of them didn’t start full shifts until after the Pre-Boarding Party started. We didn’t have 24×7 coverage in languages other than English until the week of launch. Getting them integrated into the rest of the team and establishing the procedures for determining who processes which issues turned out to be more complicated than I expected. And we still don’t have RightNow fully localized.

We also upgraded RightNow to a new version the day before the Pre-Boarding Party started. This new version is better, stronger, faster in many ways, but I was surprised by how different the UI is. I was grateful to get the conversion done before we went live as it entailed an entire day of down-time. But it meant that even the veteran GMs were starting launch day with basically a new tool to learn.

But the biggest surprise, to me at least, was the gold sellers. We knew they would be there, but we didn’t foresee how cumbersome our reporting and banning systems would be for this purpose. Because the gold sellers have up to six characters per server on eleven servers, we get tons of reports from players that are essentially duplicates of each other. We didn’t set up our reporting tools to easily recognize when fifteen reports on fifteen different characters are really just all one account behind the scenes. Don’t get me wrong we very much appreciate the reports. We’re serious about stopping these cheaters from destroying one of the best in-game economies in the industry. It’s just taking us longer to process them than we expected and the daily overhead in doing so means we are slower at responding to other issues. We are definitely working to improve these tools and systems, both to make it easier for our players to file the reports and for us to process them.

We’ve hired additional GMs to help cover this unexpected increase in the number of incidents we’re processing. They should be fully trained and on the schedule by the end of the week. If you graph our average response time, the curve is upward when it should be downward from where we are and then level when we’re achieving our goals. We expect the curve to turn starting next week.

Is there anything you can do to make this process work better for you as well as us? Yes, there are a few things.

1. When you log an incident, supply as much information as you can as clearly as possible. If the description of a problem is confusing or incomplete, we’ll just have to send it back to you asking for more info which further delays resolution. Sometimes log files are useful and sometimes they’re not but it doesn’t hurt anything to include them just in case. Think about whether a screenshot would help you clarify the problem. If so, feel free to include one or two or however many you think you need.

2. Once you’ve logged an incident, you can update it by going to the Support site and clicking on the My Stuff tab. If you have more info to add to an incident, do it that way instead of logging a new incident. If some of the info is in one place and some is in another, that’s very confusing to us and likely to result in a variety of people giving you conflicting information because none have the full picture. You can also mark an incident solved from this tab.

3. There’s an Incident Type drop-down. Choose the Incident Type that’s most applicable to your situation. We prioritize and batch processing based on Incident Type. If you’re stuck in a mission but your incident is filed as a General Bug, it will take us longer to find and help you.

While we started our efforts at customer service early, we’re not where we want to be yet. However, we are working to improve every day. We want your experience to be as enjoyable as possible. One way to do that is to deploy a fun and stable game on reliable servers. Another way is to be responsive and helpful when the inevitable issues arise. We’re working to improve on all fronts and we hope you’ll bear with us during our growing pains. And thanks for sailing the Burning Sea!

01/30/2008    |    Devlog    |    Misha    |    Discuss