You may have been wondering what happened after the 10k invite? Are we going to invite any more?

We’ve been just about too busy to breathe. And we have invited more folks and plan to continue to do so. Within a few days of inviting the last 1000 of the 10k chunk, we invited a few thousand more because things were going so well. Since then we’ve put inviting more on hold for a bit while we beef up our infrastructure.

We’ve now got 24/7 GM support, which means we now have more staff for addressing the issues players log. Also, now if a Beta Tester gets trapped in a mission and can’t get out, they can log a support incident and expect to be helped within an hour or so, probably much sooner. We’ve also got Operations and Development staff on-call 24/7 in case the GMs hit a snag they can’t fix. Poor Gray Noten got a call about 2 a.m. almost every day or a week there. Most companies don’t provide that level of service until after launch but we’ve found that the easier it is for our testers to help us, the more likely they are to do so. :)

We’ve also improved our tools for finding and fixing bugs. When a server crashes, it now automatically emails us its crash dump. With the crash dump, Devs often don’t even need repro steps in order to fix the problem. We’ve also made the server logs easier for us to access than they used to be. And we’ve been flogging data for quite some time (logging special bits of info in a database) but now we have better searching and aggregation tools on that database. One big result is that it’s easier for us to identify exploits, enabling us to address them much more quickly than before. Each time we added a new feature to our tools, though, we were suddenly able to see many more issues to address. We’re just about to dig out from under the big list and are now able to start addressing things as they crop up.

We ran a stress-test on a Friday night our time. We handed out free in-game cash and XP and other goodies to enable even noobs to exercise portions of the game they wouldn’t be able to otherwise. We had many hundreds of players logging in at the same time. This was the first time we’d seen our queue system in action and it worked pretty well. Sadly (or happily depending on how you want to look at it), we found a ton of graphical performance issues and some server perf issues as well. Some folks crashed regularly as their video card crumbled under the load of too many avatars in town. Others would send chat messages that didn’t appear for minutes. Server-side, we didn’t have as many crashes as we’d feared and the hardware withstood the load reasonably well. All-in-all we learned quite a lot and are grateful to the testers who showed up and kept on showing up even when the game was problematic for them. We plan to run more of these in the near future.

After that, we wiped the Beta database and released a new milestone build containing revamped missions, a bunch of skill and economy tuning, and performance improvements for our servers and our animations. Once we got that build out and mostly working, we switched over to using Sony’s installer and LaunchPad and to using Sony Station Authentication to log into the game.

That didn’t go so well. We had a bug that would kick you from the game every 40 minutes, no matter what you were doing. It took us quite awhile to track this down but it turned out we weren’t communicating properly to SOE that the player was still logged in and playing. Since we’ve fixed that, it’s been much smoother sailing. Folks are giving the economy a pounding, testing the improvements there. They’re playing the new swashbuckling missions and sending terrific feedback about those. And, of course, they’re continuing to help us nail down issues in all other areas of the game and giving us great ideas for polish and tuning.

Until we get our forums converted to using Station Authentication for logging in and some tweaks made to my Beta Invite tool, we have to hold off on inviting new testers. However, while we’ve been waiting, we’ve been stabilizing the build and the process of using Station Authentication so that the experience for the new folks when we do invite them will be easier than for earlier testers. We expect to convert our forums this week and hope to start inviting new testers again the week after. Wish us luck!