06/04/2008    |    Devlog    |    Gray Noten    |    Discuss

Aether asked me to write up a description of how we communicate operational issues and outages, both planned and unplanned. For those of you who haven’t read my initial Devlog, I’m Gray Noten, and I run the Operations department for Flying Lab Software. My team is responsible for taking builds from the development team, deploy them to Testbed and Live, and handle any technical issues that come up during day to day operations.

For planned outages we try to provide at least a one day notice, though whenever possible we will provide more notice. This includes planned hardware or central infrastructure upgrades as well as game patches. Generally speaking, these kinds of outages will be announced on the front page of the website, the patcher MOTD, and with a forum thread in the Important Announcements forum.

We try to schedule planned outages around the lowest player concurrency for the services being upgraded. For example, we use a 1:30am – 3:30am window for our US/EU server outages because that affects the minimum number of players. Our aggregate concurrency numbers for the US/EU servers are at the lowest point during that window. Similarly, we choose our AU outage window in order to minimize the number of our Australian players that have their play time interrupted.

Part of our planned outage process is to review the port battle schedule and make sure that any port battles scheduled to start within two hours of the beginning or end of the outage window get pushed out by 24 hours. This ensures that even if we go over with our outage window, we don’t have port battles that get messed up.

Unplanned outages are a bit more complicated. These can take many different forms. Generally speaking, we categorize them into three main categories, which are handled differently. That said, any longer term downtime will always have a patcher MOTD post and a forum thread for updates.

If an outage is localized to a single server such as a hung chat server or a buggy nav zone, we will generally limit notification to players actively playing on that server. We provide at least a five minute notice via in-game system announcements. The announcement will include what part of the server is being restarted, what the impact will be, and the expected time of the restart. For these kinds of localized outages we will generally not post a forum thread or update the patcher MOTD.

If an outage requires a server to be restarted, we will lock the server, announce the restart in game, and post a forum thread in the Important Announcements forum. Advance notification for these restarts may vary, but we try to provide at least 10 minutes of notice before restarting. We lock the server in this instance because we would like to prevent people from logging in after any in-game announcements, starting a game session, and then having the server restart right in the middle of something.

For outages that affect more than one than one server, we will always post a forum thread and change the patcher MOTD. These kinds of outages tend to be more complicated, but we will generally follow the same server restart procedures if the servers need to be restarted. Until the outage is completed, we will keep the forum thread updated.

As with planned outages, we try to do what we can to maintain port battles intact. Due to the nature of unplanned outages that is not always possible, but whenever we can we will avoid disrupting port battles. During unplanned outages we are not always able to tell whether there is a port battle coming soon, so you should feel free to post on the forum thread for the outage to make sure we know about it!

One final thing is important to keep in mind. When we have the outage announcement threads, we’re primarily interested in feedback related to that outage. In particular, if we post that something is fixed but it appears to still be broken, please post in that thread. We’ll put updates and notifications related to the outage in that thread so it’s easy for everyone to get the current status of a given outage. What that thread is not for is general feedback on a patch, game balance issues, etc. There are other threads for that, and honestly, the folks monitoring that thread can’t do anything about the things you’re complaining about! There should be a build feedback thread in the General forum for things like this, so feel free to post there or start your own thread in the General forum where the right people can find it.

We are constantly working to improve our communication during downtimes. Particularly with unplanned outages, downtime is a very stressful time for both the players and our staff. It can be hard to update the player base when we’re neck deep in a problem, but we try our best to update everyone when there’s news. As avid MMO players ourselves, we understand how frustrating it can be to stare at a login screen or a server selection screen waiting to be able to play. We strive to have the best possible communication with our player base, and if there’s anything we can improve upon, please let us know in the forums!

06/04/2008    |    Devlog    |    Gray Noten    |    Discuss