Hello, folks!

It’s been just shy of a week since we turned on the new user content system in a “live, but testing” mode, and we’re now ready to call the system just plain “live”.

Why “but testing”?—because we knew there would be quite a bit of tuning to ensure that items were moving through the system in exactly the way we wanted to, we started out with somewhat conservative values. We weren’t exactly sure how people would respond to the voting process, after all—especially in terms of volume. Would people vote just enough times to get a posting credit, then post their item; or would they vote like crazed madmen? The thresholds at which items get moved through the system are quite different depending on where actual usage patterns fell.

On the other (“but live”) hand, we wanted to test the system with real data. Items that were rejected for not meeting the various voting thresholds should be true rejections. Items that made it through the voting process to the review process should be ready for review. Above all else, we needed to ensure that the process was moving items through it as quickly as we intended them to.

I’m happy to report that as of this writing, the approval queue is—for the first time since the user content system came into being a year ago in closed beta—empty. Almost 1,300 items have been reviewed in the past week, and that’s in a week where many of our usual reviewers are at the ION Game Conference.

Just one of the seed items have not yet been put into review—and it is very nearly at the threshold where it’ll be moved into review or be sent back for more work by later today, meeting our goal of having nothing remain in the voting queue more than about a week. More importantly for many of you, the best items are winning enough matches that they’re getting into review within 1-2 days of being posted, and being approved that same day or the following. (I watched with glee today as one particularly great sail was posted, reviewed, and approved in under 24 hours).

Likewise, the items which need the most work are failing to meet the first threshold and are being sent back within a day or so of being posted, letting people know quickly that they need to rework their submission to get it into the game.

Finally, copyright violations are typically being removed from the system the day they’re posted through the reporting mechanism. We have some ideas for improving this process even more, but for now, we haven’t had to reject any items for copyright violation in the staff review process—they’ve all been discovered in the peer review portion of the system.

A number of people have commented on the forums that they wish they could become even more involved in the process—for example, by having the community be able to reject items for the various reasons the staff reviewers do. We do understand that the community wants to be more involved in the approval process, but we have to balance that input with building a system that lets our expert staff have the final say in a significant majority of items reviewed.

While it may seem clear that giving the community tools to remove items from voting for the same reasons as the staff reviewers do would give the community more input, we would never be able to make such a system allow the community to cause “hard rejections”—that is, for the community to remove items from the voting queue entirely.

We chose two of the most flagrant violations of submission—copyright theft and offensive images—to empower the community to “soft reject” items in most flagrant violation of our rules, which we then review and either reject for those specific problems or put back into the queue.

If we provided a similar mechanism for “soft rejection” of what the community considers “less good” items, it would cause a number of problems:

First and foremost, it would cause us to have to manually review more items which are ultimately rejected than we do in the current system, reverting the system back to the 100% staff review process we had previously. If, then, we have to individually review every item that passes through the system, we may as well simply remove the community review entirely, since neither the community nor the reviewers would be gaining any benefit from it. We don’t want that.

Second, and perhaps worse for the overall health of the system: there would no longer be items that still need work for the good items to get voted against. The system would devolve to good item being voted against good item—again, at that point, we would need to simply remove the voting process entirely, and return to FLS staff simply reviewing 100% of the items.

Finally, we consistently have always approved roughly one third of our submissions. At the same time, only about a fifth of the submissions are what our reviewers would consider exemplary—the remaining chunk of items that are approved are of average quality. Empowering the users to reject based on personal subjectivity would cause those average items to never get approved; in fact, it’s conceivable that not even the whole of that top fifth would get approved in a purely community-driven system. This would be significantly harmful to user content in general.

The reality is that the new system puts significantly more power into the hands of the community than the old did, though it does so in less direct methods. It allows peer review to quickly remove the bottom fifth of submissions with auto-rejection based on minimum threshold requirements, and at the same time, allowing that same review to push the top fifth into review (and generally, therefore, approval) quickly.

We’re very pleased with the system so far, but that doesn’t mean that there isn’t room for improvement—based on feedback we’ve gathered in the Flags and Sails forum, we’ve already made a number of small changes, and will continue to solicit feedback and ideas and improve the system in the coming weeks and months, so keep the feedback coming!