Test is live!
The 13 sites below have their votes to close set at 3. I've also reduced the votes needed for migrations to 2 (down from 3).
Almost two years ago I ran a test for single vote closure/reopen on Hardware Recommendations after one of their moderators reached out to me about the strain of being the person doing most of the work to close questions on that site. Shortly after that, Shog9 ran a three-vote test on Stack Overflow, which has also been deemed successful. Later that year I ran a test to do the same on Web Applications, which we later made permanent.
So, we have three sites where five votes to close aren't needed and ever since those tests I've been getting requests from various moderators to lower the number of votes on their sites. I have 26 known outstanding requests to test three-vote closure on sites and later this week - Thursday, May 6th - we'll be starting 45-day tests on half of those sites.
What's the problem?
We ended up with five votes to close based on very specific feedback from the early days of closure and the very immature state of review, though it had previously been three. Five became the default network wide and, now that we have so many more sites with different amounts of participation, it's clear that it may not be the right number any more.
What I've come to realize more recently is that we've been relying on moderators - our "exception handlers" far more than we realized to actually close questions on many of our sites - and on some sites, when the moderators aren't doing the bulk of closures, large percentages of questions that end up in the close or reopen queues never get reviewed at all, aging out entirely after a while.
There's a lot of really great thinking about this on both the MSO announcement of their initial test and the results post on MSO that I linked to above, so if you want a deeper understanding, please check out the questions and answers on those two posts - though they largely relate to the aging out issue rather than the moderators doing the bulk of the closing/reopening.
When reviews don't happen
When questions get flagged for review and those reviews age out rather being completed, that's usually because there aren't enough active, engaged users with the ability to close/reopen questions (3,000 reputation on designed sites, 500 reputation on beta and non-designed sites). On tiny sites, there simply may not be that many people using the site and on bigger sites, it's not uncommon for reviewers to just get burned out on reviewing.
Note, this is a different problem than a site where there are lots of reviews being completed but a large number of flagged posts were deemed to not close or reopen. In that case, it makes sense to understand why there's a disconnect between flaggers/voters and reviewers but the reviews are actually happening.
As an example, say that in a 60 day period 1200 questions get a flag or vote to close by someone. If 29% are closed and 3% are left open - only 32% are being handled meaning that 68% are aging out of review - we don't know whether those 68% should have been closed or not, we just know that no one had the time to act on them.
When five people are needed for this process, it puts more weight on more people to do the work and, with a relatively low number of reviews per day per person, the small crew of reviewers can become quickly overwhelmed even if they are actively reviewing posts. Reducing this number to three quickly leaves this group more room to act on more different tasks.
Assuming that all votes come from review and not on the post, lowering votes to close/reopen from five to three means a 66% increase in number of posts that can be reviewed per day with the same group of reviewers - so 15 reviewers can handle 100 questions instead of 60 with no additional work per reviewer.
When moderators do most of the closing/reopening
Many of the moderators are very reticent to unilaterally close questions - in fact, I've spoken to many who only close vote stuff that's blatantly off topic or if there are 2-3 votes from the community already, so they're expediting the process rather than dictating the scope of the site. And this is great - unfortunately, on many sites, there aren't really five active reviewers to handle many of these cases, so if questions need to be closed, it inevitably falls to the mods to do the work.
On some larger sites, mods are handling hundreds of closures per month! In some of those cases, due to low reviewing, moderators likely are acting sooner than they might otherwise. This means there's less in the review queues for users to handle so they assume there's no action to take because the moderators are doing most of the work - but there's one major flaw in this - Moderators can close questions unilaterally which can make it harder to reopen questions that have been fixed because it still takes five people to reopen that closed question - or the moderators have to do even more work by keeping up with the reopen queue, too.
Shifting sites down to three-votes to close and reopen means that it's easier for the moderators to step back and let the community handle these tasks - while posts may take longer to get closed, moderators feel less pressure to do the closing and reopening themselves and it's possible that more edited questions can get reopened.
Risks & Challenges
There are some things that we watch for when we're running these tests. Here's a couple of them:
- If a site does have sufficient close voters and those people are active in reviewing, this can lead to higher instances of close/reopen "wars".
- Having fewer votes to close can make it less clear which close reason is valid on posts - when you need five, it's unlikely there will be five different reasons used. When you only need three, three different reasons is more likely.
We also don't see this as a perfect solution to all things. While this has helped SO, and there's been a long-term increase in percentage of flagged posts being handled, it didn't go to 100% - many questions still age out of review. On Web Applications and Hardware Recommendations much of the closing is still done by moderators, even on Hardware Recs, where they only need one user to vote to close.
There still has to be a community willing to do the work for this to help - the site won't need five people but they will still need three or else there will still be low percentage of review completion or moderators will still need to do a lot of the work.
The Test
Starting on Thursday, May 6th for the following 45 days, we are changing the number of votes needed to close or reopen questions on the following thirteen sites to see the impact on a variety of sites. At the end of that period, we'll reset back to five and look at the data.
These sites were selected based on meta posts that were marked status-review by their moderators, indicating that they wished to participate in this test. There were 25 sites that I was aware of as of the time of writing, so this is half of the total sites that have outstanding requests for this test. I was able to group those sites in two ways - by close volume size (over a 60 day period, small <100 posts nominated for closure, medium 100<500, large 500<2000, huge >2000) and which of the two categories the site fell into.
Low review completion:
Sites where a low percentage of the posts flagged for closure were being handled at all.
| Close volume size | Sites |
|---|---|
| Large | Stack Overflow en Español English Language & Usage |
| Medium | Server Fault Software Engineering |
| Small | Arduino Home Improvement |
Moderator-led closure:
Sites where a high percentage of the closing/reopening was handled by moderators.
| Close volume size | Sites |
|---|---|
| Huge | Stack Overflow em Português |
| Large | Cross Validated WordPress Development |
| Medium | Drupal Answers Artificial Intelligence |
| Small | Software Recommendations Anime & Manga |
I'll be posting answers to the meta posts above on each of these sites announcing the test along with a new question on each meta just for feedback about the experience about half way in but please feel free to use this post as a way to bring broader thoughts or concerns up.
There's definitely good candidate sites that aren't on this list and we'll be moving into the next phase of this project after we look at the data from these thirteen sites, so stay tuned. If your site thinks this might help because, have a discussion on your meta - this isn't a one-time event. Once we figure out the impact of this change on sites, we'll be better able to decide when we think we can just change the setting without needing to watch things as closely.
I know that some sites want to test this because they'd like to see questions closed more quickly, before they get answered - I understand this instinct and I'm not saying that we won't consider testing whether this improves the situation on those sites in the future but, right now, speed to closure isn't something we're focused on and there may be more effective solutions to preventing answers to close-worthy questions than closing the question before someone writes an answer.
Analysis
My plans to analyze the results are specific to the reason we're testing this.
For sites with low completion numbers, we want to see:
- increase in completed close/reopen percentages
- no increase in moderator close/reopen voting
- little or minimal increase in close/reopen warring
For sites with high moderator percentages, we want to see:
- increase in close/reopen percentages handled by community members
- -or- increase in mod votes counting as only one vote (3rd vote)
- little or minimal increase in close/reopen warring
While it'd be nice to see more reviews or more questions closed or reopened or more people participating in review, those sorts of things are secondary effects. I would hope that people feeling that their votes are effective would actually cause them to want to participate more - but I'm going to avoid hoping for that. The effect that we're really hoping for is that the same number of people will still keep doing the same number of reviews which will lead to more closures or reopenings because there's only three people needed to get there.
Thoughts?
I'm sure that lots of you have thoughts and questions for me about this - let me know and I'll work on getting answers. Are there concerns you have? Do you think this will be effective? Are we missing something important? Is there some additional analysis that you think would be beneficial?
