What happened with the experiment to lower the vote-to-close threshold?

Question

In December 2019, the moderator team posted the question Should we test lowering the vote to close and reopen threshold?, which was positively received, and there was a clear agreement and intention to go forward with the experiment.

What is the status of this experiment? Are still waiting for the full data-taking protocol to be finalized? Is everything ready to go and we just need to push the button? Was it swallowed up by The Monster (a.k.a. the year 2020)?

I don't mean to berate anyone -- just a quick prod to see if we can get this back on track =).

Answer 1

Ah Dec. 2019... those were simpler times eh?

So what happened was by Jan. 2020 we were on the list and working to get a more complete dataset defined to extract, and the Community Manager (CM) who was going to write the scripts to do that analysis was let go from the company. So that kind of set things back on getting any sort of custom data defined.

Then the rest of 2020 was happening and more changes to the CM team and so on, and so the project was put on hold network-wide for awhile. While it was on hold, a large number of sites said they wanted to lower the threshold, so now there's a backlog of sites that would like to do it.

My understanding is that the lowering of vote thresholds on sites will resume sometime soon, possibly even this month. But rather than a FIFO queue, it's going to be prioritized on getting sites that are really struggling under 5-vote-closures done first and then moving on to other sites that aren't having a hard time but would still like to do it.

My guess is that we're going to be somewhere low on the urgency list because we do have a really good group of people who are voting to close things and most stuff does get to 5 votes, although it often takes longer than we like it to. It also probably means we won't have a lot of customized analysis done. It just isn't in the cards.

While I'm not sure where we are in the priority queue, I am keeping tabs on it because we'd still like to see how it goes!

Answer 2

Y'all may have seen the update I wrote as it was linked in the comments but I wanted to give y'all a specific update because the use case y'all have for this is very different from our goals of the test.

Our focus is on doing two things:

1. Increasing the percentage of review tasks that are completed.
2. Reducing the percentage of review tasks that are completed by moderators who cast first or second votes to close.

Let's look at your stats on those two things:

In a 90 day period starting 30 days ago and reaching back 120 days, you had 3092 questions get flagged or a first vote to close. Of those, 70% were closed, 8% were left open and the remainder 22% were incomplete. In the grand scheme of things, those are pretty decent stats. It's not 100% but some sites are doing far worse.

Over the same period, moderators closed 887 questions and the community closed 2396. Now... this adds up to 3283, which isn't 3092, which is likely because the queries aren't perfectly aligned in how they define voting or flagging to close but if we look at the numbers, that means mods are closing (at some point in the process) about 27% of the questions. I haven't looked at the breakdown on which vote the mods are closing - but, while a somewhat high number, this also isn't a burdensome percentage, compared to some sites.

So, I'll need a bit more info before I can judge whether y'all fall into the groups where we're looking for improvements. You do have a large volume of posts that are going through review, which is an important point - y'all have the seventh largest number of posts during this time period. Many of the sites with worse percentages aren't very big and don't have a ton of reviews to do. But, of the top ten sites, you have the second highest completion rate, with only SO in Portuguese doing better (84%) and Math a close third at 76%.

That said, based on a query I have, I'm not sure that this test will do what y'all are hoping. This query looks at all of the questions closed in a period of time and lists how many were closed with each number of votes and then how many of those questions were answered before being closed. It then calculates, of those that were answered, how many votes to close were there when closure occurred.

For the most recent 90 days (not the same period as above), here's what I get:

# of Close Votes when closed	Closed Qs	Answered	Average close votes at first answer
1	498	114	0
2	286	97	1
3	158	63	1.314
4	113	53	1.814
5	1544	869	1.678

The really the important rows here - assuming this is correct - when you have five vote closures on questions, the average number of votes to close that exist before the question is answered is 1.67 or 1.8 with four vote closures. That means that many questions - more than half - are answered before the question gets three close votes.

I'd have to look at a distribution to see where all of those answers hit but I'm not a SQL master, so I'll have to find someone to write the query for me. The data above is from a query that Shog wrote and Nicolas refined since Shog's version was somehow incorrect.

The takeaway here is... we can test this but I think that to get this answering before closure thing in check, it's going to require more than just changing the number of votes to close - y'all will have to really start thinking about why these questions are getting answers and understanding where the disconnect is -

• Do you have people who disagree about whether questions should be closed? If so, changing this could drastically increase close-reopen wars since it becomes far easier to close and reopen posts. Maybe better understanding and consensus could help with this.
• Do you have people who aren't informed about when questions shouldn't be answered because they're out of scope? If so, how can you help inform them to try and prevent answers?
• Something else? Y'all know your community far better than I - why do you think you get so many quick answers to close-worthy questions? What do you think you can do to change this?

Let me know what you think.

Answer 3

Update, May 2021

As recently announced on Meta Stack Exchange at Testing three-vote close and reopen on 13 network sites, SE has just launched into this experiment, on sites where either (a) questions are ageing out of the review queue before they can be closed, or (b) moderators are carrying out most of the closures.

Regarding us (and potentially other sites in similar situations?), the announcement has this to say:

I know that some sites want to test this because they'd like to see questions closed more quickly, before they get answered - I understand this instinct and I'm not saying that we won't consider testing whether this improves the situation on those sites in the future but, right now, speed to closure isn't something we're focused on and there may be more effective solutions to preventing answers to close-worthy questions than closing the question before someone writes an answer.

I'm not sure what that last bit really means, but I've asked, so let's see what the team says.