Workflow changes for review queues

I would like to reiterate something I brought up in Improving Review Queues - Design overview I: Onboarding and updating workflows regarding the workflow of the Reopen Votes queue.

The main problem with it currently is that any¹ edit on a closed question sends it to the reopen queue. This means that "meaningless"² grammatical/formatting corrections which obviously doesn't make the question reopenable - will send it to the queue. The biggest pitfall with that fact is: the question only gets one pass to the queue so if a meaningful edit comes a long, the question is less likely to get reopened. The collateral damage is simply wasting reviewers time.

Some possible changes (in no particular order) are:

• Grammatical/formatting edits do make the question better - we still want to encourage them - but not reopenable. Most chances are that only the OP can really edit the question to bring it to a reopenable state - only OP edits should push the question to the queue. This will actually align better with the new Triage's "Needs OP edit". It currently flags the question for closure expecting the OP to edit, only to be sent back to the reopen queue by any random edit.

• Add a (very) visible warning (similar to the new onboarding modals) when editing a closed question that this edit will put it in the reopen queue, so the edit should be really meaningful. Bonus: add the same warning to the review of edits of closed questions.

• Add a meaningful edit or edit and reopen checkbox to the edit screen that editors can mark if they believe the edit changes the state of the question and it should get to the queue. By default, of course - it will not.

• Make suggested edits on closed questions require 3 (or more) approved reviews instead of 2.

You mentioned that the Reopen Votes queue is the second to last busiest, but I still believe that making its items the most relevant possible will make it more focused and will encourage more people to use it and get some questions reopened! Personally, I got tired of just leaving closed due to meaningless edits and barely use the queue...

Related discussions:

• A proposal to reduce the number of closed questions needing reopen review
• Add a warning message when editing closed questions
• Suggested Edits queue should indicate (better) if the question is closed
• Now that the Edit Question button for closed questions is more prominent, it's time to stop non-OP edits from submitting to the reopen queue

_{1. Not really any edit, but most. See Which edits push closed questions to the reopen review queue?}

^{2. Meaningless in the sense of making a question reopenable. I'm not saying that grammatical/formatting edits are meaningless in general.}

One big problem I have with First Posts queue is that it has both questions and answers. This often leads to confusion when users see a question posted as an answer. They edit the post instead of flagging it as NAA. Separating the queue is definitely a good idea, but the question is whether we need a specialized queue like this. Could we not have "Low-quality answers" and "Low-quality questions" queue instead?

Triage is a Stack Overflow exclusive queue and was created to quickly assess and provide feedback on questions that had been flagged low quality. This queue is often the root cause of other review queue issues. It lacks strong guidelines for reviewers to follow and assumes that they can identify good questions, both leading to inconsistent reviews.

By deprecating Triage, we’d like to adopt how the Low quality posts queues on other SE sites function and route all posts (questions and answers) that have either been flagged as low quality or fall below the system-generated quality score threshold to Low quality posts.

I'm of the opinion that the low quality posts queue isn't supposed to be used for questions, period. To get them closed or have the community make the edits necessary to keep it open, you use the close-vote queue. The close-vote queue already asks community members to edit if that means a question can stay open, there's no need to ask the same question in a different queue.

You mention it in this post too: Triage was used to send questions where people picked "Needs author edit" to the close votes queue. Not 'the low quality queues', where people would be making the same decision (leave open, edit, close). Deprecating Triage and just sending more questions to the low quality posts queue is just moving a problem around.

If you're going to put questions in a low quality queue, make the queue about whether or not a closed question should be manually deleted, not about whether or not the question should be closed, that queue already exist.

Of course, such an implementation of a low quality queue (delete or not delete) comes with a few risks:

• Closed questions that are deleted before the OP has had a chance to make those edits only an OP can make.
• Every closed question ending up being flagged into that queue, even the ones that don't need much intervention because they will be cleaned up by the Roomba process.

Questions right now can't be deleted by the community unless they are closed first, and then they need to be closed for >48 hours, or closed and having a score of -3 or less. to be eligible for 'fast deletion'. So any question that can be flagged into this queue should probably meet the 48 hour criteria at least, so the author had a chance to make any edits.

The point about questions that would be cleaned up by Roomba is a tricky one. Just saying 'wait the entire 9 days before anything can be flagged into this queue' is probably not the best solution as it takes away some of the power of the fast-deletion mentioned in the previous paragraph.

Then again, you also don't want people flagging everything that has been closed for 48 hours into that queue. The queue would be most helpful if only posts that won't otherwise get deleted or really need fast deletion end up in it, for example closed, low quality questions that can't go through the Roomba process because they have a FGITW low quality accepted answer, or blatantly off-topic posts.

So, in my opinion, if implemented correctly, a low quality queue for questions:

• Doesn't duplicate the workflow of the close-vote queue.
• Would still give a post's author the time to make necessary edits, before their post ends up deleted.
• Should offer a bit more visibility to people into what should be candidates for deletion than the 10k tools.
• Would encourage only doing useful review work, and not just be a manual replacement of the Roomba process.

This would mean a lot more work than just dropping questions in the current low quality posts queue, but it would improve workflows a lot.

Create separate First posts - questions and First posts - answers queues

I think it's a good idea to emphasize different actions for different post types, but are separate queues really necessary? Right now, the Low Quality Posts queue has different actions depending on the post type as well. (I'm ignoring the fact that the use of the Low Quality Posts queue for questions is not universally accepted.)

If you still go forward with separate queues, what will happen with past reviews? Will they be split as well? What would happen to a Steward badge awarded to a user reviewing 600 first answers and 600 first questions?

I think of the 2 arguments one isn't precise and the other isn't accurate.

• Deprecate Triage

  This queue is often the root cause of other review queue issues. It lacks strong guidelines for reviewers to follow and assumes that they can identify good questions, both leading to inconsistent reviews.

In other words, the root cause might be that users with 500 rep get access to Triage, if there's any correlation between reputation and experience (a few like to argue there's no correlation) it's obvious a 500 rep user is unlikely to have enough experience with site moderation to be effective at flagging and duplicate finding.

There weren't tag filters just a couple of months ago, users had to click through every technology under the sun and make an unreasonable number of evaluations regardless if they had expertise or experience.

I don't think the elapsed time since introduction of the tag filters allows for any significant data about changes in the quality of Triage reviews. If such data exists it hasn't been presented. As such a major change is being proposed based on the former business rules that have themselves been deprecated. (I can only read this as a "lesson learned" that inexperienced users aren't on average accurate enough in reviewing.)

Deprecate Help & improvement

H&I hasn’t been successful in its intended purpose with issues from Triage diminishing the queue’s usefulness.

There are less than 90 stewards from H&I and I'm one of them. Most of the arguments I heard against H&I during that time did not come from someone with proven experience in that queue.

(...) and look forward to learning from your insights.

All the H&I stewards I've read commenting about the queue liked the experience and thought it added value. The art of copy-editing seems to be disdained. Besides, I think H&I is a great copy-editor school because you risk getting review suspended for not editing right.

Talking from experience, I see a ton of posts daily (perhaps more than not) that would benefit from going through H&I. The only problem is: the posts aren't being sent to the queue.

Deprecate Help & improvement

H&I is our least productive queue.

Yes but it's also the most effective. You don't see any pending review items in that queue because users compete to get their cut of the action (folks want the extra custodian/reviewer badge to complete their collection).

Over the last 90 days, only 7 tasks

Which amounts to 630 edits that might not have been done otherwise. (I fail to see how this doesn't improve "the product" - or "quality" if you prefer.)

Create separate First posts - questions and First posts - answers queues

I read this last point as the TL;DR. Raise the reputation bar for questions (because that requires more skill) and turn the "Answers FP" pretty much into what Triage queue is now.

H&I could be integrated effectively into the existing review queue business logic by automatically dispatching items to it that fulfill a number of criteria (a lot of the bots on the network already check for this):

• Fluff in post, like "Thank you", "Please help ASAP", etc... Automatic dispatch to H&I if it passes initial Triage with "Looks OK".

• 1 single tag on post, send to H&I for double check that tagging is accurate.

• 2 tagged keywords in title?! -> H&I...

• Long stretch of what appears to be code without a source fence, H&I...

• Several "lemma" programmatic keywords (like the syntax highlight is using) without inline literals?! -> H&I...

• Post that is 1 long quote and nothing else (poster inadvertently pressed quote button) -> H&I...

• Image links that aren't rendering inline? -> H&I...

(And have the system generate a digest of the criteria for potential H&I reviewers to get the hints.)

If 2 or 3 reviewers in H&I say: "No edit needed" then it probably doesn't need it. (One of the reviewers is sure to find something that can be improved in the post, there always is some detail...)

When thinking about this, and the Triage queue especially, I always recall an answer from our beloved Shog9.

Shog presented the following numbers:

So what good is Triage?

Ok, now we can get to some numbers. From my perspective, it's still a better First Posts review than First Posts Review, and a better way of handling VLQ flags on questions than Low Quality review. But that doesn't directly help folks just trying to browse the site without bumping into lousy questions.

So let's focus on views. In May of 2018, 256,414 questions were posted on Stack Overflow. Of those, 31586 entered Triage, and 25508 were reviewed completely (that is a consensus was reached). The average view count across all questions posted in May was 89; the average view count for questions that entered Triage was 64; the average view count for questions that completed Triage was 67.

Broken down by Triage consensus:

Triage result	Questions	Avg Views
Looks Good	11379	76
Should Be Improved	9825	66
Unsalvageable	4299	48

And drew the following conclusion:

....

However, Triage is not unsuccessful. Beyond its utility as a tool for moderation, it does at least partially fulfill the original goal: ensuring that fewer people have to look at terrible questions.

As we can see from the table Unsalvageable posts get about 2/3^th of the views Looks Good posts do. So in all fairness, Triage does do a job at getting rid of garbage.

Now why does it do that job, because questions don't face any delays into getting into triage. They are added as soon as they are posted, and meet the criteria.

The flowchart of the review queue's shows how interrelated the Triage and Low Quality Posts(LQP) are. They essentially serve the same purpose. So combining them could make sense, when the implementation details are good enough.

What details should we get right then if we want to merge Triage into LQP?

• Set an appropriate reputation threshold
  Currently the Triage queue requires one to have 500 reputation. It's a fair level I think, as it helps you ease into reviewing with very little you can seriously break. LQP only becomes available at 2K reputation, which requires way longer to achieve. So simply moving the Triage posts into the LQP would greatly reduce the number of available reviewers, making the queue's even more backlogged.

Are there any other options?

• Yes, just keep triage as it currently is.
  The most major problem we had with it, the infinite loop between Triage and LQP has been cut off. Also I think that it is really good at flagging posts, also something that we need to learn new users to do effectively. We could use the review system to provide feedback on declined flags that came from the Triage queue, and use that to educate users on when it is appropriate to use what flag.

Disclaimer: I set out to write a coherent post, but I kinda feel I failed in the end...

• As previously mentioned, the Late answers queue has been successful in identifying and removing low quality posts, repeat answers, and spam.
• We’re adding definitive actions to address those exact problems: Looks OK, Edit, and Delete.

Spam shouldn't just be deleted, it should be flagged as spam. The delete action would be The Wrong™ action since it won't feed spam countermeasures and it wouldn't put moderator or community attention into it, which would allow spam to fall through the cracks. Instead, it should be looks ok, edit and flag.

In addition to creating all these new queues, please also provide a place were one can access posts from all queues at one place.

Why? If one filters the review queues for a narrow set of tags one is active in, it is hard to notice when there are items to review. The red status dot is no help, because it does not take into account one's filters. So the only possibility is to visit the queue from time to time and with good luck, it will have something to review. For the close review that works okish, but for all other queues the probability that there will be something to review is so low that it is easy to give up on them altogether.

I think this would help to bring the users who are actually most familiar with a technology together with review items interesting for them.

H&I is our least productive queue. Over the last 90 days, only 7 tasks went into the queue each day on average, compared with 2,458 for First Posts (our busiest queue) and 219 for Reopen Votes (our second-to-last busiest).

I would be very cautious about this one. In fact, I would instead investigate how many "looks ok" where later edited, which would be candidates for the H&I queue. Remember that folks usually see things in the extremes, when many things are actually in the middle. Triage right now is a "flag/no flag" queue, which of course would reduce significantly the volume of questions that are sent to H&I.

By deprecating Triage, we’d like to adopt how the Low quality posts queues on other SE sites function and route all posts (questions and answers) that have either been flagged as low quality or fall below the system-generated quality score threshold to Low quality posts.

I would recommend that you review previous documents about why was this created in the first place. The Plan™ was actually the other way: remove first post and keep triage and H&I, because first post was a very suboptimal queue: post marked as "no action needed" when they definitively did need some action, which makes the entire queue a waste of human resources, since it doesn't add anything substantial towards the goal of achieving quality. Same thing as triage, look at how many First Post were edited/closed after the review task was completed and which kind of action was it completed with.

"Triage is a Stack Overflow exclusive queue and was created to quickly assess and provide feedback on questions that had been flagged low quality. ..."

On sites without a Triage queue we sometimes get questions that aren't a clear single question, or two related.

When there's too many questions in one we can use "Needs more focus", and when an answer doesn't answer the question we have an 'answer flag' called "Not an answer" - but currently we don't have a "Not a question" flag, leading to the use of "Needs details or clarity"; which is fairly unsatisfactory if it's clear that the question won't be fixed by editing.

In the spirit of the 'answer flag' "Not an answer" we need a question flag called:

◯ not an question
This was posted as a question, but it does not attempt to ask an answerable question. It should possibly be edited, but most likely deleted altogether.

The wording was kept as close to the other flag's description, with changes to accomodate the difference; I'm not opposed to different wording. It should be clear that like NAA this NAQ should either be headed to editing or deletion.

Other questions involving this issue:

• Should unclear questions be flagged or downvoted?

• Show answers on review page for questions flagged as "unclear what you're asking"

• What are the exact criteria for a question to have VLQ as a flag option?

Note: In the Charcoal HQ chat there was a hope that one-day we would have a NAQ (or this noisier search term) flag someday, for questions that aren't spam or rude but also are not a question; and are most unlikely to be saved by an edit, leaving deletion as the most likely outcome.

That could be resolved by a queue for deletion of the question, where it's survival is determined similarly to the NAA reviews. Once it enters the queue the "Doesn't seek input or discussion" result is appropriate if it doesn't appear editable.