154

As part of our Q2 roadmap, the Community Product team is working towards improving Review Queues. This project is still in the early stages of discovery. In addition to user research interviews, we are sharing our progress with you today to ask for your feedback before we begin implementation. This is the first post in a series of new communication we hope to have with the community that uses the new tag. After we collect community feedback, we will be open to including changes into the next design iteration and continue to share updates with you.

Objective & Goals

Community moderation is an important part of Stack Exchange and we want to provide the best user experience to those volunteering their time across our sites. In parallel to the Close Experience work, we hope these improvements will better support our curator and moderator communities.

We spoke one-on-one with many active reviewers across the Stack Exchange network, conducted data analysis on current Review Queue usage, and read through just about all of the existing issues raised by our community. We used this qualitative and quantitative data to inform our goals and initial design decisions.

In this redesign, we are exploring ways to improve the overall usability of Review Queues. This includes, but is not limited to:

Providing guidance and revisiting the banning experience
The current on-page instructions are not visible and could be misinterpreted, leading new reviewers to make poor decisions. Inappropriate actions on review tasks or audits can lead to a suspension of privileges with no obvious means of understanding what happened.

Adding more context to each queue
Users are opening browser tabs or using user scripts to view the original post they’re reviewing, other answers, and comments to help make educated decisions.

Improving filtering and discovery tools
The filter feature often goes unnoticed and unused. As it exists now, certain combinations of filters can leave queues empty. It is a poor method to discover or target new tasks.

Improving queue workflows and pathways
On Stack Overflow, the Requires Editing action is sending a surplus of poor, unsalvageable posts to bloat the Help and Improvement queue. H&I reviewers are then limited to either Skip or mark the task as Very Low Quality, having it return to the Triage queue and creating a frustrating endless loop.

Along with the Triage queue, First Posts and Late Answers are not fulfilling the goals that they were made to accomplish. These queues were intended to give good posts a head start toward success and filter out bad posts, but tasks could see better, faster intervention in other queues.

What’s next

We’re collecting feedback here on Meta and from 1:1 user interviews. If you have opted-in for user research, you may receive an email invite this week. We’ll be asking questions about how this design may improve (or worsen) review workflows and what you’d like to see in a curated review experience. You can opt-in to future user research opportunities by enabling Research emails in your profile settings.

This project is big. To focus your feedback, we have broken this announcement into three, digestible posts:

We’ll be using community feedback to continue to iterate on this work. We look forward to hearing your thoughts and concerns about this project. Please let us know if there are other Review Queue issues you’d like to see addressed as well.

Improve this question
18
  • 29
    "On Stack Overflow, the Requires Editing action is sending a surplus of poor, unsalvageable posts to bloat the Help and Improvement queue. H&I reviewers are then limited to either Skip or mark the task as Very Low Quality, having it return to the Triage queue and creating a frustrating endless loop." - Heyyy, I remember saying something along those lines! :D
    – Nick is tired
    Commented Apr 23, 2020 at 18:13
  • 24
    On Stack Overflow, situation with unsalvageable posts in H&I seems to be indeed worse than you describe. Reviewers who mark the task as Very Low Quality have system cast a VLQ flag on their behalf and it typically ends at moderators who decline it. Few declines of such flags trigger flag suspension. Essentially the only safe option for garbage in H&I review is Skip, can you imagine what kind of snaked toilet this creates.
    – gnat
    Commented Apr 23, 2020 at 21:47
  • 39
    A) I'm so glad this project is happening. Review has been a sore spot for years. B) The iterative approach with gather community feedback along the way is a real winner! I'm gonna model it in my current job. Thank you!
    – Jon Ericson
    Commented Apr 23, 2020 at 22:18
  • 20
    One fundamental thing to consider: if a queue is never at any point in time empty, then the system design is broken. Queues are supposed to handle peaks in input, so that they can be processed later. But if there is more items entering a queue then leaving, then no amount of queueing will save the system. This is the core reason why the review system occasionally fails completely. The goal must be to reduce manual review by having question wizards, smart scripts, maintenance bots etc.
    – Lundin
    Commented Apr 24, 2020 at 7:52
  • 23
    Optionally: don't alienate all the people who did these reviews. Moderators and veteran users. SO has consciously and consistently been alienating such users for many years. So eventually you will end up with employees having to do the reviews (or become Quora 2), meaning that you should seek to automate the process so that you don't have to hire as many paid moderators.
    – Lundin
    Commented Apr 24, 2020 at 7:55
  • 24
    Honestly, I'm really surprised anyone contributes to reviewing at all. It seems an utterly thankless task with occasional horrible pitfalls where you get silently banned, a constant suspicion that you're "cheating" somehow, and no upside whatsoever. So glad to hear it's being worked on.
    – Steve Bennett
    Commented Apr 24, 2020 at 13:29
  • 5
    As someone who stopped reviewing (on Stack Overflow) very quickly, I am so happy to see attention paid to the queues!
    – Super Jade
    Commented Apr 25, 2020 at 19:24
  • 8
    “We spoke one-on-one with many active reviewers across the Stack Exchange network, conducted data analysis on current Review Queue usage, and read through just about all of the existing issues raised by our community.” — This is great to hear, but have you used the queues yourselves to get a better feeling of where the pain points are?
    – Sebastian Simon
    Commented Apr 26, 2020 at 10:17
  • 4
    @SteveBennett Reviewing is a thankless job indeed, but also a very, very important one if you want to maintain content-quality. I review a lot more on Code Review than on Stack Overflow, but that's why I do it. Site quality goes down if I and others stop.
    – Mast
    Commented Apr 26, 2020 at 11:00
  • 20
    Banning Experience - Yes please. I'm not an idiot but I'm a new reviewer - and I've already been banned twice, and I was actually trying to be useful. I've basically rarely touched the queue since my second ban, because I don't want to get banned a third time. Very demotivating. Having some guidance or better support here would be great.
    – qaisjp
    Commented Apr 26, 2020 at 20:27
  • 6
    It is probably worth talking to people who have done review queue work in the past but not for some time; people who are active reviewers are evidently not finding it painful enough to cause them to abandon it. I suspect if you look at active users who are eligible to run queues, only a small percentage actually do, and figuring out the reasons the rest of them don't, and fixing those, should be an important part of any overhaul on the review system.
    – Xiong Chiamiov
    Commented May 4, 2020 at 16:39
  • 2
    How about getting a reward for actually spending time reviewing, closing and fixing things. It's a never ending job, it never stops. All the new people posting the same "I want this implemented, can someone help?" that we need to close is tiresome, and we don't really get a reward for cleaning up the site. Perhaps get 0.1 points for every successfully closed (voted) question etc? That would be more valuable than "tidying up the queues" which is probably needed, but in all honesty, feeling some kickback for actually cleaning up the site would be ever better. //SO user&"moderator"
    – Torxed
    Commented May 7, 2020 at 8:27
  • 2
    Banning Experience - when you ban a user, how about also stopping the showing of the red "high volume" dot. Also, in place of the unhelpful 'There are no review queues available to you' how about a link to the review area link so that we can see what the issue was?
    – Andrew Mortimer
    Commented May 7, 2020 at 11:02
  • 2
    It would be helpful if we could earn the privilege of not getting "tested" after some number of reviews. It is another thing that adds to the "thankless" feeling.
    – Katie
    Commented May 7, 2020 at 16:14
  • 1
    @Katie I would be careful about this. I did tens thousands reviews in different queues on differrent sites and I fail review audits very infrequently, maybe once in few months but I still would prefer getting these because audits help me control whether I am sufficiently attentive
    – gnat
    Commented May 23, 2020 at 22:29

3 Answers 3

Reset to default
152

I'll leave specific notes on your design overview posts, but wanna drop my overall impressions here: as someone who has spent a considerable amount of time and energy on these systems since their inception, I believe you've correctly identified the areas that most need focus (whether that focus is needed due to serious problems in their current behavior, or due to missed opportunities).

You've done your homework, and it shows. 👍

Improve this answer
14
+500

mobile review screens
unusable for so long
under way, in progress

Happy to report that UX refresh work for review task screens (including responsive design) is underway.

Improve this answer
7

Please let us know if there are other Review Queue issues you’d like to see addressed as well

Yes, mobile usage.

Currently, it's nearly impossible to review using a mobile device.

With all due respect to help center and explaining to people how review works, it's more important, in my humble opinion, to let the people who already know how to review to do it from a mobile device.

The responsive design project appears to be stuck, didn't see any update for months.

I'm sure there is a pending feature request for this, search currently not working so I'll add a link later.

Improve this answer
11
  • 5
    The UX refresh for the review task screens is planned to be fully responsive, will be tested on mobile. This phase of the project is getting started right now.
    – Yaakov Ellis
    Commented Nov 3, 2020 at 8:19
  • Wish I could add a bounty to your comment here @YaakovEllis. One long awaited upgrade I'm really thankful for!
    – Luuklag
    Commented Nov 3, 2020 at 8:25
  • @Luuklag here ya go
    – Yaakov Ellis
    Commented Nov 3, 2020 at 8:35
  • 1
    @Luuklag only 100?! ;)
    – Shadow Wizard
    Commented Nov 3, 2020 at 10:31
  • 1
    Feel free to add more tomorrow @Shadow10YearsWizard ;)
    – Luuklag
    Commented Nov 3, 2020 at 10:31
  • Your turn @Shadow10YearsWizard ;)
    – Luuklag
    Commented Nov 4, 2020 at 8:44
  • @Luuklag done! Too bad newlines aren't preserved in bounty reason, lol. (tried to make it funny)
    – Shadow Wizard
    Commented Nov 4, 2020 at 8:59
  • @Shadow10YearsWizard paragraphs are supported I think?
    – Luuklag
    Commented Nov 4, 2020 at 9:11
  • @Luuklag didn't work for me, and can't really test it...
    – Shadow Wizard
    Commented Nov 4, 2020 at 13:16
  • 1
    @Shadow10YearsWizard meta.stackexchange.com/questions/119684/…
    – Luuklag
    Commented Nov 4, 2020 at 13:18
  • @Luuklag ohhh... neat!
    – Shadow Wizard
    Commented Nov 4, 2020 at 14:04

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .