229

This post details the first in a series of releases related to the ongoing work on Review Queues. It focuses on changes to the ways in which users who have been suspended from reviewing are notified of this event. These changes are now live across the network.

“Review Ban” renamed to “Review Suspension” change

With this release, we are changing the terminology that is used to label a user whose privileges to review posts has been revoked. Up until now, this was called a “Review Ban”. It is now called a “Review Suspension”. Through this change we are seeking to soften up the language and emphasize the temporary nature of the suspension of privileges.

This change will be reflected in all of the notices described below, as well as in the admin interface that is available to moderators for applying review suspensions.

Review suspension notices

The current review suspension experience falls short when it comes to making sure that the user is aware of their suspension. While we currently remove the list of review queues from the review dropdown, no notification about the review suspension is visible there. There is a message on the reviews page while the review suspension is active, but it lacks visibility, and disappears as soon as the suspension is concluded. Furthermore, nothing points the user to visit the review page during the review suspension, so users often miss a message that gives them information on the cause of the suspension and resources for improvement.

This situation unintentionally makes it hard for the user to learn about what they did wrong and can lead to the user making the same mistakes once their review privileges are restored. It can also lead to moderators giving overly long review suspension terms, simply because that is the only way to try to ensure that the user will even see the notice during the review suspension.

Review Suspension notices within the /review section

The notice text that is included in review suspension notices is being improved in this release (in the below screenshot, the default text is the first line; there is also a more helpful explanation text given by default in the event that none was entered by the mod who applied the review suspension). We'll be creating new guidance and suspension notices as part of a future project.

Screenshot of the new review suspension notice, which includes the start and end times of the review suspension, as well as the suspension message

While the review suspension is in effect, the message will be visible on the main /review page as well as each individual review queue page and the system will track if the suspended user has seen the full review suspension message. During the review suspension, the “Got it” button won’t be visible.

If a user doesn’t see the suspension message while the suspension is in effect, the message will persist on all review pages until the user has clicked the “Got it” button confirming that they have seen the message.

Top bar review dropdown

During a review suspension, the top bar Review Queues dropdown will now show a notice informing the user of their review suspension. This is in place of the current messaging that just says “There are no review queues available to you”, without making reference to the review suspension or even giving a link to a page with the suspension message.

In the case where the review suspended user has not seen the full message on one of the /review pages during the suspension period, the top bar review suspension notice will also persist after the suspension has expired until the user has clicked the “Got it” button as seen in the prior section. But, since they have access to review, the individual queue listings will appear in the dropdown again.

Screenshot of the topbar review dropdown messaging during and after a review suspension is in effect

Other Changes, Metrics & Feedback

Some changes are being applied to the review queues listing found on /review:

  • Each review queue will have a direct link to its Stats and History pages from the listing (currently you can only access this from the queue itself after loading a task).
  • Users with review suspensions will be able to load the full list (though the queues themselves will be greyed out and will not be clickable), in order to give them an avenue to access Stats and History.
  • All permissions notices on the page are restyled using Stacks.
  • We have published a new help center article titled Why was I suspended from review queues?. This will be linked from the new default review suspension notices.

The overall goal of this functionality is to make the act of suspending a user from review be more effective in helping the user to learn and improve their reviewing skills. To that effect, among the metrics that we will be tracking to measure the success of this functionality are:

  • Number of review suspensions applied over time
  • Average length of review suspensions
  • Amount of time from beginning of suspension until when the user sees the notice
  • Percent of users who see the review suspension notice during suspension versus after suspension
  • Repeat suspension rate for users
  • Rate of reviewers who return to reviewing activities post-suspension

We are happy to read and respond to feedback and bug reports posted below relating to the areas covered by this release, for one month following the publication of this post (after that point, new issues should be posted as new questions).

Improve this question
22
  • 56
    This looks great Yaakov! It's a significant step forward. Thank you and thank everyone that's worked on it.
    – Makyen
    Commented Jul 21, 2020 at 19:10
  • 9
    I love it! Keep up the good work!
    – Dharman
    Commented Jul 21, 2020 at 22:48
  • 11
    "post-suspension, notice unseen" This is the sort of detail I love to see in software. You're obviously thinking through the use cases. Good stuff :)
    – Asteroids With Wings
    Commented Jul 22, 2020 at 8:51
  • 7
    As an aside, I still think that notifying users when a suggested edit is rejected (or maybe a review decision overruled, but that may be too noisy) may help us train some newer users out of the gate so that we need fewer suspensions in the first place.
    – Asteroids With Wings
    Commented Jul 22, 2020 at 8:53
  • 5
    @AsteroidsWithWings Maybe. The place to make the case for that is on a different post. But any type of rejection for a new user has to be very carefully done - when you are new, a single bad comment or downvote can chase a user away (and have the opposite of the desired effect of helping to encourage them and train them). I think that having a place to go though the see where review decisions were overruled is promising.
    – Yaakov Ellis
    Commented Jul 22, 2020 at 8:56
  • 2
    @AsteroidsWithWings we do have a bunch of functionality coming up having to do with onboarding for new users to reviewing, and continuing education for those who are in it. Making Stats/History more visible as is done here is just the start. Can you please drop a link to a relevant existing post so that I can have a look?
    – Yaakov Ellis
    Commented Jul 22, 2020 at 9:03
  • 6
    Picking a nit: "Got it" seems colloquial and localized. Wouldn't the standard "OK" be better for SE's international audience?
    – LShaver
    Commented Jul 22, 2020 at 14:28
  • 5
    @LShaver I think the idea is that "OK" is subconsciously ignored as a standard UI element that just means "move me to the next step". "Got it" more clarifies that the user is affirming their understanding of the message. Maybe "Understood" would be a better term though.
    – Dave Cousineau
    Commented Jul 23, 2020 at 1:00
  • Great work, But in what time period was the screenshot of the available queues taken?!? I can't remember opening that without at least 2 red and 5 silver dots next to them.☺
    – Remy
    Commented Jul 23, 2020 at 11:08
  • 1
    @Remy it was taken on my computer; we don't use a copy of the full SO database, so the dots in the review dropdown behave differently. Not really relevant here.
    – Yaakov Ellis
    Commented Jul 23, 2020 at 11:14
  • @LShaver The specific phrase "Got it" is a standardized Material Design element, so it's likely to have an understood meaning.
    – chrylis -cautiouslyoptimistic-
    Commented Jul 23, 2020 at 16:06
  • @YaakovEllis: My question (actually, suggestion to change SE functionality) meta.stackexchange.com/questions/346104/… was closed as duplicate of yours. My question is ~4 month older than yours. How can it be that a question in the past is a duplicate of a question in the future? It should be other way round. Could you please explain this weird logic?
    – mentallurg
    Commented Jul 24, 2020 at 22:54
  • 2
    @mentallurg I agree, it is not a duplicate. I reopened, and added a status tag and an answer
    – Yaakov Ellis
    Commented Jul 25, 2020 at 17:55
  • 4
    @mentallurg on a general note, comparative age of questions in cases like that is not much important, see eg Should I vote to close a duplicate question, even though it's much newer, and has more up to date answers? and multiple discussions linked to it
    – gnat
    Commented Jul 26, 2020 at 9:54
  • 2
    @mentallurg as explained by Shog9 here the issue isn't age. It's which is better.
    – candied_orange
    Commented Jul 31, 2020 at 10:57

8 Answers 8

Reset to default
106

Provide a clear way to appeal an automatic review suspension

This suggestion goes off of nvoigt's answer: many times, users can end up receiving mistaken review suspensions because they failed badly-generated audits (generally bad-quality posts that were upvoted, but also cases where the reviewer was paying more attention than was called for).

The lack of a clear way to dispute a failed review audit can result in many disadvantages:

  • It can detract users from reviewing, as it makes the system seem "unfair" that they apparently have no way to contest the incorrect test that they were given, or the suspension that clearly wasn't their fault.

  • It can result in users reviewing incorrectly: they were just told to positively review an obviously bad post, which can make them positively review some things that they shouldn't, or worse, everything - the opposite of what audits are designed to induce!

In the case of automatic suspensions imposed for failing audits, please add a link on the review suspension notice, to dispute the review suspension on the per-site meta. Example:

I believe that this review audit is incorrect

The link can prefill the meta asking form with the link to the audit, and a clear place for the user to state why they disagree with it. Example:

Disputed [queue name] audit [audit ID]

Link to audit task

[link here]

Rationale for disagreeing with audit

[Please state why you disagree with this audit here]

Update:

Catija commented about the new help center article that tells users to ask questions on meta about review suspensions. However, I feel that this isn't adequate, for the below reasons:

  • It doesn't mention anything about review audits, or how they're automatically generated by the system and that the system isn't always perfect at generating them.
  • The overall wording of the section can imply that the review suspension is correct, and that the only recourse is to ask why it was correct. (While it does mention "if you feel the suspension is in error", words like "[if] you have any questions about the review tasks that led to the suspension" and "[a] request to understand what the correct choice should have been" send a mixed signal.
  • The notice, as shown in the screenshot in the question, seems to title the link as "what review queues are and how they work", which isn't clear at all that they're supposed to click that link to find out how to appeal a suspension.

I still think that providing a clear link to dispute a suspension caused by failing audits, as well as providing an automatic question prefill for them, is the best thing to do. If that is not to be done, then the very least that can be done is to modify the help center article and notice so that these three bullets are resolved.

Improve this answer
12
  • 11
    Sounds good. If the queue of disputed audits needs to be kept low, I could see only allowing the automated picks to be disputed and have hand-picked, human verified audits that cannot be disputed this way. So after a while, when every audit has been disputed and either tossed or verified, the queue of disputes is reduced to nearly zero.
    – nvoigt
    Commented Jul 22, 2020 at 10:51
  • 1
    Like other mod actions that a user questions, wouldn't per-site meta be the appropriate place for this?
    – LShaver
    Commented Jul 22, 2020 at 14:27
  • @LShaver "the meta site" implies per-site meta. If I meant this site, I would have said it explicitly.
    – Sonic the Anonymous Hedgehog
    Commented Jul 22, 2020 at 19:17
  • Sorry, I wasn't clear -- my real question is, isn't per-site meta already serving this function? Those with review privileges would likely already be familiar with per-site metas and know that those are the place to go to question/contest mod actions (automatic or otherwise). Having a button/form doesn't seem to add a lot of value.
    – LShaver
    Commented Jul 22, 2020 at 19:26
  • 9
    @LShaver On a lot of sites, especially Stack Overflow, a lot of 500+ rep users are unaware of the meta site, and even among those who are, they may not know the fact that they can go there to appeal, or if they do appeal, don't supply enough information such as the link to the failed audit.
    – Sonic the Anonymous Hedgehog
    Commented Jul 22, 2020 at 19:28
  • That makes sense... perhaps that unawareness should be fixed first? and it seems like having those links visible would be useful as well.
    – LShaver
    Commented Jul 22, 2020 at 20:25
  • 3
    The new help center article encourages users who believe they were incorrectly suspended to ask on meta so users will be aware of how to get this attended to.
    – Catija
    Commented Jul 23, 2020 at 1:52
  • 2
    @Catija Addressed that in an edit. I still think there should be a direct link to file an appeal, against automatic suspensions for failing review audits in particular.
    – Sonic the Anonymous Hedgehog
    Commented Jul 23, 2020 at 19:56
  • 3
    I disagree. Meta is better than adding more work for the mods to do right now.
    – Catija
    Commented Jul 23, 2020 at 20:18
  • 3
    @Catija Can the help center article and notice at least be modified so that the bullets I mentioned in the update are resolved? (And also mention the tag disputed-review-audits which has been used in the past for reporting bad audits.)
    – Sonic the Anonymous Hedgehog
    Commented Jul 23, 2020 at 20:45
  • 2
    The help center article is network-wide and there are no review audits almost everywhere on the network,
    – Catija
    Commented Jul 23, 2020 at 21:34
  • 3
    I don't like the Mea culpa tone some of the post have. Where he user repeats "I won't do it again. I did wrong. I'm sorry.". I don't know it could be view as Meta lowering people head and doing some public shaming. Could be a cultural thing or a miss reading from my part. But I have some weird feeling about some of those
    – Drag and Drop
    Commented Jul 31, 2020 at 11:15
55

Those changes look great.

The overall goal of this functionality is to make the act of suspending a user from review be more effective in helping the user to learn and improve their reviewing skills.

Please also consider going over your randomly computer-selected review tests and have them scanned for bad apples by a real person. It is extremely annoying and insulting to be told you are suspended from doing free volunteer work (that doesn't even give internet points) because you failed on a test that was impossible to succeed on because the information you needed to succeed on it was not made available to you. It is even more insulting to be told to improve and learn. Improve in what? Clairvoyance? Fortune telling? Being luckier in random chance games?

Please make sure that when you show this very nice feature to people, you show it for the right reasons. Because people failed a review test that was a good test of their review abilities. Not because the computer picked the wrong random number and served them a "test" they could not possibly succeed in. No amount of good design and programming will be able to undo that mistake.

Improve this answer
4
  • 4
    I've posted a closely related suggestion.
    – Sonic the Anonymous Hedgehog
    Commented Jul 22, 2020 at 10:43
  • 8
    It is extremely annoying, when you do volunteer work and give some of your free time, to be sent a curve ball by some Big Brother AI, that makes you waste time, because of course the audit questions are usually not that clear cut. Whether you get a nod or a slap, you feel that you have been had. I quit reviewing before I got a review suspension and I guess I'm not alone...
    – xenoid
    Commented Jul 24, 2020 at 0:21
  • 17
    This is exactly why I quit reviewing. IMO, the original post here is merely lipstick on a pig. The audit questions do not accurately reflect the SO standards. I will not spend more volunteer time to post a solid presentation, merely to fix a single instance of an endemic problem. I simply walked away.
    – Prune
    Commented Jul 25, 2020 at 0:00
  • 1
    Just let moderators nominate posts they delete in response to flags to be audits. Then, let moderators remove automatically-selected suggestions from the candidate list. Problem solved. It won't be that difficult to curate, as different users can all get the same audits. We don't need that big of a list.
    – Cody Gray - on strike
    Commented Aug 2, 2020 at 3:27
41

I think this is a great step forward, but I'm wondering if a simpler solution (which can be implemented in addition to these changes) is to send a notification to the user's global inbox when they are suspended? This is probably not necessary if they're automatically suspended after failing too many audits but it does work for manual review suspensions. Regular moderator messages (including account suspensions) already cause notifications, why not review suspensions?

Improve this answer
1
  • 60
    We considered this, but want to work incrementally here. Inbox messages have a way of not being seen, especially if you have a few, and the new message in the review topbar will be sticky. So while we can easily add a global inbox message, we first want to see what effect the current changes will have, and if it will get enough visibility according to the current plans.
    – Yaakov Ellis
    Commented Jul 21, 2020 at 19:29
29

Don't notify users of items to review in the top bar when a user is suspended from review

Currently, the system still notifies review-suspended users in the top bar of items for them to review. However, this notification dot serves no purpose to such users, as they can't take action on said dot.

While I do see a counter-argument that this notifies users to open the review bar and see that they have a suspension, I find this a bit of a backstab. It's like saying, "Hey, there are some things you should review...oh, wait, sorry, you can't review these, you're suspended from review".

The dot itself can be used to notify them of the suspension, but please don't show the dot once they've seen the message.

Improve this answer
8
  • 35
    The way it will work is: the red dot will show up until you open the top bar review menu for the first time. Once you open it up that first time, it will no longer appear for the duration of the review suspension. We wanted it to appear at least once, as a way of drawing attention to the dropdown and the suspension notices, but along the lines of your thinking, we don't want to have it be in their face after that.
    – Yaakov Ellis
    Commented Jul 22, 2020 at 10:55
  • 7
    @YaakovEllis That sounds great. I just don't want the dot to show to notify them of review tasks; using it to notify of the suspension itself is much better, though.
    – Sonic the Anonymous Hedgehog
    Commented Jul 22, 2020 at 10:57
  • 3
    So, at least for the time being, it won't notify them of the suspension unless the indicator would have turned on because there are sufficient review items to make it active... So on a site where the indicator rarely lights up, a user may not be immediately aware of the suspension unless they click on the icon without the indicator lit. But, the indicator won't turn on every hour when the user is suspended after the message is viewed.
    – Catija
    Commented Jul 23, 2020 at 1:55
  • 2
    @YaakovEllis Would it be possible to change the color (or shape, etc.) of the dot during the suspension, as a further (albeit perhaps subtle) indicator that the user's situation is out of the ordinary?
    – hBy2Py
    Commented Jul 23, 2020 at 16:50
  • 5
    @hBy2Py it is something that we are considering, depending on how things go with the response to the current changes
    – Yaakov Ellis
    Commented Jul 23, 2020 at 19:36
  • 1
    Changing the shape would be a nice touch. Instead of a dot, change it to a red slash/circle (or an x if there's not enough rez for the slash/circle) until their suspension is over or they've seen the notice at least once. That would tell the user that there's a reason to open the drop down without the implication that there are things that he can review.
    – RobH
    Commented Jul 24, 2020 at 18:52
  • This your answer contradicts to your answer to the same proposal in another thread: meta.stackexchange.com/a/346106/324858. Can you explain that?
    – mentallurg
    Commented Jul 24, 2020 at 22:43
  • @mentallurg I've responded to you on that post. The thing being requested to remove here is the red dot, not the entire review button itself.
    – Sonic the Anonymous Hedgehog
    Commented Jul 24, 2020 at 23:03
13

First, an addendum to Sonic's answer: I find it more important that there is not only a "one click" solution to appeal. It is much more important to be able to get to exactly those previous reviews failures that contributed to the current suspension.

It happened to me more than once that I ignored a failed review test knowing the test question was one of those wrong picks by the system. Weeks later you fail "again" and get suspended... good luck finding the one, two times in the past when you know the system punished you making the correct decision.

But the real problem are those machine "generated" test questions. Instead of solely relying on those, two things could be done:

  • have humans identify test questions regularly. That would dramatically reduce the amount of punishing users for your heuristics being wrong. If that is too much work, then
  • have humans sit down and identify test questions once, for an initial review training.

Meaning: collect 100 or so test questions, together with good explanations what the correct review result should be. Then have people go through 10 for learning. And only allow users to become reviewers after correctly handling another 15 of such test questions.

Improve this answer
2
  • Much of the stuff you mention was already mentioned in nvoigt's answer. I carefully wrote my answer to avoid overlap with theirs.
    – Sonic the Anonymous Hedgehog
    Commented Jul 23, 2020 at 19:47
  • 1
    @SonictheMaskedWerehog The only overlap I see is that first bullet point. Also note that I suggest a more specific implementation/solution.
    – GhostCat
    Commented Jul 24, 2020 at 10:14
11

The moderator "Account Info" drop down on user profiles still uses the old URL to link to the review suspensions page. That is, it uses:

https://so.com/link/to/user/history/page?type=User+has+been+banned+from+review

instead of the new

https://so.com/link/to/user/history/page?type=User+has+been+suspended+from+reviewing
Improve this answer
1
  • 6
    Thanks. Actually in PR right now, will update when fixed
    – Yaakov Ellis
    Commented Jul 25, 2020 at 20:52
3

These changes seem very good IMHO.

Also the renaming from "Review Ban" to "Review Suspension". It's subtle but gives a more serious and reputable impression.

But I have a few questions regarding review suspensions.

  • Are you planning to change the criteria when exactly a review suspension is instated?

  • When are you actually get suspended? At which amount of failing tests?

  • What is when a user got review suspensions for multiple times in order?
  • Is this user after a specific amount of review suspensions completely banned or at least suspended for the review queue for a longer time than the period of the usual suspension?

And do you planning to change anything here?

It would make sense in my eyes to suspend a user from review queues after a specific amount of review suspensions for at least a longer period.

Improve this answer
1
  • 4
    To provide context on how review suspensions are currently levied: basically, they're either manually issued by a moderator, or automatically for failing audits. When you come off the suspension (both auto and manual), you're placed on a "review probation" for 30 days, where if you fail a single audit, you're suspended for twice the length of your past suspension. If you're not currently on review probation, you'll be automatically suspended if you fail three audits in a 30-day period, for half the length of your previous suspension, or 2 days if you've never been suspended from review.
    – Sonic the Anonymous Hedgehog
    Commented Aug 1, 2020 at 8:34
-10

When I read this title in the Stack Overflow sidebar, I was curious to see the comments, votes and replies to this post. When I actually saw those, I had to make sure I was awake and not dreaming of living in some distopian future.

By now I'm used to seeing huge backlash and deep downvotes on posts announcing seemingly benign changes with little perceived improvement. My surprise to see the opposite stems from how I see this issue.

Reviewers that get banned, are reviewers that mindlessly click through the review queues, choosing the safe option every time, not to aid in keeping the site clean, but to gain shiny badges. They are actively hurting the work of flaggers and other reviewers. When they get caught doing so, they will be banned. Or asked to take a good hard look at themselves. Or suspended. Whatever.

Well congratulations, you have achieved that people who do so now feel less offended because they are no longer "banned", but "suspended" instead.

Really? See George Carlin on Soft Language.

Improve this answer
6
  • 9
    How you approach someone when you criticize them matters. If softening the language makes some people more likely to read and follow the guidance, then it’s worth doing. Besides, “suspension” is a more accurate description than “banned”.
    – ColleenV
    Commented Jul 29, 2020 at 10:59
  • 2
    When I consistently drive too fast and receive a "minor financial inconvenience" instead of a "big fat fine", I'm not more likely to improve my behavior. But whatever makes linguists and pseudo-psychologists happy, I guess. I don't see this as an improvement whatsoever, and wanted to have that out here amidst all happy responses.
    – CodeCaster
    Commented Jul 29, 2020 at 11:03
  • 10
    Do you know the difference in definition between “banned” and “suspended”? This isn’t making up some euphemism, it’s actually more accurate to say “suspended”. Just like “moving violation” is more accurate than “big fat fine”.
    – ColleenV
    Commented Jul 29, 2020 at 11:05
  • 2
    Bans have been called that in an online context for decades, and can be lifted.
    – CodeCaster
    Commented Jul 29, 2020 at 11:09
  • 8
    And that has little to do with communicating effectively with the person who is being suspended. I agree with you that sometimes companies do dumb pointless things with language to make it look like they're solving something, but I don't think this is an instance of that.
    – ColleenV
    Commented Jul 29, 2020 at 11:16
  • 5
    This isn't coddling. It's a change moderators have been asking for for a very long time. We aren't going to be softening our stance on people who misuse the review queues. This change merely allows us to be more accurate and informative. It also increases the chances of educating someone so they can fix their mistakes/misunderstandings and become a better reviewer. That might not happen, of course, but it's at least worth a shot.
    – Cody Gray - on strike
    Commented Aug 2, 2020 at 3:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .