142

The way the current auto-bans for reviews work is rather suboptimal. Somebody who's really after the badges and doesn't care whether they're banned for two days can just continue clicking through, without fearing the next ban—because hey, it'll be two days again. You just wait it off and continue.

See for example this review robot, who has been banned five times—and nobody noticed!

Mar 28  user has been banned from review  duration = 30 days by Sathya
Mar 22  user has been banned from review  duration = 2 days
Mar 11  user has been banned from review  duration = 2 days
Mar 7   user has been banned from review  duration = 2 days
Mar 1   user has been banned from review  duration = 2 days
Feb 25  user has been banned from review  duration = 2 days

Quite recently we had another user who is quite persistent at reviewing, even though they've been told to pay more attention. They were now banned for the third time and we didn't notice.

Here's another example:

Jul 10  user has been banned from review    duration = 2 days
Jul 4   user has been banned from review    duration = 2 days
Jul 1   user has been banned from review    duration = 2 days
Jun 26  user has been banned from review    duration = 2 days
Mar 28  user has been banned from review    duration = 30 days
Mar 22  user has been banned from review    duration = 2 days
Mar 11  user has been banned from review    duration = 2 days
Mar 7   user has been banned from review    duration = 2 days
Mar 1   user has been banned from review    duration = 2 days
Feb 25  user has been banned from review    duration = 2 days

Obviously, even after being banned for an entire month, this user managed to continue failing audits like crazy. He's now banned for another 30 days accompanied with a mod message.

One helpful tool would be to see how often users have been previously banned. Even that requires moderators to actively ban someone though.

But if we expect the system to do some automatic checks, why shouldn't the duration of bans increase? Much like the suspensions, which progress from 7 to 30 and 365 days.

For example:

  • First ban: 2 days
  • Second ban: 7 days
  • Third ban: 14 days
  • Fourth ban: 365 days
  • …

Those numbers are of course up for discussion. Just an example.

This would ensure review robots can't just go on doing what they do without really getting into trouble.

Improve this question
9
  • 19
    I think the fifth ban ... stands for life time? :P
    – Himanshu
    Commented Mar 29, 2013 at 10:17
  • 4
    I'm all for a three strikes system!
    – Time Traveling Bobby
    Commented Mar 29, 2013 at 10:21
  • 55
    The progression could be softer but maybe a removal of all progress towards the badge should be done starting from the third ban ?
    – Denys Séguret
    Commented Mar 29, 2013 at 11:56
  • 1
    @dystroy Good idea. The numbers in my question are just examples and of course open to debate. I just think something needs to be done.
    – slhck
    Commented Mar 29, 2013 at 11:58
  • 3
    Some science could be applied to the data to see how often there are repeat offenders. The review problems are disproportionately caused by a very small percentage of the user base who don't know that their activities are monitored; when they get banned the first time, they may stop of their own accord.
    – user102937
    Commented Mar 29, 2013 at 16:15
  • 2
    @RobertHarvey If these users stop after the first ban, then that's fine and maybe the ban encouraged them to do the right thing, but what about those who are clearly abusing the system like that? Of course, looking into the data would be nice to see whether that's a real issue or not, but at the moment, the mods have no way of finding out really…
    – slhck
    Commented Mar 29, 2013 at 16:20
  • 7
    I would opt for exponential growth which decays over time. For instance, 2^(num bans) - 2^(months since last ban) with as minimum of zero.
    – user206222
    Commented Apr 24, 2013 at 6:40
  • 2
    If the root cause is suspected the badge-chasing why not cure that? Strip and block the attached badge. After all it should reflect good service. If service was bad, better revoke.
    – Balog Pal
    Commented Jun 6, 2013 at 2:53
  • 1
    @dystroy Maybe not a complete removal. Because what if you legitimenly reviewed questions and then started failing like crazy? Maybe remove half?
    – Cole Tobin
    Commented Jul 20, 2013 at 20:44

6 Answers 6

Reset to default
75
+100

This has been implemented and deployed in build 2013.7.26.1279 (meta) / 2013.7.26.898 (sites). The triggering mechanism was not changed. It can be turned on or off on each site. When turned on it works like this:

each review ban counts (even manual ones made by moderators).
a 30 day window is used
1st ban within the window -> duration: 2 days
2nd ban within the window -> duration: 7 days 
3rd ban within the window -> duration: 30 days

Robo-reviewers, beware!!!1

Improve this answer
13
  • 2
    With the third ban being as long as the window, does that mean that a fourth ban would guaranteed be in a new window, and thus be only two days long?
    – Billy Mailman
    Commented Jul 26, 2013 at 16:27
  • 3
    @BillyMailman yes
    – m0sa
    Commented Jul 26, 2013 at 16:28
  • Great, thank you! Where can we turn this on or off? Or is it a dev-only thing?
    – slhck
    Commented Jul 26, 2013 at 17:07
  • 1
    @slhck in site settings / staff only
    – m0sa
    Commented Jul 26, 2013 at 18:40
  • 1
    FWIW, this is enabled currently enabled on the trilogy (S[OFU]) @slhck. Review bans are strictly manual on other sites, so mods are free to set them to whatever they feel is appropriate. As we expand audits, we'll probably delay this a bit to allow folks to get used to the idea.
    – Shog9
    Commented Jul 29, 2013 at 3:59
  • 1
    @Shog9 thanks for the info. I wasn't aware of automatic bans only being used on the trilogy sites. However, that makes me wonder, have you got any feedback on how the mods are doing on the other sites? I've found it really hard to track review abuse, even with automated bans (partly because on the banning page you can't see who was previously banned, or how often).
    – slhck
    Commented Jul 29, 2013 at 6:33
  • 1
    I don't think Meta reviews have audits. I haven't faced any review audit yet.
    – Himanshu
    Commented Oct 2, 2013 at 9:04
  • 1
    Any progress on either appealing failed audits or improving the accuracy of the audit questions? I've been reviewing at a high volume and pass most of my audits but have been banned (for the 30-day term this time) for failing mostly wrong audits.
    – chrylis -cautiouslyoptimistic-
    Commented Oct 3, 2013 at 20:10
  • What exactly is this 30-day window? When does it start? From the first ban?
    – Szymon
    Commented Oct 5, 2013 at 4:58
  • What about 4th ban? 5th?
    – Shadow Wizard
    Commented Nov 3, 2013 at 20:19
  • 5
    Unless the 3rd ban is lifted early, it shouldn't really be possible to get banned 4 times in 30 days, @Sha.
    – Shog9
    Commented Nov 3, 2013 at 20:33
  • @Shog9 good point!
    – Shadow Wizard
    Commented Nov 3, 2013 at 20:34
  • 6
    "a 30 day window is used" - is it a sliding window? I mean the one that always ends "today" and started 30 days ago?
    – Mołot
    Commented Nov 5, 2013 at 10:02
22

I think the intervals would be better set up like this:

First ban: 2 days - Inform the user of escalating consequences should the poor reviews continue.
Second ban: 7 days - Remove that day's badge progress
Third ban: 30 days - Remove all badge progress, restart at 0
Fourth ban: 365 days - Remove the ability to earn review related badges altogether
Fifth ban: Permanent.

It's time our review system grew some teeth...

Improve this answer
4
  • Were you thinking any sort of time windows (i.e. the way it is now)? (I'm happy either way.)
    – Bernhard Barker
    Commented Nov 5, 2013 at 16:06
  • @Dukeling A 60 day or 2 month window seems reasonable, but I think there should be some memory stored for repeat offenders. As in if you're banned twice a every 60 days, the system should keep track of that, and escalate accordingly.
    – apaul
    Commented Nov 5, 2013 at 16:57
  • 8
    I think 5th ban should include summary execution! Who do these people think they are? failing audits is horrendous. They owe SO their very lives!
    – Dave Alperovich
    Commented Nov 23, 2013 at 21:20
  • 7
    The first three suggestions are very good. However, not in favor of irreversible punishments. It's not the Community way.
    – JonathanDavidArndt
    Commented Dec 12, 2017 at 4:24
19

I think the ban times should decay over time.

Bantime_1 = 2 days
Bantime_n = (Lb - Wa) * 3 days

where Lb is the last bantime, Wa is the number of weeks since the last ban during which the user completed at least 25 reviews.

This way, users' bantimes don't become excessively long, but still grow exponentially if they are consistently banned. If a bantime would be less than two days, make it two days.

Improve this answer
8
  • 1
    I like this idea, it would prevent people from receiving long term bans if they only stumbled once in a blue moon, but I think the specific algorithm would be better kept secret, so that people would still be careful not to stumble too often
    – apaul
    Commented Jul 22, 2013 at 17:36
  • Is Lb the number of weeks since the last ban? Otherwise I'm not really following your formula ((time - weeks)*days??). And some examples may help, maybe a grid or two. I can sit here with a calculator working out the values to get a feel for what we're talking about, but looking up the values in a grid would be so much easier.
    – Bernhard Barker
    Commented Nov 3, 2013 at 20:57
  • @Dukeling days = (weeks-weeks)*days; both Lb and Wa are in weeks.
    – user206222
    Commented Nov 5, 2013 at 4:02
  • Your formula is flawed - if the last ban was this week, Lb - Wa = 0. If the last ban was a while ago, and you didn't review much during this period, Lb - Wa can be quite large. We'd rather have it be the other way around. And there should be some sort of increasing punishment - you shouldn't be able to get banned every week (or however long) - we want users' ban times to become excessively long if they keep reviewing incorrectly (over a short enough period of time).
    – Bernhard Barker
    Commented Nov 5, 2013 at 7:59
  • @Dukeling Ah, I'm sorry, Lb is your last bantime in days. That way, your next bantime decreases as active review weeks pass.
    – user206222
    Commented Nov 5, 2013 at 8:07
  • @Emrakul What was said in my last comment holds whether Lb is days or weeks.
    – Bernhard Barker
    Commented Nov 5, 2013 at 8:32
  • @Duke Nope, because Lb-Wa=2; the ban becomes six days. The bans increase exponentially, assuming Wa does not change. It's only three to decay Lb. If you didn't review much, your ban time should hold, at least until you prove you know what you're doing. That's intentional.
    – user206222
    Commented Nov 5, 2013 at 14:39
  • The conception is o.k, although SE doesn't really like such valid calculations. There are always & everywhere direct numbers in the rules (f.e. second audit error in a 30 day window causes 7 day audit ban). If it doesn't work, they tend to implement a direct numbered system, but they hold the exact rules in secret. Maybe they aren't really good from math, and about thinking in complex formulas, generally.
    – peterh
    Commented Jun 13, 2014 at 10:57
12

While the idea that robo reviewers should be banned for longer is worth taking into account, please note the number of disputed audits.

As currently implemented, it's almost impossible (for reviewers that don't "cheat" using enhancements detecting audits) not to fail an audit from time to time! Implementing your request literally would eventually permanently ban all reviewers.

The ban should be increased only if the reviewer continues with failed audits after previous ban is lifted. Good audit history before meeting new review ban should not result in more severy ban.

Such implementations would target only those reviewers, who show no improvement and tend to robo-review. The reviewers that get a ban occasionally because audits are chosen blindly should not be punished.

Improve this answer
5
  • 2
    Agree 100%. Sometimes the questions/answers selected for reviews are just plain wrong, often due to policy change, so it's unavoidable to select the wrong review option sometimes.
    – Amicable
    Commented Feb 25, 2014 at 16:41
  • Also, you should probably submit this as a new question with the feature-request tag and link back to this question.
    – Amicable
    Commented Feb 25, 2014 at 16:41
  • 4
    Never forget that there's a fair bit of sampling bias at work when evaluating the volume of meta complaints on a given subject. There are many hundreds of audits every day...
    – Shog9
    Commented Feb 25, 2014 at 16:43
  • 1
    I have yet to see someone complaining about being banned for failing audits when doing the right action. It's absolutely okay to fail an audit from time to time, and this doesn't get you banned automatically.
    – slhck
    Commented Feb 25, 2014 at 19:39
  • You miss the point about the bad audit question. More and more of them are being reported. Then again, maybe @dukeling is right... 3rd bad should result in execution.
    – Dave Alperovich
    Commented Apr 16, 2015 at 5:27
5

I've been banned a few times. Sometimes I was careless, other times it was not very clear what the Audit Problem was.

Honestly, by failing audits I've learned better what StackOverflow looks for in Questions and Answers.

And further, little of this understanding could be built another way. As time goes by, I pass more and more audit questions.

I think the real problem is badges. I don't give a !@$*@ about these kinds of badges. They actually hurt my performance of reviews because when I get one, I almost feel like "why bother" reviewing more now that I have this badge.

But the sad part is, I wasn't reviewing for that reason before.

Improve this answer
5
  • You should be learning what's appropriate through using the site, not reviewing. Reviewing is supposed to be a moderation task, moderators are supposed to be the people who know the rules really well (and I'm not talking about who Stack Exchange calls moderators). But I agree about the badges (though I'm not sure I'd call it 'the real problem').
    – Bernhard Barker
    Commented Nov 23, 2013 at 21:54
  • If you're not sure about what to do with a post, you should be skipping it (and then feel free to come ask a Meta question about it, or see how other people reviewed it, which is far from ideal, since there are too many people reviewing incorrectly currently). If you find yourself skipping most posts, perhaps you should put reviewing on hold for a while and come back after you've used the site a bit more, or try to find a filter that works for you.
    – Bernhard Barker
    Commented Nov 23, 2013 at 22:06
  • 4
    @Dukeling, I argue that SO community goals in what makes a good question, edit, and answer often has a large amount of variance. I see many useful edits and notice other users note them as too minor. And then there's "too radical". Ugh, who can really say with any certainty. Yes, it's a learning process. I challenge you to show me any FAQ that explains guidelines well. I have seen many questions with various close/remain open votes. I have seen many edits with skisming allow/do-not-allow votes.
    – Dave Alperovich
    Commented Nov 23, 2013 at 22:21
  • 1
    There is a fair amount of variance, but there are way more things that are clearly appropriate or inappropriate. Again I'll say - skip those you're not sure about. If I force myself to review all items, I'll fail plenty of audits, but the point is that I skip plenty, and thus don't fail any audits. "it's a learning process" - but you're not learning anything when incorrectly reviewing posts (which is the problem), only when failing audits. FAQ - What are the guidelines for reviewing? (not sure how useful it is, but it's a start)
    – Bernhard Barker
    Commented Nov 23, 2013 at 22:56
  • @Dukeling, I hear you loud and clear. Me and most other 3K+ should get out of your face and let you run things since you seem to understand everything. Have fun with the close queue ;)
    – Dave Alperovich
    Commented Nov 23, 2013 at 23:05
0

What about either x days (after the last ban was lifted) or y audits with no more than z failed decreases the ban counter by one?

Ban   Duration
1st     2 days
2nd     7 days
3rd    30 days
4th+  365 days

(every ban from the 4th onwards will be 365 days)

Let's say x = 20 and y = 30 and z = 1 (I have no idea whether these values are even remotely acceptable, one way or the other).

After your first ban is over, you need to pass the next 30 audits or wait 20 days.

If you fail one of those 30 within 20 days, you'll get banned for 7 days.

If you then pass 30 audits, the ban counter will decrease to 1.

If you then fail, you'll get banned for 7 days again.

Note: I added a z variable here (it was previously just 1) in case we feel that passing y audits in a row is too difficult.

The main purpose here is to favour users who review a lot by taking into account both how many audits you've passed and how much time period has passed since your last ban.

Improve this answer
6
  • 6
    Passing 30 audits in a row is extremely difficult, depending on the queue, even for me. Audits are often arbitrarily incorrect, so I'm not sure this would work well in practice.
    – user206222
    Commented Nov 5, 2013 at 14:58
  • @Emrakul Well, it doesn't have to be 30 audits (or even just 1 failed for that matter). No-one else suggested something similar, so I thought I'd put this out there.
    – Bernhard Barker
    Commented Nov 5, 2013 at 15:07
  • That is fair. Not totally sure it would work, but it makes sense to put it here.
    – user206222
    Commented Nov 5, 2013 at 16:00
  • This idea hurts SO community. The idea that users are indentured servants misses the point. People who have earned a certain score (supposedly proving a level of understanding) are taking their time out to do something communal. You want to ban people like this? You think there is an abundance of users with scores of 2K, 3K, 5K?
    – Dave Alperovich
    Commented Nov 23, 2013 at 21:30
  • 1
    @DaveA We have a (probably huge) problem with users approving content they shouldn't. Admittedly the problem here is probably the fact that audits aren't extensive enough, but audits 'growing some teeth' isn't bad. But tell me, is this really worse than what's currently implemented? The purpose of this proposal is actually to help reviewers who review a lot (with 'growing some teeth' if you keep failing in a short time frame). Yes, I want to ban these people - if they don't know what's appropriate for the site, they probably shouldn't be performing MODERATION tasks in the first place.
    – Bernhard Barker
    Commented Nov 23, 2013 at 21:47
  • 1
    I'm all in favor of MORE AUDITS. Even more frequent suspensions. The answer should be more aggressive audits and minor suspensions. You want to educate, not ban or turn off. I'm speaking as someone who's been suspended for good reason and bad. There are different users with different goals, But take a look at the ridiculous backlog of Close-Vote reviews and tell me how many 3K+ users you want to ban or suspend for a year.
    – Dave Alperovich
    Commented Nov 23, 2013 at 22:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .