435

Background

This is somewhat specific to Stack Overflow. My concern is the "I understand" button that appears when you fail a Review Audit.

I had this experience twice recently (after never having previously failed an audit), once shortly before the revamping of the Close/On Hold mechanism and once today.

In both cases (one a Close audit, one a Reopen audit) I'd spent several minutes studying the Question under Review, and would happily explain my reasons for voting as I did. I don't expect the automated audit process to be "perfect". However my post here is to point out the "I understand" button needs an alternative, which might be useful in trapping those Review Audit cases which have significant flaws for this purpose.

Since the Stack Exchange Community determines standards and moderates itself, "I understand" that the primary purpose of the audit reviews is to discourage superficial "click-through" behavior on the part of members eager to gain the corresponding badges. While it might also serve to educate users like myself who are motivated to make a dent in the backlog of Close votes, the adjuration to "STOP! Look and Listen" doesn't really accomplish this.

Proposal: Add an "I don't understand" button

So, my proposal is to implement an "I don't understand" button. It would do two things, one mildly punitive to ensure sincerity. It would suspend Review Privileges automatically for a week, or until a Moderator sees fit to remove the suspension. It would also give the User who clicks on it a chance to explain the reasons for his/her review choices, instead of placing them in the double bind of having to click "I understand" or leave the field.

Examples

I was reluctant to give specifics of my Review Audit Fails as it might appear as a cry for a personal remedy. Now I think some specifics might help to convey the fact-intensive nature of these failures. My proposal aims to have quick impartial reviews while the Reviewers memory is still fresh (and can best accomodate a lesson-to-be-learned, if appropriate).

The first example was in a Close Vote Review (IIRC qua Not a Real Question) of a Question about current development options for Windows desktop applications. The OP disliked MS "Windows Presentation Foundation" framework but spoke favorably of some older frameworks.

A few Comments under the Question asked for clarification and complained about an erratic and distracting typography (boldfacing numerous tool names/buzzwords), and the OP had not (yet) responded to any of these. I thought that all this happened recently (within the past day).

After studying the Question I concluded that it was worthwhile and could be answered (e.g. with objective information about HTML5 technology). The annoying boldface emphasis and slightly whiny tone, but (1) I didn't assess those as deal breakers and (2) given a bit more time perhaps the OP might undertake to make those changes him/herself. With these considerations in mind, I voted to Leave Open.

Afterward the system disclosed that the Question was actually much older (a month+) and had been Deleted soon after being Closed. Given an opportunity to respond and revise, the OP may have elected instead to Delete their Question.

My second example was a Reopen Vote Review, recently enough to have the new Close Reason framework (apparently as an Off-Topic closure, IIRC). The Question asked about why a Python inequality comparison succeeds in cases involving nonnumeric values. Comments under the Question confirmed the behavior but noted its elimination in Python 3, as well as remarking that why the "feature" existed previously would be open to discussion. [Presumably the Question is still out there on Stack Overflow, but searches using both Stack Exchange and Google did not find the exact item, only duplicates.]

I voted not to Reopen based mostly on two considerations. First, the Question had never been edited, and so not eligible for reopening based on revision. Second, it appeared that the Question was something of a call for mildly opinion-based discussion ("why" does Python 2 behave that way) and closed accordingly.

While these Audit Review cases may well have been constructed programmatically or manually by someone who thought them quite cut-and-dried examples (and I accept the utility of doing something like this), there should IMHO be a feedback mechanism that weeds out those not susceptible to a "teachable moment".

Added: I have discovered a new way to fail a Review Audit, by making (or trying to make) a Comment on (purportedly) a User's First Post. According to the information presented, the 2 hour old Answer being reviewed was being posted more than a week after an Accepted Answer, and it was frankly a little terse in phrasing, so I intended to make a Comment to the effect that when posting Answers so much after the fact, brevity in supplying detail is not so virtuous as it might be when answering in a more timely fashion. The reveal: I'm actually reviewing the tersely phrased but Accepted Answer posted nine days before.

Added(2): In going through a Close Review Queue, I came to a Question that was clearly being presented as an Audit ("primarily opinion based" being the presumptive reason to consider closing). However the Question was an exact duplicate of a previous Question with an Accepted (and upvoted) Answer, so in my bullheaded way I chose Close as Duplicate (and failed the Audit).

Improve this question
29
  • 62
    An idea: The 'I don't understand' button could create a post on meta, or a chat room about the audit where people could discuss it, not simply for moderators to review. Just a thought.
    – Richard J. Ross III
    Commented Jul 14, 2013 at 14:30
  • 6
    @RichardJ.RossIII We could even make it a 10k list.
    – Undo
    Commented Jul 14, 2013 at 14:39
  • 24
    If coders really want to code, I also suggest a "I disagree" button that lets a mod review the audit. Mod agrees with your complaint => as if audit never happened and it won't ever happen again. Mod disagrees => you move towards the ban faster than if you never complained.
    – John Dvorak
    Commented Jul 14, 2013 at 14:53
  • 64
    @Richard No to Meta - then every person who fails an audit because they legitimately suck at reviewing will throw it over on Meta and we'd end up with a mountain of "why did I fail?" questions. Meta should be reserved as a manual process for good reviewers to ask about audits they legitimately don't understand. Only people who actually need help with an audit will come here, and that's a good thing.
    – animuson StaffMod
    Commented Jul 14, 2013 at 14:54
  • 10
    I disagree. I would understand "I don't understand" as "I don't agree with the audit. Some audits are pick up by random, therefore they can (and sometimes are) be quite controvertial.
    – Cjxcz Odjcayrwl
    Commented Jul 14, 2013 at 17:03
  • 10
    +1, I'd be totally behind this! I think the title of your question could be a bit more direct, though: maybe something like "Add an "I disagree" button for failed review audits"?
    – Ilmari Karonen
    Commented Sep 2, 2013 at 14:31
  • 2
    I've been using this as a duplicate so I just wanted to edit to bring out some of the key points, hope you don't mind
    – Richard Tingle
    Commented Oct 15, 2013 at 9:57
  • 12
    I'd vote for a "No, your review failed" button. I've failed several audits recently where the question was clearly off topic, or didn't ask a question at all. It is one thing if I don't understand why the question is good, but completely different when I am still confident the question is bad.
    – OtherDevOpsGene
    Commented Dec 13, 2013 at 14:55
  • 5
    Hope this happens soon... I just 'failed' a question that was (in my opinion) off topic (suitable for programmers.stackexchange), but the audit thought it was 'high quality'.
    – Gus
    Commented Dec 24, 2013 at 18:35
  • 6
    I just hit a question that needed to be closed as a duplicate (a previous commenter had left a dupe link) and got the "look and listen" BS. If this is going to happen frequently, the review queues can kiss my behind.
    – Pekka
    Commented Dec 26, 2013 at 14:55
  • 7
    I just failed an audit because I was about to add a comment on a "good question". However, before I was able to add the comment or really do anything else, I failed. Others had commented on the question. The idea that doing anything other than upvoting a "good" question goes against the idea that this site is based on voting since you can give a "wrong" vote.
    – zero298
    Commented Apr 5, 2014 at 22:24
  • 16
    On the two occassions where I got a bad audit question, I never clicked on the "I understand" button. I just navigated away from the page. It's very condescending anyway, even when it's right. I'd prefer something like "Whatever you say..."
    – ouflak
    Commented Jul 3, 2014 at 17:05
  • 7
    I stopped reviewing after getting audit fails for thoughtful and careful audits. Helped turn me away from SO. I only work on other sites now.
    – andy256
    Commented Sep 16, 2015 at 0:27
  • 7
    Coming to this discussion a little late, but I agree that the "I understand" button should be modified. It's offensive to find oneself somewhat obliged to click on something which either you disagree with, or simply DON'T understand!
    – Strawberry
    Commented Feb 17, 2016 at 14:01
  • 6
    ���I don't understand” implies that the user needs more accurate explainations.
    – neverMind9
    Commented Jul 11, 2018 at 22:39

5 Answers 5

Reset to default
579
+400

Rather than an "I don't understand" button, I could get behind a "This audit is incorrect" one that required you to type an explanation for why you disagreed (with a minimum character limit).

Such a button would remove an audit case from rotation pending review by devs (or moderators) in a special list. These cases seem like they'd be quick for us to review, and would greatly reduce traffic on Meta complaining about audit failures. A disputed audit would not count against a user unless it was reviewed and found to be legitimate.

We'd be able to quite clearly see if someone was abusing this option, and since we have the ability to apply manual review bans, such abuse would not last long.

We all recognize that, outside of the suggested edit audits, there are plenty of bad or borderline audit cases out there. If we can identify and remove those, I think it would improve the overall audit process.

Improve this answer
22
  • 95
    Will there be audit review audits for moderators? :P
    – Manishearth
    Commented Jul 14, 2013 at 19:04
  • 34
    Note by the way, that the very ability to dispute review audit can be rate-limited in addition to already established rate limits on review. I would consider something like starting with allowing one audit challenge per week (or maybe even per month) for a novice reviewer. For steward reviewer, one challenge a day feels reasonably safe.
    – gnat
    Commented Jul 16, 2013 at 13:55
  • 4
    A good idea would be to place the action that failed the audit (downvote for good audit, commenting on good audit, upvoting bad audit, etc.) so that they can place it in context.
    – Qantas 94 Heavy
    Commented Oct 15, 2013 at 12:09
  • 3
    It took a while for me to come around to the "This audit is incorrect" version because of the more confrontational tone, but I'd always envisioned typing an explanation for disagreement. I can see the virtue in directly disputing the audit case.
    – hardmath
    Commented Nov 5, 2013 at 0:33
  • 43
    Any progress on implementing this? I just hit a borderline one myself (poor but salvageable question, not all that old, named user account) and would have liked the opportunity to explain my reasoning.
    – zwol
    Commented Dec 10, 2013 at 4:07
  • 4
    On Mathematics I have just had my first fail after several passes. I think it was a judgment call, yet I was told I was wrong without any chance to engage. I think that fail rate should be taken into account before suspending anyone - everyone will disagree occasionally, and sometimes a consensus needs to be tested. And I also think that the tests need to feed a live conversation about what a good review is. Some of the Maths homework questions get closed before the original poster has a chance to respond to very helpful comments.
    – Mark Bennet
    Commented Apr 22, 2014 at 19:59
  • 2
    I like this proposal with the exception that a disputed audit should continue to count against the user until resolved. Otherwise, users have an incentive to dispute even legitimate audits.
    – Warren Dew
    Commented May 23, 2014 at 6:05
  • 1
    @MarkBennet: Well, when he gets around to it, his edit will push his question into the reopen-queue, and if he actually resolved the problems with his post, it will be reopened fast. It's working as intended.
    – Deduplicator
    Commented Sep 10, 2014 at 18:57
  • 17
    Any news on something like this being implemented? I just failed an audit where an answer was removed for being "not an answer", even though it was actually a duplicate of an earlier answer with 203 upvotes. No way to know that from the review screen, of course.
    – stkent
    Commented Jan 4, 2015 at 15:19
  • 2
    Not at this point :) Should something removed as a duplicate not show up in the review queue then?
    – stkent
    Commented May 20, 2015 at 19:33
  • 3
    Just an idea, but say if the audit was good then it should be included in a list somewhere with an explanation of why it was a good audit so other users could learn from it.
    – Diminutive Colossus
    Commented Jun 11, 2015 at 16:30
  • 16
    This Q&A has been open for over two years. Is there any update? There are many questions on meta that relate to poor review audits so it would seem that that doing something about them is considered highly desirable by the Stack Overflow community.
    – AdrianHHH
    Commented Sep 14, 2015 at 14:41
  • 13
    Another year has passed, and we still desperately need this. The audit system keeps proving itself to be broken, sometimes training opposite behaviors simultaneously.
    – CodeMouse92
    Commented May 3, 2016 at 22:42
  • 3
    Another year has passed. Still missing this feature. Still getting bit by poor audit questions.
    – Bacon Bits
    Commented Apr 10, 2017 at 13:05
  • 3
    @BaconBits Another two year passed. Some high-rep important users left the SO because unfair treatment of avid reviewers. SE seems deaf to the whole problem.
    – peterh
    Commented Aug 26, 2019 at 10:11
124

I'm done reviewing anything until there's some sort of automatic disputation option for audits. I don't want to create a meta question for every bad audit either. A number of times, I have opened the offending question or answer in a new window, taken the action I believe is appropriate on it, and moved on. But that's getting old for me.

I have not failed a lot, but the failed audits are very invalidating. Fundamentally I can not click "I understand" on a completely incorrect warning.

As Picard said, "There are four lights!"

I propose a different mechanism that the question or other answer upon clicking "I disagree".

  • go ahead and count the audit as failed (shouldn't happen that often for a careful reviewer anyhow)
  • let me click "I disagree"
  • make me prove I'm not a robot (CAPTCHA'd)
  • let me perform all the actions that I would have performed on the post if it was not an audit or I encountered the post outside the review queue (in my understanding, this would often drop the post out of the audit pool automatically as well)
  • let me click "done" and continue onto the rest of the queue

I think this has the advantages of not being any more gameable by robo-reviewers while still allowing me to do the cleanup that the review queues invite and not psychologically torturing me by making me click "yes" when I believe "no".

Improve this answer
5
  • 19
    You make a number of points consonant with my view, not least of which is not creating a meta question for every bad audit. The only further point I'd like to emphasize is that I'm personally open to some kind of learning component/explanatory mechanism when an audit fail seems wrong. E.g. the community standards at StackOverflow need not be the same as at Math.SE, and perhaps some tweaking (of my "collegial" judgements in reviews for the latter) is necessary to do reviews properly at SO.
    – hardmath
    Commented Aug 1, 2013 at 16:12
  • 16
    I find the whole concept of review audits patronising and insulting. I used to sometimes go to the review queues, in the interest of helping the Stack Overflow community. I mean, we all play our part, right? But after two "review the reviewer" questions, I decided I don't need the grief. If the community doesn't trust me to review correctly, that's fine. And I will never again review anything.
    – Dawood ibn Kareem
    Commented May 4, 2014 at 8:38
  • 21
    "...but the failed audits are very invalidating". I couldn't agree more. I just copped a review audit failure (my first, and frankly my last), for erring on the side of trying to improve an answer rather than recommend deletion (amongst the verbosity there were some very helpful points from an apparently experienced practitioner, of a relatively obscure technology such that the info contained within was of high value). To be told I should have looked more closely is quite condescending given the care and consideration I had always applied to my reviews.
    – Sepster
    Commented Apr 18, 2015 at 12:02
  • 2
    How can you get mad at (or feel insulted by) a computer which is simply doing what it was told to do. It's not the computer's fault that a bunch of idiots upvoted a bad question for the wrong reason. Just like it's not your computers fault that you got a virus from browsing porn. Don't blame the computer, blame the idiots that upvote bad questions and answers or downvote good questions and answers (also, blame the idiot that was watching porn without adblock)
    – Diminutive Colossus
    Commented Jun 11, 2015 at 21:08
  • 12
    @TinyGiant I never blame the computer. I blame the people who programmed the computer.
    – David Moles
    Commented Nov 4, 2015 at 19:00
18

Qualities of a good Audit

  • 90%+ people pass without problem
  • 50%+ of people who fail understand why

Qualities of a bad Audit

  • 50%- people fail (might as well let a nickle do my reviews!)
  • 50%+ of people who fail don't understand why (The Audit is either wrong, or has an ambiguous answer)

In general, I think the audit system should be able to moderate itself. By tracking an audit's history, the system should implicitly know the quality of an audit. If too many people are failing an audit, it should be removed auto-magically (no mods or review ques needed). At the same time, there REALLY should be a "I don't understand" option, and/or the option to dispute the audit.

As is, people failing audits are posting Meta's about why/remove the audit because they think (and it is) the only thing they can do to improve the audit system.

Other things that could be done to help moderate the audits is

  • An option to vote on audit quality
  • An option to dispute bad/low-quality audits
  • Dropping audits that nickles are better at passing (and after x uses).
  • An audit queue (where new audits go to be approved before entering the system)

But at the very least, SOMETHING has to be done.

Improve this answer
2
  • 5
    Even a low-pass automated system that removed the bottom 10% would be an improvement.
    – TemporalWolf
    Commented Dec 4, 2017 at 23:37
  • How to prevent robo-reviewers in the audit queue? It can't have any audits!
    – mathlander
    Commented Jan 28, 2023 at 20:20
14
+50

Better fix: remove the audits. We may be willing to be digital sharecroppers, but at least do us the courtesy of not 1) wasting our time with fake reviews; 2) insulting us when we "fail" particularly questionable fake reviews.

Improve this answer
24
  • 15
    Review audits serve a valuable purpose of reining in robo-reviewers, ensuring everyone is paying sufficient attention. Do you have an alternative solution for that?
    – mhlester
    Commented Apr 22, 2014 at 19:59
  • 4
    What is a "robo-reviewer"? I have yet to see a bot account accrue 500 reputation, never mind enough to cast open or close votes...
    – user152743
    Commented Apr 22, 2014 at 20:07
  • 9
    robo-reviewers aren't actually bots, just users approving everything they see in order to get the Steward/Reviewer badges
    – mhlester
    Commented Apr 22, 2014 at 20:22
  • 3
    @mhlester unfortunately, we still have those even with audits on.
    – John Dvorak
    Commented Apr 22, 2014 at 20:42
  • 4
    Then I would suggest there are two possible situations: 1) most reviews require careful thought, and "robo-reviewers" are clear outliers that can be dealt with by the mods in the same way as robo-upvoters, downvoters, flaggers, etc. 2) "robo-reviewers" cannot be detected heuristically because most reviews do not require any thought, as "accept" is correct 99% of the time, and the review queue should just be removed entirely.
    – user152743
    Commented Apr 22, 2014 at 20:42
  • My experience is that open/close queues are in the former category, and suggested edits in the latter, though this is on math.se and I have no experience what things are like on the other networks
    – user152743
    Commented Apr 22, 2014 at 20:43
  • @user152743 as for 1) - what would be the way? You can't just reverse everything. Also, you need a way to detect them. 2) is definitely not the case.
    – John Dvorak
    Commented Apr 22, 2014 at 20:44
  • If 2) is not the case, then it is easy heuristically to identify users who are abusing the system. Flag those users for moderator review, and ban abusers from the review system (or the network entirely).
    – user152743
    Commented Apr 22, 2014 at 20:49
  • 24
    I'd say get rid of the badges. A shame, but the current cure is worse than the disease. I would review out of the goodness of my heart, but I don't anymore because I too think audits are insulting.
    – Ben Millwood
    Commented Apr 27, 2014 at 14:36
  • 1
    +100 if I could. A shame I posted my comment under the other answer before I saw this one.
    – Dawood ibn Kareem
    Commented May 4, 2014 at 8:40
  • 6
    I will never understand why some people take it so seriously when they are "insulted" by a computer :D. Same with downvotes and close-votes, virtual stuff. Simply stop taking it personally.
    – kapa
    Commented May 23, 2014 at 9:58
  • 7
    @kapa This is not a game. We expect to deal with people here, not AIs.
    – Noctis Skytower
    Commented Jun 9, 2014 at 14:29
  • 9
    I also "tried to do the right thing" today and got banned. If no one is worrying that people trying to do the right thing are getting punished, something is wrong. If such people refuse to perform reviews in the future, the system contributes to lower performance.
    – Noctis Skytower
    Commented Jun 9, 2014 at 14:55
  • 8
    If you're contributing time and patience on reviewing and you're getting banned for reason you strongly disagree with, then yes - it is offending. There's little intelligence involved in judging. It's a matter of stand. There's no straight line isolating bad posts from the good ones. Just give us an option to circumstantiate our verdicts.
    – Tarec
    Commented Jul 4, 2014 at 17:16
  • 2
    "Doing the right thing" is entirely subjective. Good reviewers rarely get banned because they follow the community rules. Those who have misguided ideas about the rules and guidelines put in place by the community will fail audits regularly. One bad audit cannot get you an audit ban, and if you dispute it and you win then the failed audit is expunged from your record. The only real reason that you would get a review ban is if you are going to fast and you're not paying enough attention. I agree that any gains for the reviewer should be removed, good reviewers don't care about the badges.
    – Diminutive Colossus
    Commented Jun 11, 2015 at 16:42
7

I'd like to add here my vision (transferred from meta SO), maybe this 6-year old question is re-visited by anyone having the power to trigger the change.

For many reasons, it would be handy to have an option/feature which could allow reviewer to dispute review audits whenever they find them inappropriate right on the spot, by clicking link or button, without having to resort to making a new topic on meta. Introducing such feature could boost the process of polishing review audit system to help filter it from bad audits.

Advantages (summed up)

Much more users involved

  1. Making one click is faster than making up a new topic on Meta.

  2. Making one click is an opportunity for users not registered on Meta to report disputed audit they may not even aware it has been possible.

As a positive side effect, this can potentially reduce amount of disputed audit topics on meta. Those who never took part in audits improvement, get involved; and those who did it via meta topics do it silently.

Voting beyond personal interest

It is common for people to come here to dispute audits they were involved with, e.g. to lift a review ban. Being able to vote for an audit goes beyond personal interest. You don't necessarily vote because you were involved, you vote to improve the system, and it costs next to nothing.

Possible solutions, restrictions

One of the possible restrictions is to make this link (button) visible only for users, possessing the Reviewer badge (for any queue). This will temporarily restrict dispute reports from review starters.

There are few options on how to carry this system. One of the options is to make it collect votes, and once there are enough, the particular audit is removed.

The link (or button) should be visible regardless of the result of the review audit (passed or failed), right after it is complete:

Review audit success

Placing small link beneath the main text, stating something like "report disputed audit" (discussable) would be Ok in my opinion.

Another variant is to introduce two-way voting (up/down) options for audits to rate them. This way we have medium. And once the rating is enough, review audit is also removed.

Finally, as Tezra wrote, at the very least, SOMETHING has to be done.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .