117

I have noticed this "STOP! Look and Listen" message when the system audits you by having you review something to check how you respond.

However, this audit came up and I was tricked. I wanted to up vote the existing comment as well as provide my own. But there was no up arrow for the comment. This completely threw me off and had me thinking something was wrong. To check if any of the interface was working, I went to up vote the question. BAM I am hit with the "You didn't pass" message.

Therefore the audit itself is the cause of this. This is broken.

After running into my 3rd audit (which I passed), there are two pieces that are broken. First, I should be able to up vote comments or else I can detect it is an audit. Second, I am told if I pass/fail BEFORE clicking "I'm Done" giving me no chance to undo an upvote or to perform 2 actions.

Another problem that is caused by the system reacting before the user clicks "I'm Done" is they can click down vote for every single review. If it is an audit, it ends and they passed. If it is not an audit, they can change their vote to an up vote. While this sounds like a lot of work, it really only takes a second longer and ensures you will never fail an audit.

Improve this question
9
  • 17
    Why would you choose to click the upvote button rather than the downvote button?
    – animuson StaffMod
    Commented Nov 28, 2012 at 6:03
  • 22
    @animuson I was clicking anything to check if the interface was working. I thought my browser was misbehaving.
    – Austin Henley
    Commented Nov 28, 2012 at 6:04
  • 11
    You took precisely the action the interface was programmed to dislike .. I'm not sure how that could be done any differently? As far as I know you can't up vote comments through review (at all), I don't believe it's a giveaway. Checking on that though, ICBW.
    – user50049
    Commented Nov 28, 2012 at 6:19
  • 24
    @TimPost Wrong, you CAN up vote comments through review. Also, it did not give me the ability to undo the up vote so how is it realistic in any way? It isn't like I clicked "I'm Done"
    – Austin Henley
    Commented Nov 28, 2012 at 6:20
  • 3
    @AustinHenley You're right, (I did say ICBW) - I was trying to reproduce in the wrong queue.
    – user50049
    Commented Nov 28, 2012 at 6:23
  • Does the interface behave like this on non audits? (I have only reviewed Questions/Answers without comments so far)
    – Richard Tingle
    Commented Jul 16, 2013 at 13:08
  • My complaint is with late answers where an existing answer is shown with a User55555 type new (bogus?) author. I looked at the page and thought someone had just copied and pasted the existing good up-voted answer. So I go to flag and get a 'Stop! Look Listen' audit failure.
    – ficuscr
    Commented Aug 7, 2013 at 0:01
  • 3
    @ficuscr I know exactly what you mean. In fact, I made a post about the missing username just the other day.
    – Austin Henley
    Commented Aug 7, 2013 at 2:51
  • this request was also recently submitted as feedback in Review queue workflows - Final release (evaluated status-deferred at the time of posting this comment)
    – gnat
    Commented Sep 20, 2021 at 20:38

2 Answers 2

Reset to default
103
+50

I think what you discovered is a mistake in the design of "honeypot" test.

This mistake is likely based on the wrong assumption1 that test should catch anyone who randomly clicks on stuff deemed incorrect. The assumption looks incorrect because per my understanding, "honeypot" is intended to catch "fake reviewers" - those abusing review actions with the purpose (P-U-R-P-O-S-E) to increase their review count.

  • The only way to increase review count in First Posts and Late Answers queues is to click specific button(s) like I'm Done or No Action Needed2 - as long as this did not happen, it is wrong to assume test completed.

Note that current test design doesn't fit well to another officially stated goal of review audits: "help new reviewers hone their moderation skills" - unless, of course, a fear of doing easily reversible actions is considered a moderation skill worth "honing".

Mistakes like you have seen jeopardize the intent of the test and open the door to claim it invalid.

A reasonable way to address this issue has been recently proposed here:

...I could get behind a "This audit is incorrect" [button] that required you to type an explanation for why you disagreed (with a minimum character limit).

Such a button would remove an audit case from rotation pending review by devs (or moderators) in a special list. These cases seem like they'd be quick for us to review, and would greatly reduce traffic on Meta complaining about audit failures. A disputed audit would not count against a user unless it was reviewed and found to be legitimate.

We'd be able to quite clearly see if someone was abusing this option, and since we have the ability to apply manual review bans, such abuse would not last long...

Note by the way, that the very ability to dispute review audit can be rate-limited in addition to already established rate limits on review.

I would consider something like starting with allowing one audit challenge per week (or maybe even per month) for a novice reviewer. For steward reviewer, one challenge a day feels reasonably safe.

Improve this answer
18
  • 16
    Exactly, not being able to click "I'm Done" is the biggest fault with it at the moment and it really does defeat any validity in the audit.
    – Austin Henley
    Commented Nov 28, 2012 at 7:09
  • 6
    Waiting until all actions are performed for an audit task wastes valuable time that could be spent reviewing an "actual" review. Audit tasks are only concerned with the first impulse action taken on a post.
    – Geoff Dalgas
    Commented Nov 28, 2012 at 7:25
  • 7
    @GeoffDalgas no matter what, completion of the test before I'm Done is pressed supports claims that test missed the target - "fake reviewers". We already have review system that was intended to improve quality of the posts but somehow missed that target
    – gnat
    Commented Nov 28, 2012 at 7:30
  • 19
    @GeoffDalgas You must not review much on a mobile device. I click the wrong arrow very, very, very frequently.
    – Austin Henley
    Commented Nov 28, 2012 at 7:30
  • 10
    There's nothing being done with these audits at the moment as far as negative consequences. It's only meant as an aspect of teaching people to do the right thing such as not upvoting an obviously wrong post which was the #1 complaint for review thus far. It gives us an opportunity to inform those who make an impulsive or misclicked action on an audit task about the "right" thing.
    – Geoff Dalgas
    Commented Nov 28, 2012 at 7:35
  • 7
    @GeoffDalgas see above. If you want to teach, pick an appropriate moment for that. Current design rather fuels confusion than teaches.
    – gnat
    Commented Nov 28, 2012 at 7:39
  • 6
    @GeoffDalgas It just doesn't seem very useful to me since it isn't accurate (and for example, it prompted this post instead of informing me anything) Why does it seem like you are content with how it works instead of improving it using the things brought up here?
    – Austin Henley
    Commented Nov 28, 2012 at 7:40
  • @gnat the problem is that a single person could waste a lot of time performing all of the actions possible for an audit. You could upvote, comment, flag, downvote, close, delete, or edit a post. These actions could be a complete waste of time for audit purposes since they won't actually be applied to a post.
    – Geoff Dalgas
    Commented Nov 28, 2012 at 7:47
  • 8
    @GeoffDalgas if a proper audit can not be performed at the moment of clicking I'm done, consider dropping that audit completely. As I mentioned, this can really do less harm than jeopardizing it the way it is now
    – gnat
    Commented Nov 28, 2012 at 7:48
  • 2
    @GeoffDalgas I think that is a cost of an audit system like this, if it continues to exist. To the reviewer, it isn't a big deal because they could put in all that work just for the post to be deleted. It makes no difference from the reviewer's perspective.
    – Austin Henley
    Commented Nov 28, 2012 at 7:49
  • 5
    If the reviewer passes an audit, is it necessary to even inform them? Let them think it was real and add 1 to their review count :)
    – Austin Henley
    Commented Nov 28, 2012 at 7:51
  • 4
    @GeoffDalgas Informing them when they fail isn't enough? Why should it affect real reviewers.
    – Austin Henley
    Commented Nov 28, 2012 at 7:53
  • 6
    @GeoffDalgas I think that is invalid since it falsely tells me I am doing something wrong.
    – Austin Henley
    Commented Nov 28, 2012 at 8:02
  • 5
    @AustinHenley audit tasks are currently working in providing the right guidance to those who make an impulsive call on a post based on the data we have collected thus far. If you don't happen to select the correct action initially on an audit task - don't worry! We aren't doing anything with this data. The thing we are trying to combat at the moment is mindlessly upvoting posts in review to do nothing more than increase review stats which are harmful to review system.
    – Geoff Dalgas
    Commented Nov 28, 2012 at 8:12
  • 9
    @GeoffDalgas I think you're out of date now. I can't agree with "the problem is that a single person could waste a lot of time performing all of the actions possible" because there just isn't much in the review queues (apart from close vote queue) for it to matter. It is not urgent that reviewers review a real post any more, it is urgent that reviewers review well. Please test the full review, not the (perhaps accidental) first click.
    – AndrewC
    Commented Jan 31, 2013 at 9:59
4

While this sounds like a lot of work, it really only takes a second longer and ensures you will never fail an audit.

We're implementing both automated and manual checks for this. Remember: all review actions are tracked, so fooling the simple automated checks isn't going to work for long. And if we find folks going out of their way to abuse this, chances are they'll be blocked from /review for their troubles.

Just try and "do the right thing" and you won't have anything to worry about.

Improve this answer
12
  • 11
    I am not worried. I was notifying everyone of the bug and the broken audit mechanism.
    – Austin Henley
    Commented Nov 28, 2012 at 7:06
  • 2
    So the honeypots will always be on opposite extremes, either really good or really smelly? The only thing that comes to mind is a post where someone might change their vote after reading again. I'm sure there's a limit to how specific you can be, though.
    – user50049
    Commented Nov 28, 2012 at 7:06
  • 1
    The intent is to also not waste a reviewers time with reviewing in detail any post that is an audit. If an audit is being performed make it as swift as possible, give guidance, and move on to something more important @AustinHenley.
    – Geoff Dalgas
    Commented Nov 28, 2012 at 7:10
  • 7
    @GeoffDalgas False guidance, proven by my scenario in this post. It needs to be displayed exactly as if it wasn't an audit: comments should have up arrows and anything else that is missing.
    – Austin Henley
    Commented Nov 28, 2012 at 7:12
  • 3
    Comments are too much of a variable for audit tasks - don't expect they will be there in future audit tasks @AustinHenley.
    – Geoff Dalgas
    Commented Nov 28, 2012 at 7:22
  • 5
    I'd rather not get "tripped up" during review when I downvote a highly positive post. Upvotes do not necessarily mean correctness or quality.
    – user7116
    Commented Nov 28, 2012 at 19:45
  • 20
    I think I would be much more upset if an audit task told me I was doing the wrong thing when I knew I wasn't then for me to "waste" time performing a quality review on a fake item. I don't consider it time wasted; I consider it time spent informing the system of my value as a reviewer. If every other review was an audit maybe it'd be a problem, but as long as it's not too too often it's most likely a cost worth paying. Now if the audit stopped me early to say, "clearly you're on the right track, you can stop as this isn't real." that's fine. The problem is stopping early to say you're wrong.
    – Servy
    Commented Nov 28, 2012 at 19:54
  • 6
    High-quality posts are a LOT harder to agree upon than low-quality ones. I don't see how you can do something like that without a lot more false positives.
    – Justin Morgan
    Commented Nov 28, 2012 at 22:19
  • @TheGuyWhoDisagreesWithYou Well, you could build in an acceptable fail rate. It might be acceptable to downvote 5-10% of posts with a score of 5+ and a checkmark, but if you have a 40% downvote rate on such posts something might be up. As long as a reasonable margin of error has no negative consequences I'll reserve judgement until specifics are given.
    – Servy
    Commented Nov 28, 2012 at 22:35
  • 6
    Review is no longer urgent with so many empty queues. Wasting reviewers time is not an issue. Quality of review is now the main issue.
    – AndrewC
    Commented Jan 31, 2013 at 10:02
  • 6
    @GeoffDalgas unfortunately "trying to do the right thing" still leads to bans from reviewing even if audits are faulty. I "tried to do the right thing" today and got banned. The system is inherently faulty because it is so black and white.
    – CDub
    Commented Dec 6, 2013 at 1:07
  • 3
    I also "tried to do the right thing" today and got banned. If no one is worrying that people trying to do the right thing are getting punished, something is wrong. If such people refuse to perform reviews in the future, the system contributes to lower performance.
    – Noctis Skytower
    Commented Jun 9, 2014 at 14:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .