76

IMHO, it is quite difficult to do First Answers reviews on a daily basis without getting suspended after a few days.

For sure, not all of my reviews were correct and some of the failures and suspensions were absolutely reasonable.

Still, I'm trying to do my best and make "good" reviews and not just click "Looks OK" and go on. I do really think, that I have learned my lessons from the past suspensions, which are always getting longer. Because of "fear" of suspension, I even click "Skip" whenever I have a small doubt about my reaction.

But at all, I can't really comprehend my newest suspension because of these two reviews:

Both answers were - in my opinion - not "perfect", but suitable and OK. Meanwhile, I see some comments about plagiarism there, which of course explain the answers being inadequate. But I wonder how I should have checked this beforehand. If an answer looks helpful to me, how should I make sure it isn't stolen from somewhere? (Of course, I could do a web search for it - but this can't be the intended solution, can it?)

So, to sum it up, this "question" is to address a few "global" and a few "personal" issues:

  • How can we achieve that people trying to do good reviews - especially if it's some dozens per day - don't get suspended for reasons they cannot know of beforehand or opinion-based decisions? (Perhaps one could take into account the number of "correct" reviews.)
  • How should one be able to detect plagiarism when looking at an answer (if it is not really, really obvious)?
  • And for my personal case: What could I have done to not get suspended for these two tasks?

Please don't get my post wrong - it's not intended as a complaint or to express that I would feel being treated unfairly. I have faith in the Stack Overflow community and really want to contribute/give something back, but am a little frustrated because of the suspensions. I just want to understand what was going on here and how both the "logic" and I personally could improve here.

37
  • 10
    Related / possible dupe targets: Is a plagiarism audit justified?, Are reviewers expected to research for internet-wide plagiarism in the First Answers review queue?, How to detect plagiarism when reviewing audit answers?
    – 41686d6564
    Commented Jun 18, 2022 at 11:36
  • 14
    “ Because of "fear" of suspension, I even click "Skip" whenever I have a small doubt about my reaction.” seems like it’s working then - isn’t that kind of the point of audits and skipping?
    – user438383
    Commented Jun 18, 2022 at 13:16
  • 130
    I did the alternative to skip - I just stopped reviewing!
    – Nigel Ren
    Commented Jun 18, 2022 at 13:19
  • 8
    i find the system wrong, to get banned. it should be only reserved to to serial bad choice makers. as nobody listened to me, i make like Nigel Ren, i stopped reviewing. in rare cases i stumble about en edit tat was made and i judge that, but the number is countable on two hands.
    – nbk
    Commented Jun 18, 2022 at 13:31
  • 7
    As someone who got suspended from SO for a week after downvoting crappy questions with no effort. So changed. Then deemed to have been deliberately invalidly mass voting to close them despite the fact id say 80+% are answered by the manual. The fear is real and having been told off for getting some reviews wrong in the past I also stopped those too. It does feel its hard to do the right thing. We want quality questions but in short if you do too much to try make it better you’re bad. (I expect to be thrown out for saying so)
    – BugFinder
    Commented Jun 18, 2022 at 14:26
  • 11
    "IMHO, it is quite difficult to do reviews on a daily basis without getting suspended after a few days." - I process hundreds of reviews every week. I don't have a problem being suspended. However, there is one thing I don't do, I don't indicate that a contribution is "ok" when it's already been deleted. Do yourself a favor and open the question in another tab. You will stop failing obvious audits by doing that.
    – Security Hound
    Commented Jun 18, 2022 at 16:26
  • 44
    I unsuspended you from review. I don't think either of those had any red flags to check for plagiarism. I wish that queue wouldn't use mod-deleted answers like that. They don't reliably make good audits. Maybe it should be limited to ones with a helpful NAA flag, or something like that.
    – Ryan M Mod
    Commented Jun 18, 2022 at 16:38
  • 46
    @BugFinder you're really misrepresenting what happened. First of all, you weren't suspended for the downvoting, which was on 70 percent of the questions in a specific tag over a long period of time, and unprecedented in scale. Second, you were suspended for blatantly invalid close votes. Among other things, you voted "Seeking recommendations for books, tools, software libraries, and more: This question is likely to lead to opinion-based answers." for a number of debugging and how-to questions. It wasn't a close call.
    – Ryan M Mod
    Commented Jun 18, 2022 at 17:11
  • 19
    "80+% are answered by the manual" - that isn't a reason to close a question, let alone delete it.
    – kaya3
    Commented Jun 18, 2022 at 17:17
  • 12
    @BugFinder "I no longer feel able to point things out" ... I have a really hard time reading Ryans comment as "you're not supposed to comment". That there isn't a RTFM close reason is a very obvious indicator, that those questions aren't off-topic. You're still free to downvote and leave a comment. Misusing the close option for that isn't an option, though.
    – Tom
    Commented Jun 18, 2022 at 17:24
  • 12
    If the questions are answered by the manual then either (1) close them as a duplicate of an existing question that asks the same thing, or (2) write an answer quoting the manual. It's not hard to do the right things on SO, it's just that the things you want to do - and are choosing to do - aren't the right things.
    – kaya3
    Commented Jun 18, 2022 at 17:28
  • 23
    @user4581301 No, we need two things; we need a slightly better audit system where mods or possibly even trusted users (20k privilege, not some weird abstract definition; or lower, whatever makes sense) contribute to what goes in the audit pool, and for plagiarism specifically, we need automatic tooling. We've already posted a request internally asking for such tooling, but whether we get that in the near foreseeable future is not looking promising (in my biased opinion). Only mods have the necessary tools to deal with users who repeatedly plagiarise, so a queue for it doesn't make sense
    – Zoe - Save the data dump Mod
    Commented Jun 18, 2022 at 19:44
  • 8
    We shouldn't have to throw reviewers at this particular problem, when this is one of the few areas that could be done by a machine with an auto-flag. Plagiarism checking is even a mostly solved problem, and there's a number of companies offering it as a service. Why we don't have one integrated is something I'll never understand. The moderation system was unfortunately never intended for the volume of content Stack Overflow gets, though, and that problem isn't going anywhere, regardless of how many times we say it. SE needs to get more involved in this, but they don't, so here we are
    – Zoe - Save the data dump Mod
    Commented Jun 18, 2022 at 20:02
  • 6
    @ThierryLathuille for the record, when an active audit is displayed (read: before it's completed), the comments are often hidden. The comments being there doesn't mean they're useful from within the audit, and going out of the queue to check them alone is enough to confirm it's an audit (because the post is deleted)
    – Zoe - Save the data dump Mod
    Commented Jun 19, 2022 at 8:35
  • 12
    IMO - Just do NOT review anything at all on SO. No point taking stress for no reason and getting suspended too. I used to do it but not anymore. I don't see a point.
    – Always Helping
    Commented Jun 19, 2022 at 22:02

3 Answers 3

Reset to default
62

Those reviews are sensible and I'm not sure if the first one was even plagiarism since the link to the (open?) source was provided. When it comes to plagiarism: if you can find it, great. If you can't, you should at least not get punished for it. In particular, expect plagiarism when doing tag wiki suggested edit reviews.

Anyway, this post and literally countless before it keep showing that posts deleted by diamond moderators are not suitable for review audits! Very often diamond mods have much more insight into an issue than regular reviewers when they chose to delete a post. We shouldn't need to go to the actual post to check "huh is this an audit" every single time, that makes reviewing even slower and more tedious.

The solution to this problem is to boycott reviews until the audits are fixed, because nothing short of a complete disaster will make SO the company interested in improving the audit system. Let them hire paid staff to do the reviews if they can't be bothered to fix the audits.

13
  • 23
    Oh and if you want to change SO, don't forget to post non-constructive whining on Twitter rather than constructive feature requests on meta.so.
    – Lundin
    Commented Jun 20, 2022 at 6:51
  • 5
    "Anyway, this post and literally countless before it keep showing that posts deleted by diamond moderators are not suitable for review audits!" - eh, we also delete huge quantities of bog-standard NAAs, many of which are prime audit candidates
    – Zoe - Save the data dump Mod
    Commented Jun 20, 2022 at 7:47
  • 20
    The solution to this problem is to boycott reviews until the audits are fixed That's a excellent way to put it. Willingness to review under any circumstances on part of volunteers is part of the problem. Probably less than 1% users clear up most of the review queues. I believe this results in lack of diversity and very low quality reviews by overburdened volunteers.
    – TheMaster
    Commented Jun 20, 2022 at 8:03
  • 16
    @ZoestandswithUkraine So maybe give the mods an option to say "not suitable for audit" upon deletion. Since the kind of deletes that aren't suitable are often special cases: spammer/sock pup accounts etc.
    – Lundin
    Commented Jun 20, 2022 at 8:18
  • 1
    Re "...if the first one was even plagiarism since the link to the (open?) source was provided": Related is Answers entirely copied though properly attributed.
    – Peter Mortensen
    Commented Jun 20, 2022 at 10:01
  • 5
    "I'm not sure if the first one was even plagiarism since the link to the (open?) source was provided" It was plagiarism, for two reasons: 1) There's zero indication that the link is the source. Burying a random link that happens to have the source somewhere in the post is not attribution. 2) It's not even the right page. The code was copied from the example page, not the page they linked.
    – Ryan M Mod
    Commented Jun 20, 2022 at 11:13
  • @Lundin "So maybe give the mods an option to say "not suitable for audit"" - you are absolutely right, but it's essentially pointless to suggest it. What you suggest essentially means overhauling the current audit system and then especially how audits are picked. Good luck with that. No, it is terrible and it is going to stay terrible. the only way forward is whatever can be done to persuade people to just chew through it. Suck it up.
    – Gimby
    Commented Jun 20, 2022 at 12:19
  • Yep, I also had "mixed Feelings" about the 1st Answer, with Link provided where it was easy to find the Source of the Code. Not enough Code => gets flagged as "Link Only Answer"; too much Code => Plagiarism, ah-ah...! This Answer could have been salvaged... // The 2nd one was obvious "Fake Content" to me and has nothing to do with "Plagiarism", this is how "Fake Content" works... I've already (custom) flagged some (much more obvious) Fake Content, Flag(s) declined (because I didn't use the "magic" "P"-Word), this one got deleted by a Mod, weird to see it then being reused for Audits...
    – chivracq
    Commented Jun 20, 2022 at 12:54
  • 1
    Side note: Getting rid of the Mods is not a viable solution to the problem of reviewing complicated Mod-deleted posts.
    – user4581301
    Commented Jun 20, 2022 at 16:45
  • 2
    @user4581301 Did someone suggest getting rid of moderators?!
    – TylerH
    Commented Jun 20, 2022 at 17:56
  • 1
    @TylerH Someone will. I'm just trying to head that absurdity off at the pass.
    – user4581301
    Commented Jun 20, 2022 at 18:12
  • @Lundin I'm not a fan of adding another button or checkbox or whatever. SE is already busy throwing keybinds out the window; the new popups are not at all keyboard friendly, with support ranging from mediocre to non-existent. We do a lot of deletions; adding another click is a substantial slowdown at the scale we operate. I would still like to see mods and users have more of a say, but a button to remove any post from the audit pool is half the job. That said, a button like that is after the fact, at which point, the damage has already been done
    – Zoe - Save the data dump Mod
    Commented Jun 21, 2022 at 7:06
  • 1
    @ZoestandswithUkraine The default option would be "suitable for audit", I would imagine. But diamond moderators won't get less work if user moderators get review banned and fed up with reviews because of the broken audits.
    – Lundin
    Commented Jun 21, 2022 at 10:01
6

I've posted a feature request with one possible solution to this problem:

Don't use answers with a "Very low quality" flag on them as review audits

I believe this would address the situation, since it would avoid at least the vast majority of answers that are deleted by moderators for non-obvious reasons.

5

Both failed review audits you linked in your question had an indicator that something more than a "trivial" glance at the post in the queue itself was required. I'm referring to the "X answers" information posted near the left, just under the "Review the following answer" header.

In the first case, this showed 16 answers. Now, although I can't categorically say that you should be searching the entire internet to check for plagiarism1, in the case of a new answer posted when there are so many others, then a view of those other answers is certainly required, to check that the new answer is unique, and not copied from one of those. Although viewing those other answers from the review itself (middle tab) would maybe not have helped in that case, opening up the post in a new window (by clicking on the post title) would have immediately revealed the audit for what it was.

In the second case, the indicator showed 0 answers. This should have rung one or more alarm bells – and, again, opening up the post in a new window would have resolved that issue.

Now, while viewing each and every post you review in its own right (that is, not just from the review queue) is not something I would necessarily suggest as a matter of course, it is a technique you should use in any case where something looks 'odd', in terms of the information presented. For all audits, there will be a deliberate 'lie' in one or more of the statistics presented for the post; in fact, that is how (I think) user-scripts that detect audits work – but I'm not suggesting you use such a script to help dodge audits. (In fact, I think such scripts should not be allowed, at all.)

1 But note: For those reviewers with 5k or more reputation points, such checking for "internet-wide" plagiarism is an absolute requirement when reviewing suggested edits to tag-Wikis and their excerpts. See, for example: Let's stop tag wiki plagiarism. The plagiarism issue for First and Late Answers is a bit more subjective, but a really good reviewer will do at least a rudimentary check, like a quick Google search.

16
  • 29
    A question having several answers can increase the chance that a new answer is just a copy of an old one, yes, but I really don't see how both "alarm bells" would have helped in detecting plagiarism from a third-party-site. So this answers boils down to just "open the actual question in a new tab" and this is a good advice for every review, not just one with many answers or none.
    – Tom
    Commented Jun 18, 2022 at 17:29
  • 7
    the second can't raise any bells, it looks perfectly fine and tries to answer the question and is well formulated, i wouldn't also google the text, that is not what the review is for
    – nbk
    Commented Jun 18, 2022 at 18:55
  • 3
    @nbk The alarm bell in the second case is that the information provided indicated that there was no answer. This was a review for an answer, so something fishy is happening. Opening up the post in its own window would have revealed the audit.
    – Adrian Mole
    Commented Jun 18, 2022 at 18:57
  • 4
    the review should help the mods and admins and filter many thinks out and leave only the "bad" to be reviewed by them, and as i many many times stated, we can give only our opinion about a question or answer, a ban is far to harsh for time spend
    – nbk
    Commented Jun 18, 2022 at 19:04
  • 9
    @nbk Yes, I think this review suspension was unfair and unfortunate. Ryan has now lifted the suspension, so that's good. However, the OP also asked for advice/tips on how to avoid these audit-imposed suspensions, so I gave some.
    – Adrian Mole
    Commented Jun 18, 2022 at 19:16
  • 4
    Also worth noting that suggested edits is the only queue where plagiarism checking is a hard requirement. It isn't for whichever queue this was again. It was an audit fail to begin with, and as is well-known, the audit system isn't great at picking review tasks for audits. Blame SE for that one. Outside suggested edits, audits are a mixed barrel. In fact, all queues that aren't suggested edits regularly have bad audits that are outright wrong (notably in the close vote queue), or that are poorly picked (particularly all the answer related ones)
    – Zoe - Save the data dump Mod
    Commented Jun 18, 2022 at 19:29
  • 7
    Of course, it'd be nice if reviewers took the time to check answers for plagiarism, at least until we get some automatic plagiarism tooling so we don't have to waste as much manual work on it, but unless SE steamrolls us, plagiarism checking in review will remain a nice bonus, and not a hard requirement
    – Zoe - Save the data dump Mod
    Commented Jun 18, 2022 at 19:31
  • 1
    As the review also contains a comment (by a mod!) that the answer was plagiarized, I think that is also a sign you should do some more work to verify it, or just skip.
    – Mark Rotteveel
    Commented Jun 19, 2022 at 7:35
  • Thank you, @AdrianMole, for your answer! I agree that the comments were "red flags" and - as stated above - can't remember whether they were displayed when I made my decisions. Nevertheless, I also agree with Tom and nbk that the other alarm bells for both the answers seem a little questionable or - at least - very well hidden to me. Anyway, in the future I'll make use of the suggestion given by you and others and review the question on its own, hopefully this will minimize my amount of failures.
    – ahuemmer
    Commented Jun 19, 2022 at 8:42
  • @Mark I'm not 100% sure but I'm fairly certain that comments aren't displayed in the actual audit for "known bad" posts.
    – Adrian Mole
    Commented Jun 19, 2022 at 8:59
  • 3
    @MarkRotteveel repeating my comment on the question, the comments may not have been there during the review. Comments are often (or always, I lose track) hidden in audits, so it being there now doesn't mean it was when OP reviewed the audit. Quite a few detail are hidden or randomized in an audit, and comments being hidden is one of those. The point being, context you see now doesn't necessarily help the reviewer during the actual audit.
    – Zoe - Save the data dump Mod
    Commented Jun 19, 2022 at 9:00
  • 2
    @Mark If you mean the links to the 'past' review as presented in this question, then no. They show what would have been shown to the reviewer after the audit.
    – Adrian Mole
    Commented Jun 19, 2022 at 9:08
  • 3
    First post audits are there to check for spam, crap, confused beginner users etc. Checking for plagiarism isn't expected unless there are very good reasons to suspect it. These very simply very bad audits.
    – Lundin
    Commented Jun 20, 2022 at 6:56
  • 6
    @AdrianMole Reviewing should not include having to "spot the audit", nor should it be necessary to do secondary research. If you need to spot the audit in order to reliably pass it, then the audit is not doing its job. Secondly, volunteers wade through seas of questions and answers (many of which are probably mediocre and worse, or they wouldn't be in the queue), so the goal should be to provide them - at a glance - with all the information they need.
    – CharonX
    Commented Jun 20, 2022 at 13:35
  • 4
    @CharonX I entirely agree! In an ideal world, audits should be passed/failed on the reviewers' assessments of the post content, and nothing more. However, the audit system is ... um ... not ideal. So, until or unless the system is fixed, I occasionally proffer some "tips" to help prevent otherwise decent/conscientious reviewers from being suspended from making their valuable curation efforts by a deeply flawed system.
    – Adrian Mole
    Commented Jun 20, 2022 at 13:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .