I think the foundation of the problem is incentivizing reviewing in the first place.

We incentivize asking questions. We incentivize answering them. We incentivize making questions better by editing them (for people with lower rep). However, all of these incentive processes have review.

You only get rep from questions when people upvote them. You only get rep from answers when people upvote them. You only get rep from edits when people approve them. In every case, some human being has to look at what you did and say, "Good".

Reviews don't work that way. Nobody approves a review. You get something that needs review, you review it, and you take an action. You get incentive points regardless of what action you take (as long as you take some action).

If you take away upvotes as a "reviewing" action, people will just make inconsequential edits. The system is too easy to game because there is zero oversight; as long as you do something, anything, to the post, you get a point.

This is why the only people who get rep for editing questions are those who have to have their edits approved.

As long as there's a shiny gold badge in it, someone will grind out reviews just to get it. Take away the incentive, and many people may stop reviewing altogether. Perhaps the latter might be a better option; at least then, you're getting people who actually care about reviewing involved.

Simply letting several people review the same item, like with suggested edits, is not really "simply" anymore, and introduces more review-point inflation still.

I think this is a tangible solution to at least some of the problems you have described. At least 25–40% of the time tonight, I have gotten in "foot races" where, like you, my edit was accepted, but the post was already marked as "reviewed" by the time I was done. My differences with the "upvote or no review" system aside, this method of refereeing multiple reviews is just encouraging rushed or avoided edits on posts that really need them for, if nothing else, the instructional value to the new user.

Failing a multiple user review, having each review post designated to only the first user to view that post until relinquished by a "not sure" click would be the most fair, assuming this would be feasible.

Q&A works so well because posts can be voted, commented, rolled back, deleted, etc by the community. /review fails because reviews cannot be voted, commented, rolled back, deleted, etc by the community, let alone the moderators. So, there has got to be some system for that, or we have to remove any incentive for /review such as badges and reputation.

To start, I'd stop giving every monkey with 2K instant access to /review and start with a moderator-controlled system. Most straightforward example would be some kind of an invite-only system wherein each user — starting with moderators — can invite/propose ~10 other users for access to the review system. Moderators should be able grant/deny access based on the activity of the invited/proposed person. With this we can start with better quality reviews and thus end up getting better reviewers. (striked 6 march 2015; I imagined this would be too much manual work)

To start, I'd stop with a reputation based system and switch to a flag ratio based system for access to /review. Say, only allow users with a minimum of X-amount of flags (perhaps relative to the total number of flags on the platform) of which an Y-amount has been approved, access to /review. With this we can start with better quality reviews and thus end up getting better reviewers.

The alternative is to remove the incentive for /review. Ones who are really willing to cleanup the site the right way would generally not care about badges and reputation for that anyway (like as in Wikipedia). Stats should however be kept visible in user profiles, that would be tremendously helpful for among others future moderator elections.

Just a quick status update: we are taking this seriously, but don't want to jump to conclusions (or make rash changes unless they can be demonstrated to help).

• As Geoff mentioned previously, we've fixed the bug whereby if two reviewers review the same item only one gets credit for it (realistically, this only affected some types of reviews, but editing was one of them so that's particularly bad).

• Emmett has implemented some better analysis tooling for these queues, which should let us get a better idea of the scope and extent of the problem (once they're enabled).

update 2012-12-13

Manually suspending reviewing privileges has been introduced:

...for folks who fail multiple review audits in a short time...

update 2013-01-29

Automatic review suspension has been introduced:

...We're kicking blatant abusers out of the queues automatically now...

A radical solution:

Either add a "Looks Good" button that increments the review count or make the "Not Sure" button increment the review count, but not count to removing the post from the review queues for anyone else. There needs to be a way to let people say:

I've reviewed this item, but I don't think it requires any attention from me.

The fact that "Not Sure" doesn't increment the review count discourages people from clicking that perfectly valid option.

That way people won't be tempted to vote up just to get the increment in the view count.

We'd have to change the requirement for the badges though - a simple increase in the numbers (500 and 2000 for silver and gold) might be enough, or have the requirement be that you must have > 50% of real actions (edits, etc.).

Reviews could be made flaggable, and moderators could remove the review capability from a user for a while. Surely not ideal but it could educate some users.

Addendum: We cannot see later what a reviewer actually did (at least, for now), so this will probably not work. See ↓ the comments.

I think one aspect of the problem that could be addressed is "reviews per hour".

Good reviews take time. Reviews that are gaming the system do not.

Your review that you took time and effort for was undercut by a review of someone who could have been gaming for a badge, not willing or able to put the required time in, or not competent enough to create a review (despite accruing 2k+ rep).

Limiting the rate at which people can perform reviews won't necessarily fix how good the reviews will be, but it will prevent the damage that can be done by gamers sprinting towards a badge rather than performing a proper review for the good of the community.

This could be fixed by letting users know they can only perform 5/10/n reviews per hour, potentially by review type, and informing them when they reach that limit, as they are going too fast to be likely to be making a positive impact.

Edit:

Based on S.Albano's comment below we could use a dynamic rate to protect review tasks when the queues become smaller than the allowed rate, e.g.

int ReviewsPerHour = Math.Min(10, Math.Round(reviewTaskCountInQueue/10));
ReviewsPerHour = Math.Max(ReviewsPerHour, 1);

Which would hopefully protect 90% of the review tasks.

I have to agree with Nicol Bolas's answer that there is a fundamental problem here of incentivising users to review content without any oversight of the reviews. (I'll come back to this later.)

There is also a very interesting comment raised by Ben Broka

I'd like some actual evidence that reviews are doing more harm than good if we're going to pretend they're a problem.

I think this is worth discussing. So, let's assume (just for the sake of argument) that a significant number of people are going through the review queues, upvoting an item without even reading it, and then clicking "I'm Done". How would that be any different then them doing nothing at all.

Well, there are only two changes that take place:

1. The post now has an upvote on it. If it's a great post, this is correct; if it's a mediocre post, then it's not a problem; if it's a terrible post, then it's doing some harm, but not a huge amount because it can only add at most one vote (from the review queue) per post. Adding a single vote is somewhat unlikely to push a poor answer above better answers. It's possible, yes, but we're at the point where very few posts fall into this category. So, net effect: close to zero. I imagine the positives will be comparable to the negatives.
2. The item is removed from the review queue. It is unlikely to be seen by someone who is capable of making more substantial actions such as some combination of flagging, editing, voting to close, commenting, downvoting, etc. as appropriate. Here the issue is in the opportunity cost. The poor review has prevented more positive actions from taking place, or make it harder for those looking to do proper reviews from finding content that is worth their time. This of course only happens if the queue is emptied. As long as there are items in the queue to review for the legitimate reviewers, their time isn't wasted.
3. A reviewer is getting credit for reviews while not actually reviewing content; this reward is potentially in the form of badges or a place on the (daily or total) leaderboard. Now, the value of those rewards is somewhat diminished because so many people know that it is often undeserved, but clearly there is enough "value" in the reward for people to spend time faking reviews.

I also want to address the fact that the discussions around suggested edits have been brought up as being related. There are a few key differences between suggested edits and these other review queues.

If an item never gets reviewed in the first/late posts it's not stopping any action. Having an item sit in the queue for years without being looked at simply means it might not possibly be fixed by someone if something is wrong. There's nothing concrete there. Suggested edits on the other hand cause problems when they sit in the queue. When an edit is in the queue it means that the post can't be edited by other users (if they would want to make more substantial edits, or fix things that the edit missed, or just add more info in the case of the OP). It also means (if the edit adds value) that the positive edits aren't being seen by everyone viewing the post. That's a problem. Finally, to prevent that queue from getting excessively long, it is limited in size. If nobody is reviewing suggested edits the capacity will be reached (the old system was frequently hitting capacity on SO several days a week, or at least sitting close to it). When the queue is at capacity new suggested edits (presumably of value) can't be made. Because of all of these reasons there is a legitimate argument to make that value of approving suggested edits quickly is worth more than the cost of having some bad suggested edits approved (ideally we'd want fast and quality approvals, but we haven't figured that one out yet).

First/late posts are different though, because there is substantially less value in reviewing content early (most of it really can wait) it doesn't offset the cost associated with bad reviews. As such it has been positive to add more incentives for the suggested edit queue (although I still think it could be improved further) while adding incentives to the first/last posts are detrimental.

So, what exactly should we do?

Just to explicitly state it, I suggest that both the gold an silver badges for review queues be removed. The bronze badge is fine for introducing users to the queue and is set low enough to not encourage significant abuse. The leaderboards (both daily and total) for all queues should also be eliminated. There are users who will place being on those boards as goals, and let the quality of their reviews suffer so that they can get/maintain positions there. Removing both of those features should take away the most significant incentives for users to try to get review points without regard for the quality of the reviews themselves.

</WallOfText>

A partial solution (well, not "solution", that seems too strong; maybe "medicine"?) would be to remove the post from the queue as soon as someone starts editing it. That would at least prevent those moments of sheer frustration, where you finally finish injecting some sense into an atrocity committed against English grammar, only to find that someone has pulled the rug from under your feet and taken your credit without so much as making a token effort. (If the edit is abandoned, then naturally the post goes back in the queue.)

The other part might be to go ahead and count all review-type actions, even if they're on posts someone else has already reviewed; heck, even if they originated from someplace other than the queue. Naturally, the badge limits would have to be greatly increased. Also, I don't think votes should count as review-type actions: I'd limit it to edits, flags, and comments.

I'm starting to think that having badges (particularly gold!) for something that can be so easily gamed, might not be a good idea after all.

Whatever criteria a badge has, there will be people gaming the system and taking the path of least effort to get that badge. In this case, though, there's so much gaming of the system that any countermeasure that requires peer consensus seems doomed to fail. All the gamers will just do the same thing, and since they all "agree", bad reviews now look legit.

I'm about this close to recommending removal of the gold review badges entirely. I'd gladly give up mine, if it means we have fewer crap posts getting rubber-stamped by badge whores. (There will still be some gaming in the short term, but something tells me it'll taper off once people get their silver badges.) The bronze badge can stay, and probably the silver, but as long as there's a gold badge, people are going to farm it any way they can.

How about taking a "test" for each type of review queue? In order to get full privileges to each of the review queues you'd have to go through a vetting process. Here's what that could look like.

1. A new user wanting to participate in the review queues would be presented with posts that have a clear "right" answer in terms of reviewing. The user would have to review 100% of these "test" reviews correctly in order to get access to the real queue. You would have to go through this for each queue.

2. Just because a user gets past the initial test for a queue doesn't mean you stop throwing in "test" reviews. For example, when you open the floodgates to them, keep showing them "test" reviews. If a user gets enough of these wrong at any stage, temporarily revoke their access to that particular queue (or warn them or something). Alternatively bump them down a "level" (See below).

3. As the user proves themselves by answering more test reviews correctly, decrement the number of test reviews they have to get correct.

4. Provide other reviewers a way of flagging people that are not reviewing things correctly. Enough flags could increase the number of test reviews a person is shown (this could be an automated process or perhaps moderators would have to be involved).

You could break the process up into "levels." Obviously these are flexible:

• Level 0 reviewer: 100% test reviews
• Level 1 reviewer: 30% test reviews and 70% real reviews
• Level 2 reviewer: 20% test reviews and 80% real reviews
• Level 3 reviewer: 10% test reviews and 90% real reviews
• Level 4 reviewer: 5% test reviews and 95% real reviews

You could display the "level" of the reviewer on their profile, thereby still gamifying the review process but making said gamification a little more useful. You also would not tell them which reviews were "test reviews." So they could, at any point, lose their "Level 4 reviewer" status and be bumped back down. This would prevent people doing the right thing up until they reach Level 4 and then just reverting back to their badge-grinding ways.

Other notes:

• When a user gets a "test" review wrong, let them know why and point out exactly what's wrong with the post that they missed. That way we're hopefully educating users while we vet them.
• I know I'm glossing over a few problems like "how to determine canonical reviews" for each queue.

An additional option might be to only provide the badge incentive when there is a backlog, thereby not rewarding undesirable behavior.

Here is the use case:

1. The review queue hits a certain threshold (say 500), triggering reviews to be counted towards the badge incentives.

2. Some indicator in the user interface indicates that the review queue is in that mode.

3. Additional reviewers are attracted by the reward, while there is no apparent race with other reviewers to reduce the quality of review.

4. The review queue hits a certain low threshold (say 80), triggering reviews to not be counted towards the badge incentives.

This scheme has several benefits:

1. It calls in backup for the review process when needed.

2. It makes it less likely that upvote reviewers will trample on the well thought-out edits of another.

3. It provides a cushion between the high and low thresholds, so that the time during which the queue rewards cheating is minimized.

4. It should select for the more desirable edit action both during both phases thanks to the rep. and diminished sense of urgency.

5. It provides lull time for users who's only reward is to improve the site to do so in peace.

EDIT #4:

Okay. This edit really does cinch a few things wrong with the system. Is there progress on removing poor reviewers from the system? It doesn't feel like there is any, yet.

Please. Start removing the bad reviewers from the system. I'm seriously questioning my participation in the review process as a whole, if edits can be blindly approved like this. This has got to stop.

EDIT #3:

This is getting painful. I'm starting to seriously rethink this whole review queue thing. The very fact that this edit was not immediately rejected by all participating reviewers indicates that the reviewer(s) that decided to accept the revision just doesn't care.

These sorts of poor editors have to be removed from the system. And I would hope that this culling is done at a much more accelerated rate than what's going on now. The rate we're going at now just doesn't seem to be cutting down the lack of quality reviews.

EDIT #2:

I'm sorry, but this edit made me about lose my salad here. It's not the spelling or anything like that, but it's the lack of completeness. What are we getting at when we want reviews to be peer-reviewed or editable? We must find a way to get quality from the full review process, not just the (oft empty) review queues.

EDIT:

Take a look at these string of edits. The first few don't address the critical shortcomings of formatting the question, only taking time to make simple changes but leave the critical stuff alone. The need to remove poor editors from the system has never been greater.

So here's my beef with the current system. Here's my recent example; let's try and keep it clean/egoless as I go through this.

So the main thing that I do when approaching a question/answer to edit is gauge how much actually needs to be revised. Is it just the code formatting? Was there some grammar issues that came up? Did the question need a bit of spit and polish to come across cleaner? It's a process that means less edits from me, but (I believe) higher quality.

When I'm reviewing questions/answers, I apply the same rubric I do as if I were editing the post myself. Is there more that can be changed? Was there enough work done revising the question to be useful to the next person?

Let me stop myself right there. The interesting Catch-22 of the edit system is its ultimate goal: The revisions must be substantial enough to benefit someone else reading the question in the future. The current review queue obviously does not enable the community toward that goal, not unless they are disciplined and patient enough to go through each detail of the question and revise it.

But let's be honest - few of us are.

So, the example I have posted up - take a look at revision #3. There wasn't anything of actual value added to that revision - the grammar mistakes went largely unchecked, and code formatting was actually reverted. This was a poor edit, and it was correctly rejected - except for Community coming in and accepting the revision. This is likely due to someone else editing the file and claiming that the edit was "useful".

So, three part plan to fix the system.

1. Enforce, to some degree, a small checklist of what should be fixed in the post. This can include but isn't limited to:

   • Spell checker; catch words that are commonly misspelled and are missed in a review. If a user repeatedly misses these, after some given threshold, restrict their ability to participate in reviews for a while.
   • Code format checker; catch blocks of code that can be interpreted as a particular language, and see how it's formatted. This may be tricky to implement, since there's plenty of holy war on where braces go in Java, and the type of whitespace used in Python examples won't be clearly conveyed.
   • Miscellaneous checker; this can suggest problems such as, "Hey - this post has a signature at the end, that would be one thing to improve." May be tricky to implement if it had a broad range of miscellaneous items to check through.

2. Penalize users that do not sufficiently review the question/answer with negative reputation. This can apply to anyone, as to encourage those without the suggested edit restriction to actually sit down and review the question.

3. Accounts with a history of poor reviews should not be allowed to participate in the system. I can't stress this enough - the only way we'd be able to stop poor reviews from affecting the rest of the site is to stop the poor reviewers from partaking.

Few solutions (mix and match), in three steps:

Filtering for a bad reviewer

Sticky question

Whenever users give opposite votes on the same question in a queue--Make the question a bit stickier in the queue (doesn't disappear immediately). If you have enough conflicting votes, the guys who were upvoting probably were fly-by reviewers.

Consecutive upvotes

If a user has too many consecutive upvotes in too little time, it's quite likely he is a bad reviewer.

Remember that posts which take very little time to go through are many times downvote-able posts. If the user is just zipping though the queue, something's amiss...

Confirming it

Monkey on my back

If a user rarely downvotes in a review queue, and goes through the posts way too fast, attach a "monkey" on their back. This triggers the honeypot posts(or whatever other traps you have) more often. If they answer the honeypots correctly, the monkey goes away.

Community flags

Have Community flag what it thinks are bad reviewers--those who have tripped up the filtering algorithms too many times

Honeypot

Already implemented

Penalizing them

Have warnings shown to them for the first few offences, with no penalizing. If they continue, gradually increase the severity of the punishment. Various (mix and match) ways to penalize:

• Rep penalty
• Block from review for X hours
• Reverse all reviews by the user in the last X hours (this includes resetting their review stats)
• Reverse the last X reviews by the user

Though the simplest one (and least annoying) is this: For every bad review, the user's review progress goes down by 2. (Maybe more, if they have too many bad reviews)

How about a reviewer can only upvote an answer if they have a minimum number of tags earned in that subject? This way all users unfamiliar with a particular language or subject can screen for spam, Not an Answer/Real Question, and other obvious undesirable postings and still be able to leave a helpful comment or fix broken/missing code formatting - But if they haven't demonstrated at least a minimum competence in a topic they haven't earned the credibility to upvote (they can still go back and upvote later on but it wont count as a review).

Personally I know subjects relating to LAMP webservers like PHP, MySQL, JQuery, Apache, etc but I'm not a Ruby or C++ guy, so when I see questions in my area I upvote them if they are good answers and just skip if they aren't anything special. But when a Ruby or C++ question comes up I usually always skip if its not obvious junk needing flagging or commenting because I feel like I have no business upvoting posts on subjects I don't understand, for all I know the poster has no clue what they're doing either and their answer doesn't solve the problem. Upvoting should be done because its a good answer, not because it isn't spam or obvious crap.

Example: My Current Tags

• At a set minimum of 25 Posts, I could only upvote posts tagged with php or htaccess.

• At a set minimum of 25 Upvotes, I could upvote php , svg , gd2 , imagemagick , jpeg.

• Either way I couldn't upvote mysql or regex or the other lower tags until I get my postings in those tags up a little bit and prove other users agree I know a bit about these subjects.

Example of blind upvoting this might address:
How can I get the CheckBoxList selected values, what I have doesn't seem to work C#.NET/VisualWebPart

2 users upvoted this (and another edited it) while I had it open for review, all without any of them catching the fact the original question was about CheckBoxList values and the answerer just pasted some random completely unrelated C# sql connection script. If review upvoting privileges for this question were restricted to the C# crowd then I would hope they would recognize right away the code is unrelated and filter crap like this out.

I realize that restricting review upvotes based on subject experience might make it more difficult to get some of the more obscure topics out of the review queue. Maybe it would have to only apply to the more popular subjects like c# java php that have large user base. Maybe it would have to only apply to answers because its easier to know if a problem sounds legitimate than to know if an answer solves the OP. But it would help prevent users upvoting simply because it's not obvious junk and it earns them a review credit, even though they probably have no clue how good the answer really is!

Incentives should only be given for work. Answering a question takes time and effort. Clicking an upvote button is not sufficient work to earn an incentive. Sure everything needs to be reviewed. But only grant rewards when someone takes the time to edit and improve.

Once an item is shown to a user for review, it should not be shown to another until the first has finished or some substantial timeout has passed.

There is still an incentive to do reviewing. You will skip past items that don't need help quickly until you find one that needs your help. You improve it, you get rewarded.

Possible downside: worthless, unneeded, or even damaging edits made solely for the purpose of getting a review credit.

It seems that there are possibly simple solutions to the two problems:

Problem 1: Flyby reviewers

I think the biggest part of the problem is that people are penalized if they pass. They loose their time investment for no reward. Award points for the pass button, and change the criteria for the badge to be twice as many reviews with no more than 50% pass. This will at least train folks that pass is a good and valid option when you don't actually know. Much better to have pass be the default action than approve, and I'm sure it's not any harder to design a honeypot for flyby use of the pass button than it is for fly by use of the approve button.

I do think failed honeypots should explain what the problems were.

Problem 2: Theft of a review in progress

1. Reserve the review. One does NOT need a lock to do this. Merely update the record with a timestamp and don't give the review to another user for 5 minutes. No need to come back and update, and if the review takes more than 5 minutes, you probably went to make coffee in the middle of it anyway. OR
2. Allow multiple reviews of the same content. poof problem goes away :). Possibly raise the badge counts to compensate.

This sort of follows what Mac suggests, but I've often wondered if there is a filter on the review system that checks to see if someone's reviews are generally in line with what other users are doing. Obviously there will be differences between different reviewers, but I see the opportunity to implement some automatic flags (that at least alert mods to the behavior).

1. Detect if votes (upvotes and close votes) on specific posts are in line with other reviewers on that review. If a reviewer is out of line with the community more than x% of the time (and has more than y total reviews in that category) it triggers a flag.
2. If the speed at which a user goes through reviews exceeds some % above the median speed, it triggers a flag.
3. If a user reliably uses the same mechanism to handle a review (i.e. almost always upvotes, almost always votes to close, etc) in a statistically suggestive way, it triggers a flag.

Users with review flags could either be noted for moderators, and/or

A review of reviews review category could be created, which puts suspect reviews in a pool for review, with similar incentives/badges as currently exist for other review pools (that's right, I just used "review" 6 times in one sentence).

Review Allotment

Another option with precedent on the SE system would be to have a system similar to the flagging system, where a user has only a limited number of "reviews" to spend, but may be entrusted with more for appropriate reviews.

• Each time a user completes a review task that is approved by members of the community, such as a helpful edit, or flag, or a comment that is later upvoted, they are entrusted with a few more reviews. (carrot)

• If an action is rejected by the community, their review allotment is decremented by several reviews. (stick)

• Review tasks that are not approved simply decrement the review allotment available to the user by one. (stick-lite)

• Perhaps have a slow regeneration of allotted reviews to some minimum to give users a second chance.

This system provides both the carrot and the stick, while still keeping the badges. It provides incentive for work, a penalty for bad work, recognizes that an up or down vote is OK in some instances, and makes certain that voting is not the only thing that the user is doing.

You could simply increase the rep required to do reviews when the queue is small.

Lots of good solutions here. Seems that this is likely a complex problem that needs more than one solution - it happens sometimes. So I have two ideas (related to each other) into the "pot" for consideration:

• Throttle the number of edits and reviews a user can do in a given period of time. It's obvious that quality edits take an average of X minutes per Y posts; I'm sure admins and super-users can fill in the best numbers there. So (for example) when someone does 50 edits in an hour, you know there is no way those edits could be good - no matter how fast or smart the editor is. Doing that many, that fast, should either be prevented, subject to scrutiny, and/or flagged for low quality - and badges gained could be revoked if super-users vote for such action. PS - don't quote me on the number/hour - superusers and admins know the math and have great formulas :)
• Throttle number of "review"-type badges/period a user can get.
• 'Lock' the question or answer while it's being reviewed - all other parts of the question/answer are 'open'. This prevents simultaneous and overlapping reviews, edits, and up-votes on the part being reviewed. Since each question, answer, comment, and vote is handled as a separate object on the server, locking each part currently being reviewed shouldn't be a herculean effort. This 'lock' also helps throttle "hurried" users who do the system injustice by skimming through content. It's only part of the solution, but it's important for the overall health of the system.

If people want to do quality reviews, they'll never be impacted by these limitations. The ones that fly through, simply up-voting for badges, are diminishing the experience for all, so their incentive and ability to do so needs to be reduced. These 3 methods, along with some others on this Question, should help restore these badges' value, and disenfranchise those who wish to accumulate points at the expense of quality.

A solution might be to only award reviews if a person's review is consistent with other reviews, or the user is generally consistently accurate in their reviews.

For example, if I review a suggested edit, perhaps a review should only be awarded when I choose "approve" and the final outcome of the review process is "approve" (after all votes are cast), or vice versa (I select reject and it is ultimately rejected). I'll get penalised when I make the wrong choice (by not getting a review), but this will encourage people to make calculated, intelligent choices rather than blindly selecting an option. This, ultimately, can only improve the system, by ensuring that reviews are only incentivised when the reviewer is contributing to a consensus.

This can also be used to get an idea of my review accuracy: if I consistently choose to approve/reject suggested edits that are ultimately approved/rejected, then I'm a pretty accurate judge of what is and isn't a good edit. If the ratio of correctly judged reviews to incorrect ones is high, perhaps only then should reviews be awarded.

The particular circumstance described by the OP is probably a little harder to fix this way, but I'm sure it can be done.

We have two problems:

1. People are getting free review 'points' for their shiny badge 'chievos.
2. Real reviews are not being accepted, and not being prompted (had you not been part way through review, it may have gone unnoticed)

So I would propose a subset of the review options (upvote, 'not sure','looks good') etc pass the baton on for someone else, and it stays in the queue until it's had several reviews etc.

So here's the way I see it.

You start to edit, you are flagged as 'in review' by the system. Person two swoops in and upvotes it, then has to press either a 'looks good' or a 'not sure' button to register their action, which is added to a 'decisions queue'. Part way through you get an AJAX'd update saying '<insert UserName here> upvoted and said "Looks Good/Not Sure" ', in the same way you are updated when a new answer comes in or a new edit on a post comes in. When you finalise your edit, you can click on 'not sure' (maybe you doubt your edit is enough) or 'looks good'. Perhaps even a 'reason' field that says "The last review did not fix the problem" that extends the number of necessary reviews?

We then pile up about 5 decisions on the 'decision queue', for instance: Upvote - LG, Edit - LG, No Action- NS, Upvote - LG, Upvote - LG. Where LG = looks good, and NS = not sure.

The system then looks at who said what. I don't know how best to do this, but I'd reckon that the first upvoter shouldn't get any points, as an edit was needed. The editor should get points related to the 'views' of the other three reviewers. That is, there are more Looks Good votes than Not Sure votes (within our descion queue), so the editor gets points. As do the people who voted NS and LG because they reviewed it to pass it on to the next.

Had it been: Upvote - LG, Edit - LG, No action - NS, Upvote - LG, No action- NS.

The first upvoter still gets nothing, the editor gets no/fewer points (as it is now part of the LG/NS review from the other reviewers, i.e. by majority they don't think the edit has improved the post) and the other three get some points, the decision queue gets wiped and it's back up for review.

In this system you wouldn't get points for each review, but you might get 0-3 points dependent on how well received your review is.

Or we have: Upvote - LG, Edit - LG, Edit - LG, No Action - NS, Upvote - LG.

Here the system requires a deciding 'vote' (were the two edits good or not sure?), and because that might incentivise speed reviewing, we hide the number of non-edit actions. Then a NS/LG either commits the review, or it pushes it back into the main list of reviews. We can then give the first editor fewer points, and rely on the last three actions to show us if the second vote was good.

Conclusion:

1. People who speed review no longer get points if an edit is needed or a majority claim 'not sure' after them
2. Edit's are peer reviewed and then points are distributed among the review contributors according to effort (two points to the editor, 1 each to the last three reviewers?)
3. We might catch people who 'fake' edit a post, by the subsequent voting.

Obviously we end up inflating the number of points so we make the badge goal get X points, not review Y posts, and make X>Y in some sensible fashion.

I hope that makes sense and is helpful.

tl;dr: Make each review 5 reviews and only commit when a consensus is met. Give points out according to contribution to consensus making.

As a side/first thought, it seems to me that the 'serial lightning reviewers' are the problem reviewers. They go through the queue, up-vote every post, and then light a cigarette.

I think that when reviewing a post and downgrading it, the karma should not be hit with -1, this way the serial reviewers can at least rate bad posts as bad with impunity.

Update: apparently there is no penalty for down-voting a reviewed question, this should be much more obvious in the UI.

Update 2 : There is a penalty to downvoting a reviewed answer though! There should be none when reviewing first answers! Most first answers are not good. I just lost 2 points ;)

Update 3 : This is awesome : C# Test if user has write access to a folder The late answer copies the solution with a few comments. Pretty pointless but of course the reviewer +1's it because he only checks the question and the late answer, not the existing answers..

I would solve the problem with a penalty for bad review. A review turns out bad in any of the following cases:

• upvote review: the question is closed
• upvote review: the question gets negative total vote after a week [1]
• downvote review: the question gets positive total vote after a week [1]
• comment review: the comment was flagged and removed
• edit review: the edit was rejected

edit: [1] votes consistent with the reviewer could count double into total, to help in case of controversial questions with lots of votes in both directions

Someone else will know better what kind of penalty fits best and whether it should be applied always or only on frequent review errors. I guess it will also improve question's quality which I find very poor.

What if you allow multiple "provisional" reviews of a question, and then the questioner picks the winning review within a certain time window after the question is asked? The time window should be short - an hour might be enough. If the questioner doesn't act within the given time window, fall back to the first-review-wins system. Since no questioner is going to select a downvote review as the winner, they might only be allowed to select the winner if the majority of reviews are upvotes - otherwise the first downvote reviewer is wins credit when the review window expires.

I'm still trying to puzzle out where the incentives would point if this were the situation. A few thoughts (not all on the positive side)...

• Each individual review is less valuable to the reviewer's reputation, since there's a good chance it won't be chosen as the winner. This might discourage gaming the system.
• Reviews that substantially improve the question might be favored over simple "vote and run" reviews, since the questioner has an interest in having a quality question, however...
• Pride might keep questioners from selecting quality reviews that alter the presentation of the question, instead preferring the affirmation of another reviewer who simply says "yeah, looks good"

Also, making the interface for this process grokable to a novice questioner would probably be a significant challenge.