What prevents reviewers from providing biased and very negative reviews?

Question

One of the fundamental features of science, maybe even the most important, is that publication of scientific results is peer-reviewed.

I want to understand why peer review is effective in the scientific community, because I want to apply principles of peer review to a different domain.

As I understand it, a researcher is reviewing work of a competitor who is producing scientific results in exchange for grant funds and reputation, usually same as the reviewing researcher does. Of course, this competition may not be very direct, but the reviewer is in an overlapping area of research. In addition, there is some natural bias of every human being to groom their own ego. Based on this, I expect most reviews to be very negative. However, it seems that biased and extremely negative reviews are rare for decent submissions.

What stops reviewers from providing biased negative reviews?

Answer 1

The term competitors does not really describe the relationship between different scientists in the same field. Of course there is some competition for grants or such, or even to put one's name in a new result, but some aspects are deeply different:

• A publication in your field is a good thing, even if it's not from yourself. It makes your area important, and alive. It's actually very important when it comes to applying for grants.

• You cannot benefit from someone else's work by writing negative reviews. There is no way to simply slow down publication of one paper in order to publish the result first.

• A rejected paper will not slow down the "competitors" research. Sure they will have to improve it and submit it elsewhere, but this takes a marginal time compared to the research process itself.

• Dishonest reviews are clearly dangerous, as some people will see your review with your name on it (I'm thinking of other reviewers on such systems as Easychair). These people might very well be the ones reviewing your grant application a few weeks later.

• why would one risk being exposed as dishonest, when anyway there are several reviews for the same paper, and any deep difference should trigger an in-depth investigation by the editor?

The most frequent bias I have seen are review of the form "you should cite those 5 obscure and vaguely related papers all from the same author", which clearly indicate reviewers in need of citation, but nothing much worse.

Answer 2

I think the main factor that makes the peer-review system to not fall in the trap you described is that academics are finally interested in the advance of science. If I have to review a good paper, then I enjoy it and want to see it published. If I review an almost good paper, I suggest some improvements and enjoy if these are incorporated and the work is published. The very mild advantage I may have if I reject more papers does not really seem to be worth it, if I have to act unfair and unreasonable to get this advantage.

Another factor may be: The reviewer is not anonymous for the editor. Editors are often influential people and writing consistently bad and unfair reviews will make you look like a mean and unfair person. Also, the editor will not choose you again as a reviewer if you write such reviews and you will be effectively eliminating yourself from the review system for that journal (while you can still submit papers).

In view of the bounty I gave the question another thought and here is one more thing: Community. Science is well organized in communities (e.g. I consider myself as part of the mathematical community, the applied math community, the community on mathematical imaging, the optimization community and some more). As part of a community one has a sense that there are some rules one should follow to be a valuable member of the community and of these rules is fairness. Being unfair is misconduct by the unwritten community rules. So even though one may get away with several cases of unfair behavior, it feels like one is cheating the system. But staying a respected member of the communities is very important both for the scientific standing and also for productivity. This reasoning also explains that bad behavior sometimes happens, when several competing communities form which are somehow "enemies of each other": One can stay a respected member of one of the communities while still treating members from the other community unfair.

Answer 3

I have nothing to do with academy, so take my answer just as something that I'd expect based on my understanding of human interaction and behaviour.

Reputation

The main resource you have is your own reputation, and that will be severely harmed if people are accusing you of being unfair, biased or "not objective enough". You might be able to get away with it once in a while, but overall, it's quite enough for the system to work. Worst case scenario, you slow down the propagation of the work in question, but in practice, the paper will find another way (another reviewer, journal...). Even if you succeed, you risk harming your reputation, which is extremely important in a field centered around collaboration with peers and promising understudies.

Competition

You only considered competition between individual scientists, which is mostly a thing of (1) competing for grants and (2) competing for reputation. I've already dealt with reputation. Competition for grants might be important for you if you're trying to adapt the process for something like performance reviews - it's the clearest cut case where hurting others can help you. However, it doesn't have much to do with the peer review process - that would indeed introduce a very strong motivation to be "as unfair as you can be without actually appearing unfair".

However, there's also another competition going on - that between individual reviewers and their journals. If your paper was rejected based on grounds that are seen as fair and objective, you'll likely also be rejected by other journals. If not, the other journals might jump on the opportunity to publish your paper, while also implying that another journal has treated the paper unfairly. You can't do this very often if you want your journal/reviewers to keep being relevant!

Points to take away

If you want to use a similar system for another domain, make sure that similar incentives are at play.

• Have multiple independent reviewers, and let people choose their reviewer (while the reviewer has a chance to decline).
• Make sure there's not a lot of "authority" in play - for example, superior-underling relation doesn't make for good peers. Peer review works best with consensus and with reasonably objective / shared values.
• Make everything public (in the team / company). No anonymity, no "hidden" reviews. This is necessary for the reputation-based controls to work. In a way, it's a redundancy in the peer review system - it allows people to "review" the reviews themselves.

It works best in mostly flat hierarchies. Thinking in terms of a performance review in a company, peer review will be a poor choice if managers order people around. On the other hand, if managers have to persuade others to follow with their plan, it might work great :)

Answer 4

In addition to the mechanisms listed by others, most journals I've published in so far allow the authors to list people who they feel have competing interests and thus should not be considered as reviewers. So if there's someone who you know is working on competing research, you can put them on that list.

Of course, this isn't perfect, but if you get three reviews out of which two are largely positive and one is deeply negative, the editor is likely to decide in your favor anyway.

(I'm in biology, specifically plant genetics. I'm sure it varies a lot between fields.)

Answer 5

Double-blind review (neither reviewers nor the reviewed are shown who the opposite parties are) was invented to help with some of these issues. It's used almost exclusively in some fields for conference and journal peer review (various parts of CS comes to mind). Mostly, it's to help prevent unconscious bias against underrepresented minorities and women cropping up and tainting reviews. Given that it's relatively easy to spot women and foreigners from names, double-blind review tries to stop these biases from creeping in by hiding the names from reviewers, too.

Answer 6

There is one aspect that has been mentioned in @weronika's answer, but not been stressed enough in my opinion and probably not obvious to the author: There is usually more than one reviewer.

In my area there will normally be three different reviews, which seems to be a good compromise of keeping total workload low and allowing to identify "outliers". An editor may not be able to see if a single review is biased, but if they see reviews contradict each other considerably, an unfair review might become noticeable.

---

Added for clarification: Three reviewers are certainly not enough to get meaningful statistics (that's why I wrote "outlier" in quotes), and only multiple choice or numerical ratings in the reviewers' sheets could be compared in a statistical way. But there are several critical factors that can reduce the quality of a review, and their possible impact can at least be reduced by having more than one reviewer:

1. inadequate knowledge in the field of expertise,
2. negligence or missing time, and
3. bias or malicious intent.

Answer 7

We've seen randomly generated papers accepted, Universities acting like paper mills, and researchers being scored soley on the number of papers published.

The review system is based almost soley on the wish that everyone has good intentions. Unfortunately this doesn't make the system work flawlessly.

The most common complaint is that a lot of reviewers don't take enough time to review the material. Making people invest precious time in a peer-review system might be a harder problem than making them behave honestly.

I haven't heard of cases where reviewers intentionall gave bad reviews. Though I have heard of feuds between authors. This is especially problematic in small fields where the reviewer isn't as anonymous as we would like. Note that feedback might identify a reviewer. Also note that the author is never anonymous (and is hard to make anonymous) which makes the system more suspectable to malicious reviewers.

The partially mitigating factor might be the editor. Which acts as a sort of referee. But what if the editor is malicious?

To answer your question: there is no definitive factor that prevents malicious use of the academic peer review system. There are instances where the system has been abused. I also do not believe that scientists are less often malicious than any other person :).

Answer 8

To be honest, it is very hard to justify (or in this matter to generalize) the behavior of a group of people (reviewers). I'm sure some of us were victims of irrational and unfair reviewing process i.e., getting reviews after 4-5 months with poor questions, harsh language or the most obvious case where you can find a very similar work to your paper published couple of weeks of your article that you received after a delayed review!!

Even with double-blind peer review process, a reviewer can guess who are the authors of a certain paper based on their research topic, wording, experimental set-up and/or even funding account/agency (usually mentioned in the acknowledgement section). So, it is very PERSON dependent! Think of it like athletes who compete in sports, all of them are subjected to the same rules, however some will always try to cheat the system!! whether with drugs or with exploding loop holes!

The good news is there are more reviewers who are genuinely interested in advancing science than those who are eager to unfairly compete with other researchers. Don't let any of this put you down, think of it like this.. You must have been doing some very interesting research!!

Answer 9

Your methodology itself is biased. Anytime you are doing something to alter the review process you are creating a biased review. This could be as simple as wording before or during the review process and at the other end of the spectrum leaving out some potential reviewers.

My advice:

• worry more about your science than your review

• welcome the biased reviewer. I would rather see the flaws that my research/science has as soon as possible, not after it has been fully publicized.

• understand that there is an element of risk or wasted time involved. If someone is biased and using corrupt review tactics this is more time consuming. But and big but this also allows you to learn more about the outside forces you are going against and how to refute them better.

• and a further point is an obviously biased review will be easily refuted. As it is, it will lower the reviewer's stance/reputation and highlight the work that you are doing. People are far more passionate when there is an A vs. Z dynamic than just hearing about A.

So really the only thing that you need to do (and I write review/survey systems) is to have a mechanism of review rebuttal and make sure the person writing the review understands this. If I cared about my name I would not write an untrue/biased review about something if I knew that someone could refute my stance directed at me. Transparency is key.