What goes into a proper review of a paper?

Question

I've been asked to review a paper for a publication. I've always been teaching faculty, not a researcher and I've never done a review before, but the paper is within my area of expertise and I have some time, so I'm inclined to give it a try. But some basic guidance on the format and content would be helpful. What do you look for in a helpful, high quality review?

Answer 1

In my field, all reviews that I perceive as "high-quality" follow a certain structure:

• Summary of the paper (one paragraph)
• High-level remarks that justify your recommendation to accept/reject/revise the paper (e.g., a list of pros and cons; or a discussion of blocking issues that have to be addressed so that the paper can be accepted after a revision)
• Detailed remarks, addressing specific sections in the paper
• (If any:) Grammar problems and typos spotted

Answer 2

Here are some issues to address:

1. To what extent is the content of the paper original? What is the value added to what already exists?

2. Is the paper correct? If not, do errors affect the central message of the paper? Do you think these errors can easily be repaired or removed without affecting the value of the paper much? Are there issues with the arguments and reasoning?

3. Is the paper written well? This concerns the language but also whether everything is explained in such a way that a typical reader of the journal could understand it, and whether the arguments and reasoning are clearly and concisely presented. Is the writing too long-winded, or on the other hand is too much assumed that is not explained? If you think that anything specific requires clarification, mention this in the review.

4. Is existing literature taken into account properly? Give references to literature that you think is important to acknowledged but is not currently acknowledged in the paper. Is the current state of research properly represented and the content of the paper put into proper context? One thing that I see often is that existing literature is cited but not represented properly, particularly if authors make an argument that their new method is better than an existing method - the existing method is often not quite as bad as portrayed. Reviewers cannot be expected to read a lot of literature they don't already know; however I do look up cited papers that I don't know occasionally, to get a quick impression of whether they are represented appropriately.

5. Is everything explained in enough detail so that an independent researcher could reproduce it? Are data, software, and other important material made publicly available?

6. Are visualisations done properly, in a useful and not misleading manner, with clear explanation what they mean? (Same thing can be asked about formal/technical notation.)

7. Even if original and correct, is what is proposed of practical use, or of real theoretical interest? Is it well motivated; is the material elaborated enough that it can be used as it is, or is there still much work left to get it at this stage? Should some of this left work really have been done by the authors? You may ask for a revision in which certain remaining issues are to be addressed if you think that not enough work has been done to put this into practice (also other kinds of work such as comparisons, see 8) as far as this can be expected of a single journal paper (obviously some projects are so complex that several papers are required to treat all relevant details - in other cases I really think the authors should have done more before submitting for publication).

8. If the paper involves comparisons, for example between newly proposed and already existing methodology, are these comparisons done in a fair and informative way?

9. If the paper involves generalisation from a specific situation, do you think this generalisation is justified? Is there an appropriate acknowledgement and discussion of uncertainty and limitations? (I will criticise overblown marketing claims that can be found in many papers.)

By the way, many editors like it if you do not say directly in your comments to authors what your recommendation is (reject/revise/accept), but rather leave this to the Comments to Editor; in many systems you can just click it in somewhere. The editor then has an easier job to justify an overall decision that may deviate from your recommendation.

Answer 3

Before you start reviewing, have a look at the reviewing form they want you to use. It can be quite annoying if you have already written a review in your head, but then find out on the spot that they want you to break it apart into several specific subcategories. It's best to be one step ahead of that.

On the one hand, you want your review to be informative to the editor. They will need to decide whether to accept the paper or not, and they are asking you to provide arguments that help them in making that decision. So make sure you do write a compelling argumentation. If the reviewing form has a field "confidential remarks to the editor" or somesuch, use it to quickly summarize the main reasons for your recommendation.

On the other hand, you want your review to be informative to the author(s). The authors of course want you to accept their paper. However, if I am on the receiving end of a review, my appreciation of the reviewer mostly depends on whether they provide something actionable. The authors may not be happy with your reasons to reject the paper, but it helps if you make clear what the authors should do next. For instance, instead of "the authors should run comparative experiments with more state-of-the-art methods", write "the authors should include methods [A,B,C] in their experimental comparisons"; the former is a generic handwavy thing, the latter gives me a specific, concrete thing to do.

Answer 4

I have had to find this kind of resource lately for colleagues and found this description Wiley's website quite useful: https://authorservices.wiley.com/Reviewers/journal-reviewers/how-to-perform-a-peer-review/step-by-step-guide-to-reviewing-a-manuscript.html In particular, the section on How to structure your report is quite good.

Answer 5

Some journals have specific formats or forms they want you to use. Check that before you start; otherwise, you may end up doing a lot of work for nothing.

If there's no guidance, you could put something together like this:

1. Short statement (1-3 sentences) that says, "I read this paper, it (is or is not) an important contribution to (the field) as it (does something/solves some problem/answers some interesting question). I recommend (your recommendation)."

2. Most authors (and some editors) like it if you tell them what's good about the paper. In some cases, this could be the most difficult part of your review. Write as much as you think you should; no more than ~a paragraph or so.

3. Assuming there are problems, make a numbered list (numbering them makes it easier for the authors/editors to refer to them later). Each point should be a clear and concise item discussing your issue with whatever they wrote. "Issues" can be technical issues, logic problems, mathematical mistakes, code failures, problems with tables/figures/equations, or with the overall flow of the paper.

4. Group all of the English language issues into a single point (if you can). If there are many, you can either suggest that they get help or have a general numbered point about language with bullet points below indicating specific issues. Resist the urge to rewrite their paper.

5. After you've finished laying everything out, write a short (1-2 sentence) recommendation. Your recommendation should follow and be supported by points (3) and (4); i.e., don't accept the paper without reservations if you end up with 20 major issues and don't reject the paper if you found one typographical error.

6. Finally, close your report with a note to the editor with a statement about your willingness to be a part of the rest of the review process (which will be journal-dependent).

Reviewing papers is a very important job of everyone in the scientific community. Make sure that if you recommend the paper to be part of the literature, you would be happy if your name was on it (note: it won't be - the editor should keep you completely anonymous). You are, in effect, the goalkeeper. Don't let trash get by you.