Peer review: Dealing with vague/open-ended/out-of-scope questions during review

Question

I recently received some reviewer's comments on a paper I have under review. One reviewer (out of three) in particular has raised a large number of points. Some are very clear and target specific weaknesses in the paper (like 'do uncertainty calculations there' or 'are the findings robust to alternate choices of paramets x', etc). However, a number of issues stand out in variations of the following ways:

1. They don't address weaknesses or omissions in the paper, rather they ask for additional explorative analyses that are outside the scope of the original paper.
2. They are open-ended, in the sense that there's no clear conditions under which the issue has clearly been addressed. Often they are downright 'rabbit holes' that can't pausibly be 100% exhaustively explored.
3. The paper is fairly clearly scoped in the issues ask for highly labor-intensive analyses outside of that scope, without arguing that this is necessary to answer the research questions.

In short, some of issues seem more like additional research ideas that actual issues with the paper. I'm uncertain how to best handle this. Can I just plainly state that while interesting to persue in future work, that is outside of the scope? Or would I be better off doing some short analyses (that I fear will only raise more questions, cf. point 2 above)? Or some third options? What are your experiences and advice regarding this type of situation?

Answer 1

Can I just plainly state that while interesting to pursue in future work, that is outside of the scope?

Yes, if proposed experiments are not possible or would take as much effort as the rest of the paper (that is, it's asking for a whole new paper in addition), it's fine to say it's out of scope.

Or would I be better off doing some short analyses (that I fear will only raise more questions, cf. point 2 above)?

Sometimes, reviewers ask for new experiments that you've already done in some fashion, though, or you can offer other data to support the same lines of inquiry even if it's not the "ideal" approach. In those cases, it would certainly make sense to do some brief analyses of what you have and present them, even if it's not quite complete. I find it common to see "half-done" analyses added to papers that make me speculate that they were added last-minute by a reviewer suggestion; generally, you can frame such things as exploratory and if they're added in the midst of peer review they won't typically raise deeper scrutiny, as long as they don't add wild new conclusions. You can also ask the editor's advice/preference.

Or some third options? What are your experiences and advice regarding this type of situation?

A third option, related to the first two, is to explicitly discuss these possibilities, rather than doing them. Add the proposed experiments to the discussion section, possibly supported by preliminary analyses in that direction.

I've certainly had reviewers ask for some relatively ridiculous (in terms of cost and scope) add-ons, and some utterly ridiculous ones too (such as proposing experiments that would never ever be ethically permitted in humans; I think this is a case where a busy reviewer briefly forgot which primate was being discussed); editors are familiar with these requests and will respond accordingly to your response.

Do consider, however, that sometimes there is a big hole in your work, and the added experiments are really the only way to address it. Hopefully that doesn't describe your case, but know that it also isn't necessary to address every possible avenue for criticism; it's simply not ever possible to do so, there's a reason that research progresses incrementally.

Whatever ends up fitting your paper best, do be courteous and appreciative to the reviewer in your response. I think seeing some next steps indicates a reviewer is rather excited about your work and sees the future potential in it, too - these are good things.

Answer 2

Can I just plainly state that while interesting to persue in future work, that is outside of the scope?

Yes.

In your rebuttal letter, acknowledge that the reviewer's suggestion is interesting but you don't have time/resources to investigate right now. If you want to quantify the extra work you can ("would require hiring three additional post-docs and purchasing $1 million of new equipment" or whatever) but you can also just say "out of scope", or "too labor intensive". Then, add a few sentences about the reviewers suggestion to your future work section.

Answer 3

Just the normal reminder that when your write your rebuttal to the review, your doing so to the editor not the referee. While the editors will take on board what a referee says, they are also human. So if you explain carefully why something would be inappropriate for inclusion in this paper (be it the time needed to complete, out of scope, or just a plain misunderstanding of what you did) then the editor should take that on board.

That said its also usually makes things easier to give a little ground even to inappropriate suggestions. Like adding some citations or a few sentences discussing future work. Just so it doesn't look like your completely ignored what the referee asked for (they presumably think what they asked for was appropriate).