13

Initially in my career, when I was sent an article to referee my decision to reject/approve used to be influenced on how "good" the journal was (IF, SciMago's Q). I.e.: if the article was ok but the results seemed "not enough" for a Q1 journal, then I would advise not to publish there.

I no longer do this.

My current approach is: if the article is well written and the results are reasonable, even if they seem small or not entirely novel, I'll approve. My reasoning is that the researchers themselves should be the judges of whether the work is important or not in the field; the impact of the journals on the process should be restricted to publishing reasonable work and nothing else.

Do journals expect us (the researchers/referees) to filter out articles based on their status? If this is true, should we even care?

I understand this is a heavily subjective topic. I believe the discussion is relevant to the site, if it's not go ahead and close.

Improve this question
11
  • 7
    While one could argue that the editor is fully responsible, ask yourself what input you would want if you were the editor?
    – Jon Custer
    Commented Apr 20 at 15:55
  • 3
    @JonCuster At point here. The reviewer should provide the editor with the information they need to make an informed decision.
    – Captain Emacs
    Commented Apr 20 at 17:53
  • 8
    Debasing all journals to the lowest common denominator does not seem to be in the best interests of actual progress.
    – Jon Custer
    Commented Apr 20 at 23:02
  • 2
    @Gabriel: Let's say you are a scholar on Gerard Manley Hopkins. Let's think about how other literature scholars judge the state of scholarship on Hopkins. One not too inaccurate model would be that they look at papers in Modern Language Quarterly on Hopkins, read them, and consider how interesting those papers are to be representative of how interesting current scholarship on Hopkins is. If MLQ publishes a bad paper on Hopkins, that makes other literature scholars less interested in Hopkins research, meaning you are less likely to be invited to speak, fewer grad students study Hopkins
    – Alexander Woo
    Commented Apr 22 at 2:24
  • 2
    which means you have fewer colleagues producing papers you would want to read in the future (and the conference you attend may no longer be able to support a session of papers on Hopkins), and if you don't yet have tenure, your work is looked on as less important by association so you are less likely to get a job or get tenure. So it's in your interests for the one or two papers MLQ publishes every decade on Hopkins to be papers that interest others in studying Hopkins.
    – Alexander Woo
    Commented Apr 22 at 2:28

7 Answers 7

Reset to default
15

From my experience, I am surprised at this question. It has happened several times that I have refereed a paper, recommended acceptance, and had the paper rejected pretty much solely because my recommendation was insufficiently enthusiastic. It has also happened that I have recommended rejection on the basis that the paper was of borderline importance (relative to other papers published in the journal) and had my recommendation overturned.

This is actually quite a tricky matter, because different people write reports with differing degrees of enthusiasm simply because of their natural mode of expression, and it can be hard for editors to parse just how enthusiastic a referee is about a paper.

If a paper is important, I try to briefly explain why the paper is important. If a paper is not important, I also try to explain that. For papers somewhat in the middle, I find it helpful to give comparisons to other papers published in the journal or recent papers published in journals of about the same prestige as the journal in question.

Improve this answer
2
  • 1
    This has never happened to me. My field works on a single-reviewer almost exclusively, and what they say usually goes.
    – Gabriel
    Commented Apr 21 at 13:06
  • 1
    @Gabriel: Yes - my field works on single reviewer also, but the editor is almost always someone reasonably knowledgeable about the subject area of the paper.
    – Alexander Woo
    Commented Apr 21 at 18:30
11

Yes, of course your recommendation should take into account how strong the paper is relative to the journal. This is information that the editor wants, and typically will not be able to assess as well as the referee can. It is also what other referees do, so by ignoring this aspect you may be giving undue advantage to people fortunate enough to get you as a referee.

In my experience the invitation to review will normally clearly indicate this. Here is an example from a journal I review for. (Of course, you might have your own opinion of how selective different journals actually are, even if their blurbs are very similar.)

In making your expert judgment, please note that [journal] is highly selective. It is the journal's policy that manuscripts which do not contain a major new research finding or a novel approach to [subject area], even if technically correct, should be rejected.

The responsibility of evaluating the paper's importance certainly should not rest with the author, since the author's job (usually literally) is to get it accepted by the strongest journal they can. The only thing that stops people submitting everything to the strongest journals is the expectation that referees will say "this isn't significant enough for journal X: try journal Y."

Improve this answer
7
  • 2
    Although I understand your point and I opened this question to read other points of view, I will not be doing this ever (again). The journals are already getting my labor for free, they don't get to tell me that I should keep their "high selectivity" in mind. That's just giving them even more power and I refuse to do that anymore.
    – Gabriel
    Commented Apr 22 at 0:24
  • 3
    @Gabriel: That's an entirely foreign point of view to me, because the editors who ask me to review papers are almost always colleagues whom I respect or even friends. Strangers whom I don't even know by reputation asking me to review get a 'No' by default. Also, I do have a professional interest in giving more visibility to the best work in my field and giving less visibility to the not so good work, because people in other areas will judge my area of research by its most visible work.
    – Alexander Woo
    Commented Apr 22 at 2:13
  • 4
    @Gabriel I appreciate your point of view, but I think that in your case the appropriate thing to do is refuse the review request, rather than accepting it but not actually doing what they asked. (I should probably have prefixed my whole answer with "if you decide to review".)
    – Especially Lime
    Commented Apr 22 at 8:03
  • 3
    @Gabriel I don't understand how the normal model is "handing over power to journals/publishers". In fact it is the opposite: the journals are handing over power to the referees, subject to oversight by editors. Hence I intend to continue using this power responsibly.
    – Especially Lime
    Commented Apr 22 at 11:20
  • 2
    Because that's how it works today. Journals are used as a benchmark of how important the articles that are published in them are, instead of judging the articles by their own importance. Publishers don't "handle over power" to us, they recruit our labor for free! My point exactly is to use our position as reviewers responsibly: by doing a service to academia and our fellow researchers, not to the publishers. Why do you think thecostofknowledge.com exists?
    – Gabriel
    Commented Apr 22 at 11:43
2

I'm sympathetic to your view of the role of the peer reviewer. I think the best information the peer reviewer can give to an editor, at least in the fields I work in, relates to whether the manuscript is methodologically competent and whether it deals with similar work correctly (shows how it fits into literature, doesn't ignore previous innovations).

I find it difficult as a reviewer to know whether a manuscript clears the novelty/impact bar, especially for some journals that I don't spend as much time reading. For me, the editor has the best judgment of what is important, interesting, and high enough quality for their journal. I generally assume they can tell whether the argument is important and novel, if it is correct. But they may be unsure of the correctness.

My reviews tend to focus on whether I believe the paper is accurately representing prior knowledge on the subject, doing something that follows logically from their take on that existing knowledge, and uses methodology that that is rigorous/adequately answers their questions. Sometimes I get stuff that seems fairly trivial but is otherwise very trustworthy. In the social sciences, we are often trading off between importance and quality of evidence so I try to tell the editor that the evidence is of high or low quality but they can decide whether the rest of it fits for them. If the work seems to be of especially high quality and importance, I will still mention that though...even if I may add a caveat that I don't intend to usurp the editor's judgment on such matters.

Improve this answer
1

My philosophy has always been that a paper should be critiqued for it's flaws and compromising issues no matter what the impact factor of the journal is. We shouldn't let low quality, scientifically questionable work slide just because an impact factor is low. I treat all peer reviews equally.

Improve this answer
3
  • Many a journal editor believes that a paper can be have no flaws or compromising issues, and yet not be revolutionary enough for their esteemed title.
    – Ian Sudbery
    Commented May 3 at 6:53
  • That should be the editors call. If it gets to peer review then the editor believes it's worthwhile at least to be scientifically evaluated.
    – Rami N.
    Commented May 3 at 18:01
  • Yes, we'll, I think that's the OP's point of view, but some editors need reviewers to tell them how exciting and ground breaking it is, because at big, generalist journals, the editors might not feel they know enough to make that call. They only know it must be "ground breaking", but they don't know what ground breaking is.
    – Ian Sudbery
    Commented May 3 at 22:35
0

As a thought experiment, let's imagine what would happen if journals stopped caring about the "level" or perceived impact of the papers they publish, and instead published all the correct, on-topic submissions that they have space for.

Would this be a good thing? I don't think so, because all journals would be equal, and all publications would look equally good on a CV. But the intense competition for academic jobs would still be here, and the best/only way to stand out would be quantity of publications. Do we want to incentivize that? From my perspective, there is already too much emphasis on quantity over quality in academia, and this would only make it worse.

Improve this answer
2
  • 3
    I think your thought experiment starts the same as the OPs: imagine that all the technically correct research gets published. Then, in OPs thought experiment (and some would say, hopefully, in reality too), the best (highest "level") papers would get noticed, extended by others and cited. The worst papers would get ignored. So there would be another way to stand out: who has the most cited papers. That obviously has it's own problems (popularity of the general topic and size of the research field); but even in this "tought experiment" world, there would be metrics other than quantity.
    – penelope
    Commented Apr 22 at 18:36
  • Alternatively, the people doing the hiring could accrual read the papers on your CV and, together with your job talk, form their own judgement as to the importance of your work.
    – Ian Sudbery
    Commented May 3 at 6:56
-1

My reasoning is that the researchers themselves should be the judges of whether the work is important or not in the field

I have spent a lot of time researching published technical research papers for accuracy and usability and validity. I never, in all of my years of doing this, have found any technical research from a certain extra large country to be of any use via any of their findings. Never. These were published in scientific journals that showed up in major universities in the United States of America. My guess, but it is still just a guess, is that some journals publish some papers from some sources because it makes them look more "inclusive" even though [if] the paper is trash.

Your question:

Do journals expect us (the researchers/referees) to filter out articles based on their status? If this is true, should we even care?

Due to what I just told you: They might expect it. But their actions do not support that. Should you care: Not really. Publish facts and honest conclusions and leave it to the reader to judge if the journal does not.

Industry commonly has personnel reading and studying and considering research reports in their selected support journals. Universities sometimes do the same. Some sources have for so long a time been a waste of journal space that they are almost automatically discounted and their articles not read. Some sources are so useful that their papers are sought out. Be useful. Do not worry about the rest.

Improve this answer
-2

One stop gap measure to address this and other problems with peer review is to make the reviewers public and pay them a fee from the journal company. This creates an open market rather than an arbitrary merit based socialist system of this research is better than that. But eventually, peer review needs to die off and arxiv.org with pubpeer should be the new model of publishing research.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .