What to do about a dubious competing publication in the same journal I'm submitting to?

Question

Before submitting my predictions paper, I spotted a paper with alternative methods for the same predictions. It appeared two weeks ago online-first in the same journal I'm submitting to.

Digging into it, I found several issues. To validate predictions, the authors cherry-picked empirical results from the same table of another paper. The results' type and time period, clear from the table, were not the ones assumed for the predictions, which essentially invalidates comparison. The paper's code is on GitHub, but not the scripts for input filtering; so no reproducing the results. The model stated in the paper is commented out, and the one in the code is trained on both training and validation sets, contrary to practice and the paper. If assessed fairly, the predictions would be rather less impressive. And it is hard to tell how they were actually obtained.

Q1: What should I do about these issues?

There are similar questions, such as 1, 2, 3, 4, 5, 6. However, the usual answers --- to move softly and slowly, or just park it --- is probably not the way to go: a flawed paper on public health in a major journal can have consequences. I've doubts about emailing the authors as I'm not sure how honest their mistakes were and why they would respond. (Surely, authors would triple-check this type of papers? Literally, "we publish; you perish".) I guess I could open code-related issues on GitHub, and see whether the authors engage. At least people using their code might take notice.

Q2: How should I proceed with the submission?

This journal is the best place for this research, and for many reasons I'd like to submit as soon as possible. The reviewers would probably request me to compare predictions with that paper's, which is fair, but not in the circumstances. Shall I just submit without mentioning the other paper? Or should I attach a diplomatic letter describing the issues? I don't think the original reviewers are at fault: the issues are hard to spot without manually cross-checking numbers and references and digging into the code base. Still, would it be wise to make some requests, so that my paper be reviewed fairly?

Answer 1

My suggestion would be to submit your paper without additional comment. Making comments initially will only slow down the process. If you are asked about your view of the other paper by an editor or reviewer then respond at that time.

At most, point out that you come to different conclusions than the other paper and believe your methodology to be superior, resulting in better predictions.

But submitting your work will get it into the review process.

---

Caveat: Note that there is quite a lot of "opinion" in this answer and a weighing of likely outcomes. It is the editor of the journal who's opinion carries weight here. Other "solutions" might work. I suggested the above to get the quickest feedback. Or at least get appropriate questions asked early to help guide later actions.

Answer 2

My suggestion would be to prepare your paper for publication ASAP, which you would have done anyway.

You raise a very good point about correcting the record, especially when it comes to public health. In your situation I would distill the disagreement between your results and theirs into as few words as possible -- one sentence for the biggest difference in results and one sentence for the biggest flaws in methodology -- and then stick it in the appropriate place in your paper. For example:

Kaufman (1924) predicts that widgetitis will only infect one American a year, while our models project anywhere between 2000-4000 widgetitis deaths annually instead. We attribute this significant discrepancy to Kaufman's likely inaccurate estimate of population densities and demographics at the county, state, and national level, as well as improper training procedures of the HAL9000 neural network, which we discuss in the Supplementary Information.

Then include a brief summary of your own replication findings in the Supplementary Information. The best cure to a wrong study is to make your study infinitely more citeable -- putting a lengthy critique in the SI probably doesn't help, right as it may be. You could set up a GitHub Page (or even Code Ocean page) presenting your counter-analysis in more detail and then link to that.

If you follow this course, you can request to the editors to not ask Kaufman (in this case) to review your paper, since there would be a natural conflict of interest.

I don't know about norms in your field, but in my field I would almost certainly email the authors to let them know that I was about to raise issues with their paper in mine. You can easily send off that email and then continue working on the paper; if they respond, you can continue discussing with them as needed, and if they don't, it's on the record that you tried to discuss this matter more collegially.