Posting commentary papers to preprint repositories

Question

Short version:

We noticed a problematic but influential preprint, wrote a commentary paper to criticize and rebut it, but were not able to post the comment to any public preprint repository. In this situation, what can I do against bad science?

Note: Some focus on our wording being too harsh, even accuse us of "personal attacks". In fact, we enlisted the help of many proofreaders to ensure that we did nothing like these, especially when the situation was so politically sensitive. Most feedback says that we were too friendly.

Long version:

Background

On 8 Jun 2020, a preprint is posted on Havard institutional repository "DASH". It suggests, based on satellite imagery of hospital parking lots and internet search trends, that COVID-19 outbreak started in Wuhan, China in August 2019. The officially documented first case was on 1 Dec 2019.

This study attracted significant media coverage and political attention (cited by Trump). But from a scientific point of view, its method and argument is really ridiculous. It is criticized by many, but mostly on social media.

We decide to write a commentary paper to criticize the study. To explain our concern and motivation, here is a summary of the conversation between me (A) and a colleague (B) when they tell me about this study:

A: So did they submit the paper to a journal? It will be rejected.

B: They don't need to if the purpose is media attention.

A: But we do have things like PubPeer that allow people to criticize anyway.

(checking PubPeer)

A: OK ... DASH does not assign DOI, so the paper is not indexed by PubPeer ...

B: Yeah, that's how they announce shitty studies now. Post online, news release, then who cares about publication.

A: Not necessarily. We can always write a manuscript criticizing it, and post it to arXiv.

B: Can we? Let's do it.

So I organized a team and drafted a paper. We fact-checked many information, rerun the statistical analysis to spot the main fallacy, and asked many people (academics and journalists) for proofreading. We identified many problems, including statistical fallacies and cherry-pickings.

Our attempts

Our original plan: posting our commentary to arXiv so that everyone sees it. Then we wait for the original paper. If they get published in a journal (to my surprise), we will submit our comment to the same journal.

As a mathematician by training, I'm a big fan of preprints, and a heavy user of arXiv. I usually post to maths category, occasionally to condensed matter physics category. I know that arXiv accepts commentary papers like ours. Since we are criticizing a paper about "digital epidemiology", I checked related arXiv preprints, and determined that "stat.AP" is the most suitable category.

But to my surprise, our submission is put on hold by arXiv. Then the major category is changed to "None". So I wrote to arXiv moderators to explain our choice of category. Then I get the following "rejection letter":

Our moderators have determined that your work would benefit from additional review and revision that is outside of the services we provide. Our volunteer moderators are not referees and do not provide reviews or other detailed feedback for improvement of submissions with their decisions.

As a result, we have removed your submission. Please instead send your paper to a conventional journal for the necessary reviews.

Please do not resubmit this paper without seeking permission and obtaining a positive response. Resubmission of removed papers may result in loss of submission privileges. We will reconsider this decision if your work is published or accepted with a resolving DOI (Digital Object Identifier) or link to the journal's website showing the status of the work.

This was very shocking to me, but I will refrain from commenting on this decision.

We then think about medRxiv and bioRxiv, but they made clear in their policies that they do not accept commentary papers.

We ended up posting the commentary on my personal academic website.

My question

Announcing scientific studies by press release is becoming a trend now.

Imagine someone, out of malicious motivation, decided to post misinformation in the disguise of an "academic paper". To avoid peer review, they do not plan to submit the "paper" to any journal. To avoid PubPeer, they intentionally avoided arXiv, medRxiv, or anything that assigns a DOI. Then they promote their "paper" in media, and gained significant attention. They might receive critics, but only on social media. The critics are mixed up with other misinformations and conspiracies, and will not be taken serious.

In this situation, what can I do as a concerned peer? If I made effort to write a detailed critical analysis, how is that helping if no preprint repositories accept it?

Answer 1

You stated that you "wrote a commentary paper to criticize and rebut it" in order to do something about "bad science".

However, your preprint does much more that that: it does not solely focus on the content of the manuscript, but it attacks the authors!

The most striking instance of that is found in your Section 6:

The authors do not show an adequate statistical literacy. They misused statistical tools on insufficient data and misinterpreted the result, leading to premature claims that only bring media popularity but harm academic reputation. It is a pity that a field at such an early stage already gets plagued by statistical fallacies.The cherry-picking of search keywords for internet trends is a very questionable research practice [21]. We understand that digital epidemiology suffers from insufficient data, as the authors admit in the manuscript. This does not justify the abuse of statistical methods and the suppression of evidence. In the case of insufficient evidence, the only right thing is to refrain from making any claim and simply stop writing a paper.

Personal attacks on authors instead addressing the shortcomings of a manuscript is widely seen as bad science. This could have contributed the rejection by arXiv.

Answer 2

Arxiv normally accepts commentaries. But your commentary is not just a technical criticism of Nsoesie et al. Starting with the abstract, you include non-technical considerations such as "This claim received widespread media coverage despite the lack of validation from peers.", "This review serves as a pre- publication evaluation of the study.", "We also reflect on scientific publishing in a time of public emergency."

This might by why arXiv rejected your commentary. In this case, the solution would be to rewrite it in a technical style, and leave the sociological comments for another venue. (Interviews with the press?)

Also, it would probably not hurt if you framed your commentary less as a denunciation of the errors by Nsoesie et al (since their paper is unpublished, and not even on arXiv), and more as an assertion of your own conclusions regarding the pandemic, even if you only have negative results.