How do Academic Journals protect against empirical results given by bugs?

Question

As the title says.

My background is in Economics/Finance(mostly), many topics in those fields (and I am sure other fields) require fairly complicated programming, enough to where one can easily screw something up. How do Academic journals defend against results that are generated by bugs?

As far as I understand nobody ever sees my code, it could be hundreds or even thousands lines of garbage code without a single function in it (not to mention no unit tests) laced with bugs, and I kept "fixing" things until my results "made sense", and then happily reported them. How could a journal tell that my results are trash?

Answer 1

Journals never make any guarantees regarding the validity of the content published in them, though this may seem to be implied. Ideally, errors are caught during the review process. Note that this problem is not specific to programming bugs, subtle errors can occur in all kinds of settings including physical experiments. It is important to always be critical about any results in any article, even highly cited articles in top journals (though these are less likely to be wrong, they still might be).

Letters to the editor are not uncommon in my domain when questionable results are published. In a worst case scenario, published papers can get retracted after the validity of their results has been formally rejected. Retractions for this reason seem to be fairly rare, though.

This is one of the reasons why reproducible results are so important. If several independent researchers seem to reach similar conclusions, they are likely correct.

Answer 2

In addition to Marc Claesen's answer:

• There exist a few journals that require the code to be submitted and where the review explicitly included that code, e.g. the Journal of Statistical Software

• In one of my last papers, one reviewer asked exactly this question. Here's our reply in the text:

  The checks include unit tests to ensure calculational correctness, which consist of ca. twice as many lines of code than the actual function definitions.

  I include this because even if the journal does not have an explicit policy regarding the code, both reviewers and authors can already start with better coding and testing practices: I'm encouraged by this experience to include such statements also in future as author, and I will ask as reviewer.

Answer 3

This is a problem that is beginning to be recognised, and has been described by some as a "crisis of reproducibility". There have been examples of papers in prominent journals being withdrawn after bugs were found in the researchers' code. This article describes some of the problems in more detail.

In my view there are three primary paths to addressing it,

1. Teach scientific programmers good software development practice
2. Make source code and datasets available and citable, with DOIs and with confidence that they will be available and unchanged for the long term.
3. Get journals to require that source and data are available (although comprimises must probably be made where data is commercially confidential), and that reviewers conduct code review. This may not be straightforward since (as noted in the Post & Votta article linked above) it will not always be possible to find one reviewer who is qualified to review the code and sufficiently expert on the science involved. (and that's before we even consider how long code review might take!)

The Software Carpentry project is aimed at addressing point 1 and 2 above, and may be of interest.