I just realized that my published papers are based on a buggy version of my dataset (due to a SQL error). How to proceed? How to tell my advisor?

Question

I am in an advanced stage of my phd, finishing next year. I have 2-first author publications in which I used data from my group. My research group has different datasets stored in MySQL server, in which each researcher accessed their relevant dataset.

In my 2 previous publications, I exported my dataset into a csv file and worked with it. During this data export, I had to perform a couple of table join operations, involving filtered data in a kind of "summary" table, joining it with a table containing original raw data.

Last week, I was working on my third review paper, and I needed to get some statistics from the MySQL server, when I discovered that the dataset I used in my previous research was actually incorrect. Incorrect in the sense that I had an error in my data filter SQL query (used to export it), therefore the exported dataset was incomplete, kind of a "subset" of the actual data.

All my conclusions in previous work were drawn from this incorrect data. I am in panic, and don't know what to do. I am not sure what the reactions of my supervisor would be, if I explain this problem to him. He is such a person that expects "near perfect" from his students.

I am now sure how this will affect my PhD. The years and efforts invested are likely in vain. What is the best course of action?

Answer 1

Oh no! This is bad news for sure, but try not to panic.

The main thing is to rerun your analysis with the corrected dataset. Hopefully, you have been organizing your work in such a way that you can just replace the csv file, push the button, and have results quickly. If not, this is a good opportunity to improve your old code such that you can quickly reproduce the old numbers and then produce the new numbers. If possible, I would try to do this quietly (i.e., before telling others of the potential problem).

Once you have these results, you'll be in a better position to talk to your advisor. If you are lucky, your results will be the same (or even better, since you have more data) and it is just a matter of updating the paper without changing your conclusions. This shouldn't be too much of a problem. You should still tell your advisor, as a corrigendum may be required, but his anger should be limited.

You should also consider approaching this from the other side -- what claims did you make in the paper about your dataset? If you claim to have used, say, ImageNet, then there is not much wiggle room. But if you claim to have used "500 original datapoints from a medical database," then you might be able to simply replace this with "300 original datapoints from a medical database" and there is no need to change the papers' results or conclusions.

Of course, it's also possible that your bug biased the dataset in such a way that your conclusions are no longer valid. This would put you in a much more serious situation for which there is not much advice to give -- all you can really do is tell you advisor, show him the before/after numbers, and hope that his reaction is reasonable. The way forward will depend heavily on the technical details of your paper, so only your advisor's advice is really valuable in this case. Good luck.

Answer 2

Don't worry, everyone makes mistakes. Identifying and correcting mistakes is a crucial component of academic research and developing new theories.

From my experience, as long as you correct the mistake in a honest way, the journal will not retract your paper and your PhD will be unaffected.

1. Most papers are force-retracted because of ethical problems or political problems. No paper from my knowledge is forced to be retracted due to honest mistake (unless you require it)
2. If mistakes are identified, the best scenario is that you correct it by yourself. The second best scenario is someone else find it first and correct it for you.
3. Even if someone else find your mistakes, your paper will usually not be retracted. For example, see this example in the most rigorous journal in economic statistics: https://www.econjobrumors.com/topic/kasy-and-sautmann-econometrica-2021-proven-wrong/page/16

Here is what I would do:

1. I would use the corrected dataset to rerun all the analysis and start writing a corrigendum or addendum.
2. For some results that still hold, I would write that those conclusions are verified with the new dataset.
3. If some new results are different from the old results, I would update the new version of the results in paper, explain why the new results are correct in this scenario, and write a short discussion on under which scenarios would the old results likely to hold and under which scenarios would the new results are likely to hold.
4. Finally, you can post your addendum/corrigendum online and send it to the same journal.

Answer 3

First of all, don't panic.

This is not necessarily a fatal flaw. You said that the analysis was done on a subset of the data. Rerun the analysis on the full data set. If the conclusions still hold and follow from the hypothesis, your work will not be substantially affected. You can revise the paper even after its been published. Mistakes are happening in science. This happens particularly in the more difficult fields like theoretical math, where the results are very difficult to understand (even by experts).

The only way you are in trouble is if the conclusion is changed.

In any case, its better to find the mistake yourself than to have someone else find it and rub your face in it.

Regardless, you are going to have to tell your advisor. These things happen though. You aren't the first person to have this happen to them. At least you are still in your PhD program. You can at least fix everything before publishing your thesis.

You need to go about this the right way. The worst thing you could do right now is to go into a full blown panic and start calling everyone you know, including the journal and telling them you made this terrible mistake. Take some time and understand the mistake. Try to understand if it can easily be fixed or if the entire paper is going to change. Once you do that, then go to your advisor first with the complete analysis of the situation. Due to the highly political and sensitive nature of the academic world, you need to handle this correctly and carefully. If this is a minor error, it can likely be corrected with a simple revision and it should not be blown out of proportion. I would recommend going through the appropriate channels and deferring to those that have more research experience on how to correct this mistake correctly.