How to overcome discouragement on finding major error in work just before paper submission?

Question

Recently, we were working on a new idea in physics and engineering. At the end (after writing almost 6 drafts and even ready to submit draft) only I realized that I had made a stupid error in calculation and some errors crept up from numerical simulations. So it changed one main result which was the crux of the paper. Now the new result is not appealing as well. I explained the situation to my advisor over email and I am yet to meet him.

I am feeling so hopeless and diffident in the work. Has anyone else faced similar situation? How did you overcome?

We spent about an average of 4 months on this. Do you think I wasted a lot of time of my advisor on this and how can I try not to make errors in the future?

Answer 1

I think that the best way to overcome your situation is to realize that nothing out of ordinary happened - to err is human. In my opinion, research is about discovering truth and enriching knowledge (including the one of the researchers'). And making mistakes is a natural part of the process. I would just discuss my work with advisor openly and do my best to learn from that.

I don't think that you have wasted anybody's time. I (along with many other people) believe that negative results are also valuable (for example, see this paper, this journal and this workshop). The same applies to other results, such as similar to existing results, not impressive results, etc.

As for strategies for not making mistakes, I don't think there are any to prevent them completely, as I said, but you can reduce their probability by not advancing too fast in your research (i.e., rushing to obtain results or to publish) as well as asking feedback on your work from other people beyond your advisor or other people closely involved in the research (perhaps, even, from other disciplines, in order to obtain opinions, based on different perspectives).

Answer 2

Recently, we were working on a new idea in physics and engineering.

I take it that "we" means you and your thesis advisor?

At the end (after writing almost 6 drafts and even ready to submit draft) only I realized that I had made a stupid error in calculation and some errors crept up from numerical simulations.

It's good that you caught it before submission and great that you caught it before publication.

So it changed one main result which was the crux of the paper. Now the new result is not appealing as well.

Do you mean that it is not as appealing as the result you thought you had, or that in addition to having to change the statement of the main result, the new main result is not appealing? Anyway, "appealing" isn't really the standard here. Are you left with a novel, publishable result or not? That's something to talk to your advisor about.

I explained the situation to my advisor over email and I am yet to meet him.

Okay. Of course you definitely need to meet with him. Now for your questions:

1) I am feeling so hopeless and diffident in the work. Has anyone else faced similar situation? How did you overcome?

Yes, so far as I know the majority of working academics have faced similar situations to yours (I, as a PhD student, spent substantial time on work that turned out to be faulty). How to overcome is personal, but very broadly: first you diagnose the situation to see what can be salvaged: very often it's something, and you have to work yourself through the psychological hurdle that what you actually have is "not as appealing" as what you thought you had. But almost the entirety of published academic results are not what the academics really wanted to do: that's the way research works. If you thought you had something and you are sure you don't (but first make sure: a lot of times a mistake turns out to be fixable, and stupid mistakes can be just as fixable as "smart mistakes", sometimes more so), then after a suitable mourning period, kiss it goodbye: there is no point comparing your present situation to the lovely situation that turns out to be counterfactual. Then you pick yourself up and get back to work. It is good to try to identify specific morals. If you can learn anything from your mistakes, then you will have made clear progress, all in all.

2) We spent about an average of 4 months on this. Do you think I wasted a lot of time of my advisor on this and how can I try not to make errors in the future?

You haven't described the nature of your advisor's involvement. If he was not involved at all, then I don't see how you could possibly have wasted his time. If he was involved, then you're in this together, and it's his problem and his mistake as well as yours. Every thesis advisor knows that all researchers are fallible and novice researchers are, on average, more fallible than veterans. A faculty member who writes a paper with a student and just assumes that the student's work is correct knows that he's taking a gamble.

I think this is a good opportunity for reflection in your research process, both individually and together with your advisor. Were each of you assuming that the other person was double-checking the work? That's something to talk about and avoid in the future. More specifics of how to do better in the future -- if there is some dramatic improvement to be made; maybe you just got unlucky -- are exactly what your advisor should help you with. This is worth at least one lengthy conversation. If what you hope to do better in the near future is not clear in your own mind, it may be worth multiple conversations and multiple check-ins regarding your future work over time.

Good luck.

Answer 3

Both previous answers are excellent. First of all, you should be very happy that you discovered the error BEFORE publication. Errors do happen, but you should also acknowledge the fact that there probably was an error in your workflow. Speaking from a CS prespective, for any new problem introduced in a paper, one must clearly describe what are the expected results when solving this specific problem. AFTER clearly defining the problem and its expected results then you must try to find a method that tackles it.

For this reason, many CS papers propose a naive benchmark method for any new problem, which might be super slow but it guarantees 100% that it gives out correct results. Then the same papers propose an "optimized" version that does the same thing, but in a much better optimized way. Although in the paper the selling point will be the fast method (and how faster it is compared to the naive version), during the "debugging" phase the naive version is very important because it always gives you the correct results. That means that everytime you do an optimization, parallelization or something similar to your proposed method, you must always check that you get the exact same results with the naive method. That guarantees that no errors are introduced at any phase of your implementation which leads to a better paper, safer assumptions and correct results.

Speaking from a personal perspective, even after publishing more than a handful papers and having done the coding in most of them (and knowing by now what usually works and what not), any time I add a new major optimization that makes the "optimized" version, e.g. 10% better, I almost always get a subtle mistake in the results (that might give wrong results in 3 out of 1000 experiments). Again, these errors are easy to spot because after any major code change, I always compare results to the naive method. Then I fix the "subtle" bug, get the correct results in all cases and move-on to the next optimization. Rinse, repeat.

Regarding your case, for a bug to go unnoticed for 6 months, it is obvious that you probably had no "naive" method that gave you the correct results to compare your results with. This is a big mistake that is mostly done by inexperienced researchers. We all should have confidence in our abilities to think of a better method to tackle problems in our area of expertise but thinking a correct idea and actually implementing it to be correct are two different things. You should always have a naive method that 100% gives correct results, before starting working on your "optimized" method. This slows down experiments (because the naive method might require several hours to run for all your test cases) but it is your safety belt when you are speeding towards a better solution. That way, errors are easily spotted within a few days or weeks and are easy to fix, many weeks before even starting to work on the paper. Moreover, it shows early if the method you thought has any true potential, before losing another six months to make it better. Also, it makes you confident that what you write in your paper is correct and no unpleasant surprises will happen, when other people try to recreate your results. Thus, next time, when you will work on another problem and a new paper always write a slow, bullet-proof method that gives correct results (checked manually one-by-one) before writing the new optimized version. The will protect you from losing another six months or publishing a paper based on wrong assumptions.

Answer 4

You would be astonished at the amount of time that can be flushed down the toilet during your PhD. However, you have to remember something important: that's what research is. People get some strange idea that we always know ahead of time how an experiment will turn out. Nonsense, of course. If we knew what would happen ahead of time, it wouldn't be an experiment, would it? And every time one of those million dollar instruments goes down and you find yourself on your knees, elbows-deep in the cockles of random, unlabeled wiring, sometimes for months at a time.

Your PhD is not for you to show off how amazing you are to the community. Some folks have everything go right. Others don't. That doesn't make the person who had everything go right have a more valid PhD than the other. Ultimately, your PhD is about making the statement that you are a master in a particular field, and acknowledged as such by your peers. That doesn't always mean success. Sometimes it means tackling a problem in such a way as to make it easier for the next person ahead of you.

We all want to be the person who invents an FTL drive, cures cancer, whatever. Few of us will be. Few will become rich, famous, or particularly successful as a result of our research. That's how it is, and eventually (usually after the PhD) you learn to accept that.

It's hard to say what's best, to write more papers with less content, or fewer that were more powerful. The second would seem obvious, but it isn't. I knew a woman who got her PhD with something like 23 papers during that time. Which sounds amazing, and you would think she would be fast tracked for greatness. And you would be correct, her career continues in the Ivys, last I heard. But other folks sneered, since her papers consisted of "I changed the pH to 6 and wrote a paper" and "I changed the pH to 7 and wrote a paper" and you get the drift. But despite such sneering, she succeeds. So which is better? Clearly, hers was an example of a degree in which lots of things went "right." So what?

Your validity as a scientist has never been and will never be about your results. A PhD doesn't mean you are smarter than anybody. It means you had the luxury to dedicate the time to a degree. What a PhD is supposed to mean is that you have shown yourself trained in a particular manner of thinking. That you don't just plug the numbers in and read off the result. As a PhD, your job is to be he person who squints at the numbers, mutters under your breath, and tears the work even when it is your own and it tells you exactly what you want to hear to shreds.

So, for what it is worth, rejoice! Your post convinces me of nothing but that you are in a good place. Of course, it isn't easy to do-confronting an advisor with a screw up, facing this stuff, it's terrifying, humiliating, painful. But that's what it is to be a PhD, to have that ability to do all of this and walk through that fire, having the integrity to see what is true and what isn't, having the courage to pick your course, and having the strength to go through all of the slings and arrows that come at you when you go along that path.

Answer 5

I had the same experience: on the night before my Master's thesis defense, I discovered a mistake in my statistical analysis that completely reversed the result of the paper. Worse, it reversed the result from something that was "interesting" to something that was only merely a confirmation of what we might already have suspected. I spent the entire night correcting the mistake and reprinting all of the hand-out copies of the paper, and walked into the defense with only a couple of hours of sleep.

I have rarely felt more discouraged and disappointed in myself, especially because the mistake was such a simple one.

The professors had reviewed my previous incorrect drafts beforehand, so right at the start of the defense I pointed out the mistake in the previous version and how it reversed the results. In addition to professors I knew from my department, they had also invited a statistician from the Math department, and as I described my mistake I could see him nodding.

During the QA at the end of the defense, one of my professors, who had always somewhat intimidated me, said (paraphrase from memory): "You could easily have simply ignored this mistake, walked in here and defended the opposite result, and none of us would have known any better. I appreciate your honesty." There's no way to know whether what she said is true -- for all I know, that statistics professor came in having noticed the mistake and ready to tear the results apart -- but her attitude about the issue certainly helped set my mind at ease about it.

They granted the degree.

You haven't wasted your advisor's time. If your result is true, then it is still important whether it is appealing or not. As far as avoiding errors: you'll never avoid all errors, but my own experience did leave me an important tool that has helped me to find errors earlier rather than later in the years since. Whenever I start to feel that a result or explanation is "good enough", but I haven't proven it yet, though the urge is to move on and build on that result, I just have to think back to that all-nighter and the defense the next day to muster the motivation to check my inputs, assumptions and conclusions one more time.

Answer 6

Here are my lessons-learned from a firm where a "mistake-hunting" culture is strong. You can save a lot of time and have a clear mind on more important things, if you simply know or follow these simple rules:

• Errors are human. The more you work, the more you make mistakes. Yet more disturbing: the smarter you are, the more you will make mistakes.
• Ergonomy, structure and good practice will help you avoid making mistakes. For example using an appropriate code editor, write comments, etc. Don't hesitate to write a plan for your code, as you would do for a document, this will help to keep the overview.
• Most mistakes are discovered during a first check, less are discovered during a second check, etc. So, it's inefficient to check again and again, just do a job as clean as possible from the start, and one or two thorough checks.
• Check your methods and your results in a systematic and structured way: list of strategic things to check, what is expected, what is obtained.
• If possible, show your work to someone else (or even have someone do a test independently). It's amazing how fast one can find one's own mistakes, simply by explaining, because it requires to set the mind in a different way.