What should I do after finding out my published results were not substantial?

Question

I am currently working (not in academia) after having graduated from college recently. I hope to apply to PhD programs at the end of the year.

I worked with a professor on a paper about a year ago and it got published in a top-5 AI conference. I was hoping to ask them to write my letter of recommendation when I apply for PhD programs. In fact, my first choice would be to work in their lab doing research similar to what I did with them earlier.

I wanted to explore the idea in my paper a little further and see if I could improve it on my own time. My paper presented a new AI algorithm to do a task that performed better than the baseline method. While doing this exploration, I realized that the results obtained in my paper with my novel algorithm could also be obtained with a slightly different, trivial baseline that behaves randomly. The baseline used in the paper behaved uniformly.

So, basically, my novel algorithm does just as well as a baseline method performing randomly. Nothing in the paper that got accepted to the conference is wrong, but it seems like the new algorithm I presented actually isn't all that effective.

What do I do? I'm obviously hesitant to tell the professor I worked with since they would be my strongest recommender and I want to work in their lab. I also know of other groups that want to use this algorithm. Should I tell them that they might as well use this random baseline?

Answer 1

If the results in the paper are correct, I don't see an issue here.

It sometimes happens that a simple algorithm can do the work of a previously known complicated approach.

If you can prove that the simple baseline method outperforms your previous complicated approach, then any reasonable researcher would see this as a net positive.

This is the point of research - you discover something, it works, then you continue exploring it and discover that something else works better. If I were the professor I would laud you for discovering a simpler approach to solve a problem!

Answer 2

If the original paper was strictly correct, then it should be left alone.

What I would do is continue to investigate your complex algorithm with an eye to determine when it outperforms your second randomized baseline.

Is there some specific type or subset of data where the complex algorithm outperforms baseline 2? What types of inputs break baseline 2, and how does your complex algorithm respond to those? Is there a modification to your algorithm that can make it beat baseline 2 in those cases?

If you can answer those questions, you can publish another paper saying that although simple baseline 2 beat your algorithm, it still works better in these cases or with these changes.

That way you're reporting on improvements and evolution of your algorithm. This is how many things work: AI and artificial neural networks were first proposed in the 1960's and they clearly didn't work very well until multiple decades of advances got them to where they are now.

You also want to report the work in this way to your professor. You can say that it appears our algorithm doesn't work well against baseline 2, but preliminary results show that these changes would fix the problem, and here's a plan for further investigation (if you hire me). That's very impressive as it shows the skills of independence, drive, and "stick-to-it-iveness" needed for a PhD.

Even if your complex algorithm is unsalvageable, you now have a new task: do better than baseline 2. Your next paper could be our complex algorithm A was defeated by baseline 2, but now complex algorithm B does much better than A and 2. If you keep going down paths like this, you now have a career in research.

Answer 3

I suspect some of the other answers are (over)generalizing from other fields. In AI, publishing two papers on something that doesn't work is indeed unfortunate. All the worse if the baseline propagated by the second paper is something that you should have tried while writing the first paper. Fairly or not, publishing a second paper with the "trivial baseline that behaves randomly" is a little like saying "Breaking news: I just realized that the algorithm I was hocking last year, and some of you bought, is total snake oil!" Not a great look, even if you are acting in good faith.

I would consider posting an updated version of the paper on the arXiv. Not a new paper, nor an erratum, but simply a new version with a brief new section showing the new results. You can then write to the other researchers who you know are using your method, and point them to this new version. This is probably the cleanest way to fulfill your ethical obligations while drawing minimal attention to this unfortunate incident.

Answer 4

Write a new paper showing this result

It sounds to me like you have a topic for a valuable new paper. In your previous paper you showed that there is a novel method X that solves a problem better than existing method A. In your next paper you can show that method X unfortunately does not perform better than existing method B, so it does not appear to be effective after all. Both results are publishable and both advance the discipline. Contrary to your question title, both results are still "substantial" --- they introduce a new method and show how it compares with various other methods.

Answer 5

Remember that your previous work was examined by peer researchers during the peer review, and if none of them suggested to try the random baseline you described here, then it means either this random baseline is not that common in your field (in which case your random baseline is a new contribution to the field, or at least to make it more known, so that other researchers will always think about using this strong baseline), or that probably it's common, but the reviewers also missed that (in which case, you shouldn't feel too bad, since it's not only you who missed this baseline).

At any case, one thing that you shouldn't do is to not mention this to your advisor. You should definitely mention this and discuss the implication. Sometimes, when a method that seems good gets outperformed by a simple baseline, it indicates that the thought process of coming up with the model might have something that needs fixing. And this may or may not spark new discussions and new models/algorithms.

I understand that having our work be defeated by a simple baseline feels devastating. We spent much of our time proudly working and proving and testing that method to work, and wouldn't like to see that the method is apparently not that good. But that's okay, we can treat this as getting new knowledge that perhaps our the mental model of how we approach the problem might be incomplete. We can think of this as a possible new direction where we might contribute to.

So, in terms of actions, the normal course of action would usually be:

1. Mention this random baseline with your advisor and discuss it
2. Think of another new method that improves over this random baseline
3. In the paper describing this newer method, also mention the fact that this baseline is quite strong and other researcher should use it. (or, if the task is kinda solved with this random baseline, then you've shown that it's not an interesting task, and maybe move on with other more interesting tasks)

Step 2 and 3 would probably be suggested by your advisor too after you do step 1. =)