In performing a comparison, how can I get authors to actually respond to a request for source code?

Question

I'm doing my PhD in computer science and am working in computer vision. I have come up with an algorithm that my supervisors consider to be promising and publishable. It took me a long time to get these results.

Now the thing is that I have to compare my code to other recent papers tackling the same issue. Which I understand is very necessary as to show how my work compares to previous work. My supervisors want me to compare against at least 4 or 5 other recent papers.

The problem is that the database I am using is very recent and no journal papers have till yet used it. So the other solution left for me is to read the journal papers, understand it and try to implement their code.

This will definitely take way too long and in my opinion waste a lot of time. These journal papers are very advanced (obviously) and implementing their results on my own will take up a lot of my time whose only sole purpose is to get a result.

One solution would be to email the authors and ask politely for their source code, but I found out that many authors don't reply.

This question Can I request the code behind a research paper from the author? stated that they were more likely to get a response if they promised to add that author as a co-author in their paper. I do not want to do that as that would be being dishonest as I see no sense in adding the author to my paper just because I compared my work against his, and if I am to compare against 5 papers then that will be a very long list of authors.

Maybe I am asking for something that is frowned upon. Or maybe not asking in the right manner?

I emailed the authors and asked for their code solely to test their work on my database for comparison purposes, which gets no replies. Am I doing something wrong?

Answer 1

I'm doing my PhD in computer science This will definitely take way too long and in my opinion waste a lot of time

Well, that is what a PHD in CS in experimental algorithms and areas is all about. You must prepare your algorithm, implement it, implement previous works and compare your work with them. So, how much time it will take you it is of no interest to anyone else but you and your supervisor. So, this line of argument is naive.

One solution would be to email the authors and ask politely for their source code

Yes, but it not the only one. You could ask for their datasets and run your algorithm on their datasets, instead of making yet another dataset. Moreover, experimental algorithm communities have well-known benchmark instances and all related papers work on them for easy comparison of results. Why do you need to build yet another dataset? It is OK to use this extra dataset AFTER you tested your algorithm on those community benchmark instances. You could also send your datasets to your "competitors" and ask them to run their experiments on your datasets and just give you their results. Prepare you experiments relatively to older papers, notify them of your PC specs (and provide alternatives PC configuration - you probably have different PCs on your lab) and tell them to repeat all your suggested experiments on an PC close to your suggested specs.

If you like so much to share code, you can also send them the source code of yours, provide explanation how to compile it and use it and then let them run experiments on their PCs and give you the relative results to their work. I know that this thought probably never crossed your mind. Why? "They might steal my work, how do I know they will give me correct results, it is too much work to do so, I don't trust them with my src code". And now you know why people do not want to share their code.

But you also probably forgot the most easy way out of your problem. Let your supervisor contact the first author AND the rest of the authors. Unless you are an exceptional PHD student with many amazing papers you are practically Mr Nobody and people will easily brush off your requests. It is harder to do so to your supervisor (unless he is Mr Nobody as well). Usually people do not want to say NO to future reviewers, collaborators and respected members of the community. Also it is important to CC all paper authors. The first author (PHD student) might be protective of his code and hide your request from his supervisor. If you CC the supervisor, he might be forced to share his code or at least reply.

Last but not least: Be nice when asking.

Answer 2

Reproducibility in computer science is an important area for improvement. It is reasonable to expect authors to to make source code available for algorithms that they present. I've heard that some journals and conferences are putting pressure on authors to make source code publicly available (although I haven't personally encountered that).

In my experience, most scientists do make their source code available. Not doing so both casts serious doubt on the legitimacy of their result, and is counter to the normal premise of public-sphere science. It's also pretty tough to to justify, since sharing source code has no cost associated to it.

Being a PhD in computer science is most assuredly not about re-implementing existing algorithms, except maybe for pedagogical reasons. You're right that it's a waste of time. There is simply no good reason to do it when the code could be hosted on github for free!

A good strategy is as follows.

• Start with the corresponding author, or first author. Tell them what you're doing, that you intend to cite their work, and that you'd like to benchmark your method against theirs. Do be cordial, and feel free to tell them what you like about their work. Most likely they're a cool person.
• If they don't respond after a reasonable delay, say a week, send essentially the same email to the entire authors list, explain that you're having some trouble getting in contact with the corresponding author, and inquire about who is in the position to speak on behalf of the group. This has yet to fail for me.
• They should provide their source code at this point. If they don't, it's fair game to inquire as to why not. Since it's published work, there's sort of a premise that they should make it available except if there's something really making that a problem. It's pretty fishy if they don't. You can also remind them that you wouldn't want to make any mistake while implementing their work, so that the comparison is legitimate.

Don't worry about being a "nobody". The idea that a researcher should only respond to high-status people is just ridiculous. What you do need to be aware of, however, is that a prominent researcher can receive a lot of incoming communication. If they're professional, they'll have set up triaging for that. The best thing to do is make it easy for them to respond, and they will probably be pleased to help you, a student, which they once were.