Should I supply code along with my paper submission?

Question

I am submitting a paper to a machine learning/computer vision conference. Is it worthwhile to include some source code of my implementation so that the reviewers can test it themselves or gain a greater understanding of my idea? It will take time to clean up my code and make sure it is anonymised.

Is this a common practice? If so do reviewers pay attention to the code?

Is it worth the time to prepare anonymised and clean code?

The code will be included as a ZIP archive through the paper submission website. After the paper has been accepted I intend to release it with an open source license.

Answer 1

Should I supply code as supplemental material?

Yes, but it would be better if you published it in an open access fashion.

Is this a common practice?

No.

If so do reviewers pay attention to the code?

Depends on the reviewers.

Is it worth the time to prepare anonymised and clean code?

That depends, don't do more than necessary.

Also read:

• Ten Simple Rules for Reproducible Computational Research
• Top Ten Reasons to Not Share Your Code (and why you should anyway)
• Open Source Software Publishing for University Researchers

Answer 2

Making code available is always a good idea, as long as the code works. It is probably a good idea even if it doesn't, since even non-working code can help reviewers and users/readers of your work to understand details of your implementation that your paper does not cover. However, if the code does not work, you should clearly indicate this in the source. In the case of working and tested code, bear in mind that sufficient documentation to run the code is highly desirable. You only mention reviewers, but you should also be thinking about general readers of your paper.

I think it is probably reasonable to supply a current snapshot of the code to the paper as a zip archive (or similar) for reviewing purposes, but why not just put it online directly as a Git or Mercurial repository on Bitbucket, Github, or similar, and reference this on the paper? I also recommend making repositories available in more than one place, in the interests of redundancy. For example, I have used both Bitbucket and Google Code for my Mercurial repositories. This has various advantages over a zip archive file; for one thing you can push corrections and other changes to your repository, and everyone will immediately have access to them.

If you are concerned about releasing your code before your paper has been published might mean someone else will "scoop" you, that seems unlikely to me. At least, it is not something I've ever worried about.

Answer 3

Absolutely, yes.

If you don't release code, then your work is not de-facto reproducible; and if you will get me as a reviewer, I will do everything in my power to either get you to publish code, or get rejected. Rare exceptions to the contrary, where an implementation is trivial. But implementations are rarely trivial; and often a great place to sweep a lot of quirks under the rug which conveniently arnt mention in the paper.

Granted, it is not hard to get away with not providing code; journals are particularly lax at enforcing or even adopting standards, even though most people in the field will decry the lack of reproducibility of published work. But nobody will resent you for providing code either. Writing code you actually feel comfortable releasing out into the open requires work of course. Time which you could spend doing other things. If you don't feel comfortable zipping up your code and providing it right now, that's probably a good indication that any papers written about said code arnt ready to be published yet either.

Unfortunately, there is a tension there, between doing what is right scientifically, and doing what is right for your career. But as for me as a reviewer; no code == no publication.

Answer 4

This largely depends upon the reviewers and the field customs. Some reviewers, specially in journals, may ask for source code. However, it is unlikely that in Computer Science conferences reviewers ask for source code specially when the paper is well-written.