Should I make my code and data avaiable online when the article is not accepted yet?

Question

We have received a set of reviews for an article that we had submitted to a journal. The decision by the journal editor is a Major Revision. That means, there is a possibility that the article could still be rejected.

Two of the reviewers (out of 3) have commented that the data and code to be made available publicly. Sharing of code and data is a very common practice in computer science conferences; it is not very common for journals.

I have the following confusions:

• Should I make my data and code public straightaway?
• Should I mention in the response letter that "Our code and data will be made available at this link after the article is accepted"?

(There is a relevant question here that looks it from the reviewers' perspective.)

Answer 1

Ultimately, it's up to you. I do not really see a reason to keep your data and code private, but you may very well have a valid reason to do so. Hence, I cannot answer the question what you should do.

If I were in your position, I would do the following. I would include a link in the manuscript to the data and code, and write that at this link you make the data and code available for reviewing purposes. The reviewers can then access it all so that they can fairly review your paper, code, and data, but if they behave responsibly, none of it should leak to the wider world. I would add a footnote specifying that the code and data will be made available to the public upon paper acceptance.

Answer 2

There are of course pros and cons and various personal preferences, but here is what I would do based on my experience as both author and reviewer:
It is probably relevant for the reviewers to be able to see your code and data. Whether they actually take the opportunity to examine it more closely is up to the journal's policy and probably the reviewers' level of engangement and available time. In my opinion, we should always make our code and data available if at all possible, so since we intend to do that, it should also be available to the reviewers.
Now as you mention, the paper may have to be revised and that could mean the code needs to be changed a bit as well. That means several versions of the code could arise during the review process and that complicates the "history" of the code in the end with respect to which version of the code is the "correct" one. This could be handled by sharing the code "privately" with the reviewers during the review process on a personal homepage, a Dropbox shared folder or something else where you can easily remove it afterwards.
A platform such as Zenodo offers free deposition of code and data. It is possible to automatically deposit specific releases of a GitHub repository as individual "versions" of code on Zenodo. Depositions in Zenodo are equipped with a DOI which makes it easier to reference and find the code/data at a later point.
An example from a publication I was involved in a few years ago can be seen here:

• Code + data (first version from submission)
• Code + data (final version after first round of review)

At the time when we published this, we did not have (or did not know we had) the possibility of versioning the dataset, but nowadays Zenodo has a "new version" feature where these two pairs of depositions could have been created as two versions of the same pair of depositions.