Why isn't implementing scientific libraries in software considered equivalent to a publication?

Question

In academia, researchers live by the mantra of "publish or perish". This is because the contribution to society that academia gives is the expansion of human knowledge.

However, given the importance of software to conduct research, I don't understand why implementing many algorithms or building API's for research purposes is not equally relevant when trying to get funding.

See, for example, how few papers in math, computer science and software engineering provide source code, despite the entire focus of the paper being on the implementation of an algorithm and its performance in real hardware.

Moreover, see how much research relies on API's. We have linear algebra API's, calculus, computer visions, physics simulations, etc. Many of which are paid software (e.g., Matlab).

If someone is willing to spend time developing an API for a given subject (a computer vision library, a rendering algorithm, quantum physics / computing simulations) and that software provides new and useful tools in open software and for free so that researchers can conduct their research more efficiently, I don't see how this is not equivalent in usefulness as making a paper.

Answer 1

Software is a research product has value and is seen as valuable, but you as a researcher need to ensure that you extract the value out of it. Extracting value out of any research product requires doing some kind of formal write up and distributing the ideas among the relevant audience. For software, this is many times in the form of a software publication. Many traditional scientific computing journals take a software paper and these papers can do quite well in terms of citations. For example, Shampine's "The MATLAB ODE Suite" is one of the most read and cited articles in SIAM Journal of Scientific Computing.

Taking that paper as a reference, notice that it identifies the problem it's solving, explains how it had to modify the algorithms to achieve its goals, and the kind of results it can achieve. This then gives a citable resource and many of the people who use the paper will then cite this source. Software tends to be under cited, but since it can get so widely used you will find that successful software has some of the most cited publications. Another classic journal like this is ACM Transactions on Mathematical Software (ACM TOMS). More recent software journals are the Journal of Open Research Software and SoftwareX.

In a slightly different direction, the Journal for Open Source Software reduces the importance of the paper by making the software and its documentation part of the peer review. As an example, Optim.jl has a JOSS paper which you can see is much shorter than traditional academic papers, but at least summarizes the value of the software in a citable form.

When applying for jobs, you'll notice that people naturally assign a different value to software papers. While it is definitely changing, some professors I know take hardline stance that software is just implementation and doesn't extend knowledge and will value it less than other papers (which is why these exact points should be refuted in a publication about said software). In contrast, some individuals care more about impact or citations, and so a widely used software will have a very large effect on them. Generally in my experience, the value of software seems to be increasing over time, with senior professors more likely to take the former stance and newer professors more likely to take the latter.

But my advice would be to just do whatever you think has a high impact to the research community. If it does have a big impact, people will notice, and sometimes just being noticed is what you need.

Answer 2

You are mistaken in your assumptions. Scientific software is valued and there are now in fact a number of journals dedicated to exactly this -- I know this among other reasons because I happen to be the Editor-in-Chief of the ACM Transactions on Mathematical Software (ACM TOMS). There is (actually: was) also the Computational Science and Discovery journal, and then there is Computing in Science and Engineering (CiSE). All of these publish articles that describe computational software.

More generally, there are substantial efforts at making sure that the authors of scientific software get credit that is comparable to other kinds of publications such as journal papers. I would encourage you to take a look at the WSSSPE series of workshops, the Force11 task force, and a number of other efforts to ensure that software can be cited in much the same way as papers can (e.g., via the Zenodo service).

Answer 3

I am not sure what field you are in, but to me, this question almost doesn't make sense.

Nothing is equivalent to a paper ... except a paper. You may as well ask why doing research without publishing it isn't equivalent to a paper. That's just not the model we have.

In my fields, if you publish a library/toolbox, you write a paper on it. If you put up an open dataset, you write a paper on it. If you do research, you write a paper on it. There are many people who focus on developing toolboxes, and many people who might never touch development at all and will only ever publish "actual" science.

In other words, it's not validated and "counts" until it's been at least somewhat peer-reviewed.

Answer 4

There are different kinds of value. And different people measure it differently.

Scientific research resulting in traditional publications is about extending knowledge. The value is in the knowledge.

Implementation is normally about building a product of some sort, whether commercial or not. But to implement something for which the knowledge is already present, doesn't extend knowledge, so isn't the same sort of value as research resulting is new ideas. It isn't less valuable, just differently valuable. An implementation may enable something else to be done, without extending knowledge. Building out scientific libraries feels like this sort of thing.

But you are wrong in some ways in your assumptions. Some people get at least a local reputation boost by building stuff. That is certainly true in IP based businesses such as Google, but also in some universities. It isn't either-or. I've been given props for some of the stuff I've built whether it was, strictly speaking, research or not. My stuff mostly enables students to learn something, rather than knowledge extension per se.

But also, a lot of doctoral research in CS is, in fact, based on implementation. One famous doctoral thesis explains how to build a compiler based on the semantics of a language rather than (the usual thing) it syntax. It was a big deal. But Peter Lee, had to build it to demonstrate that the idea was sound. The implementation wasn't the dissertation, but the research and the extension of knowledge that the implementation proved sound was. Also in the compiler field, if you have an idea to make a garbage collector for a new language run 3% faster and use 3% less memory than the best known solution you can get a doctorate. But you have to build it and measure it (extensively) to prove the idea.

So, the implementation may not be, in itself, the valuable thing, but it enables the verification of a new idea - knowledge extension.