Has Stack Overflow saved billions of dollars in programmer productivity?

Question

John Carmack is a renowned developer and CTO. His Twitter account has over 100,000 followers.

On the occasion of Stack Exchange's 5th anniversary, he quipped on Twitter:

[... Stack Overflow] has probably added billions of dollars of value to the world in increased programmer productivity.

The tweet was widely read and retweeted over 100 times.

Is this claim supported by evidence?

---

What have I tried?

I've made a quick calculation to see if he was completely off the mark, but the numbers seem to add up to a layman like me.

• Our visit counter recently overflowed Int32: 2,147,483,647.
  _{The above is a reference to my own tweet but I can also provide supporting evidence: the site is 5 years old and currently does around 6m visits/day}
• Average pay for a programmer in the UK is 45,000£/annum (source) or 22.5£/h. Let's assume a low pay of 20US$/h. I assume an hour is saved every time an answer is provided.
• Each visit will potentially save some money, but only 77% of visits land on answered questions
• There is a cost relative to answering a question, however answers are useful to hundreds of people.
• There are many other factors I am not calculating: for example, does answering a question make you more knowledgeable? Does finding an answer make you less knowledgeable than you would be otherwise? I am ignoring these as this is a ballpark estimate, certainly not a valid answer to my own question.

To sum it up:

• value provided = 2 billion page views * 77% answer ratio * 20US$ ≈ 30 billion US$

What kind of answer do I expect?

I expect answers to provide evidence to the value (or lack thereof, or cost!) of the increased (or decreased!) programmer productivity as to ascertain whether it is true that they are "probably billions". I do not really want to see calculations like the one I provided.

Disclosure

I work for Stack Exchange.

---

A Rather Unprecedented Moderator Note from @Oddthinking

This question has caused a startling amount of interest and discussion. A big welcome to the users from Hacker News and Reddit.

Everyone seems to have an opinion. Unlike those sites though, opinions generally aren't welcome here - as answers or in the comments. We are looking for definitive answers based on empirical evidence. Comments should be directed at improving the question, not at discussion.

This is a tough question. It isn't obvious how economists might measure value to an economy. It isn't obvious how computer scientists might measure productivity. It isn't obvious how much time it takes to answer or how much time is saved for each page view. There may be unintentional side-effects of making answers easy to find. etc. Arguments from incredulity are not a valid response to such difficulty.

Sklivvz has given openly naive answers to those questions, as a back-of-the-envelope calculation, and is asking if anyone has done the calculation professionally. If you can find such calculations in the literature, please answer the question. If all you want to do is point out the naive calculations are naive, let's take that as stipulated; it doesn't help answer the question.

In the meantime, the question has been "protected", so only users with a modicum of rep on the site may answer, and the comments have been cleaned up several times. If you want to comment on this mod note, please take it to meta.

Answer 1

One of the few published papers on the relationship between Stack Overflow and productivity finds that active GitHub users "ask fewer questions and provide more answers than others" on the site.¹ The authors write that users who ask questions "distribute their work in less uniform way," than those who do not (indicating interruption). But, they also note that "despite interruptions incurred" Stack Overflow activities correlate positively with participation and commit rates on GitHub.

In other words, there's evidence that Stack Overflow tends to cause interruptions (as a stand in for loss of productivity),² but people who code a lot (on GitHub) also tend to contribute to SO, and the other way around. The relationship between GitHub and SO participation is in this way used to approximate impact on productivity.³

Neither this study nor the studies it cites provide direct evidence for or against Carmack's claims. Fundamentally, the authors conclude that:

Despite the popularity of Stack Overflow, its role in the work cycle of open-source developers is yet to be understood: on the one hand, participation in it has the potential to increase the knowledge of individual developers thus improving and speeding up the development process. On the other hand, participation in Stack Overflow may interrupt the regular working rhythm of the developer, hence also possibly slow down the development process.⁴

Because the original claim argues for value through productivity, under conditions where the impact of SO on productivity "is yet to be understood" in systematic study, it (the original claim) remains speculative and not supported by evidence beyond anecdote.

1. Vasilescu, B., V. Filkov, and A. Serebrenik. “StackOverflow and GitHub: Associations between Software Development and Crowdsourced Knowledge.” In 2013 International Conference on Social Computing (SocialCom), 188–95, 2013.
2. "Goal: Is participation in Stack Overflow related to productivity of GitHub developers?" (p.188)
3. "Here, following Adams et.al., we look at only one, but representative, facet of productivity."
4. Emphasis mine.