2020 Developer Survey results feedback

Question

Depending on how you count it (there were some beta tests and various versions) this is the 10th year Stack Overflow has been conducting its annual developer survey. The software industry has changed substantially over the last decade, but it’s also true that no single technology has been quite as disruptive, at least in the short term, as the public health crisis the entire world is experiencing right now. We’re proud of the way our community has been responding.

The results of this survey reflect the opinions and experiences of nearly 65,000 developers. It’s important to note, however, that the survey was conducted in February, before COVID-19 had been declared a global pandemic, and countries across the world had gone into lockdown. We’re eager to share with this community and the public some of the interesting statistics and changes reflected in this data, but we also understand that it’s important to be humble and realistic: a lot of the answers developers gave might look very different if the same survey were conducted today.

That said, there are plenty of exciting, interesting, and amusing highlights from the 2020 Developer Survey, so let’s dive in. You can view the key results here. As always, we’ll make the anonymized results of this year’s survey publicly available under the Open Database License (ODbL) shortly. There are sure to be some insights you’ll glean that we missed.

It seems that significantly less people took the survey this year compared to last year. — NoDataDumpNoContribution, Commented May 27, 2020 at 19:46
Can someone with more understanding than me explain how something that 97% of respondents knew nothing about took "most lovable" (Rust)? How can 3% take out Python or Typescript? What was the question? — Nathan, Commented May 28, 2020 at 1:37
@Trilarion I certainly didn't, and I have been happily and consistently taking it for years until this one. SE Inc. got so worried about making new users feel welcome, they took the existing userbase for granted. — Boaz, Commented May 28, 2020 at 11:11
@Nathan the survey results page explains that question fairly well: "% of developers who are developing with the language or technology and have expressed interest in continuing to develop with it". In other words, people who have used Rust are very likely to want to continue using it. These are ratios of the developer base rather than absolute numbers. — E_net4, Commented May 28, 2020 at 11:17
@E_net4theRustacean They should probably just change the title from "most loved languages" to "most loved languages among its users". It could be that Rust is so special in the way that almost all people who dislike it, really don't use it and not in the way that people generally would like it (if they would know it). Might be or might not be the case.If Rust became more often used, satisfaction with it might drop (and that might not even be the fault of the language). — NoDataDumpNoContribution, Commented May 28, 2020 at 11:24
I would love to read results of the survey, but you have used dark theme so I can't because of my eyesight issues. — Dalija Prasnikar, Commented May 28, 2020 at 12:12
First year I didnt participate. I think the survey was going on during Monica-gate and I was just really annoyed with how SO was handling the situation and communication that I didnt feel like taking the normally very interesting survey. @Trilarion — TomE, Commented May 28, 2020 at 21:48
@Trilarion That would be the case for me. After filling the survey a few times in the past, the novelty has worn off and I didn't feel like doing it this year. — Pooks, Commented May 29, 2020 at 5:56
For anyone (like me) who forgot the question was asked last year, here's a question about what "weighted by gender" means :). — Heretic Monkey, Commented May 29, 2020 at 13:59
@tri The 2020 Developer survey had "nearly 65,000" participants. The 2019 Developer survey "nearly 90,000" That's a difference of 25,000, more than a quarter. I doubt we'd need to have to wait until next year to see if this is a trend or just statistical noise. This is an undeniable collapse. — IInspectable, Commented May 30, 2020 at 17:50
I did complete all the answers, but didn't submit it, because the questions were all just meh, it seemed to have lost the fun element (tabs vs spaces, etc), which rather reflected the mood of the company towards its users. The results seem to reflect that too. — DaveyDaveDave, Commented Jun 1, 2020 at 6:04
Not like any more anecdotal evidence is needed, but I have refused to take the survey this year over SE inc's overall policies and ideas, on a variety of levels. Which is just a side effect of a (what should be) far more worrisome development: I virutially stopped my SE and SO participation for that reason, AND actively worked IRL to ensure SE commercial products would not be used at workplaces where myself or people who listen to my opinions work. — DVK, Commented Jun 1, 2020 at 14:52

Cody Gray - on strike · Accepted Answer · 2020-05-30 01:56:42Z

66

I'm still reading through the results, but this one stuck out to me because welcoming new users has been important for a long while now. So let's start here:

More than 15% of people find Stack Overflow at least somewhat more welcome than last year. We still have work to do, but it’s a start.

On the surface, that's great. Improvement!

But, it's hiding some other details with that summary.

The blurb with that graph is misleading too:

Most respondents say they feel about the same as last year, but respondents are twice as likely to say they feel more welcome than less welcome.

This section doesn't have the total number of responses like many others do so I can't do exact math, so forgive the rough numbers.

Assuming 55,000 people responded to this question:

At least 8,200 people are more welcome (~15%)
At least 5,500 people are less welcome (~10%)
Everyone else is a nice solid "Meh".

Where does the "respondents are twice as likely to say they feel more welcome than less welcome" come from? What's the math going on here?

At the cost of welcoming people we unwelcomed about 2/3 of that number. Has any research gone into why? Does it matter since "more welcome" out paces "less welcome"?

We still have work to do, but it’s a start.

Welcoming users has been a big deal for two(?) years at this point. What is the goal Stack Overflow was trying to reach with "more welcome"? Is 15% of users within that goal? Is 70% unswayed by the efforts within that goal? Is 10% less welcome within the goal?

I'm sure these numbers are driving some discussion internally. What can we expect in the future related to the welcoming initiative, based on the results here?

Update: See "official" response from Jason Punyon.

edited May 30, 2020 at 1:56

Cody Gray - on strikeMod

243k84 gold badges718 silver badges762 bronze badges

answered May 27, 2020 at 14:50

AndyMod

50.2k24 gold badges135 silver badges182 bronze badges

The intention of mentioning it might be some kind of confirmation that SO is on the right track. I guess it does not include some concrete goals achieved. It's only a relative statement. Feeling more welcome could include still feeling not very welcome.
– NoDataDumpNoContribution
Commented May 27, 2020 at 19:40
20

As someone in the less welcome group I will just state that all the reasons you quoted happened as a direct result of SE trying (and failing) to be welcoming (M was fired due to over-reaction on a "welcoming issue" at the least) so based on your data I don't see what you are trying to say @JasonPunyon. Note, its been at least 5 years with the welcoming issue or maybe I justl had more hope back then
– LinkBerest Left - SO is AI Now
Commented May 27, 2020 at 23:43
15

The number-one word used by folks who feel more welcomed was emails? If that's implying that somehow people feel that emails are welcoming, well let's just say that comes as a major shock to me. I find emails to be one of the most annoying things that a site can do to me. I certainly do not see why someone would find spam to be welcoming. This may, therefore, suggest that the "word clustering" method of interpretation is misleading, if not outright wrong.
– Cody Gray - on strike Mod
Commented May 28, 2020 at 0:36
12

I also just realized I did not take the survey this year (they finally added educator too :\ ) - so unwelcoming = people who found it "so unwelcoming they skipped the survey entirely but that [the answers you gave in your clustering] would have been their answer to why it was less welcoming" needs to be considered as well (since there are less results then usual) @JasonPunyon
– LinkBerest Left - SO is AI Now
Commented May 28, 2020 at 1:08
1

@CodyGray " "word clustering" method of interpretation is misleading" It's certainly not the same as meaning. Somebody with enough time to read the thousands of free form answers to this one question would probably come up with a meaningful result, but the word cluster alone should not get too much importance.
– NoDataDumpNoContribution
Commented May 28, 2020 at 6:48
1

"Where does the "respondents are twice as likely to say they feel more welcome than less welcome" come from?" Are we comparing against the last survey? That's the only way to establish a benchmark/denominator.
– Braiam
Commented May 28, 2020 at 10:36
@JasonPunyon can the word analysis be done in two words groups, with synonyms? It would be more expensive/complex, but more illustrative of what people were thinking of.
– Braiam
Commented May 28, 2020 at 10:41
5

Well @CodyGray, who could have thought that. Looking at those samples we can conclude that our mutual distaste of emails and constant push to get less of them is actually driving people away. On the plus side, they didn't mention hand written letters ...
– rene
Commented May 28, 2020 at 17:35
@CodyGray based on the (now) linked email, the people who find it welcoming as the sort who may have forgotten about SO, and then been 'welcomed back' by being reminded via email... Thats probably not common among people who would come back anyway.
– AncientSwordRage
Commented Jun 1, 2020 at 10:04
@CodyGray "I certainly do not see why someone would find spam to be welcoming" I'm sure, phrased that way, 100% of respondents would agree. But that's a mis-framing of the issue; the emails are not spam; they're sent to registered users who are opted into emails (or have not opted out), are directly relevant to the site sending the email, the mailing list isn't shared with other parties/external companies, etc. Calling these emails spam is... highly inaccurate with regard to what spam is. tl;dr Just because you don't want to receive an email doesn't mean that email is spam.
– TylerH
Commented Jun 1, 2020 at 18:29
4

@TylerH I recently got an email from that company that did not feel at all "welcoming"; in fact, it felt like a slap in the face. I have the option to receive emails from Stack Overflow turned on so that I might receive important announcements. I never intended to receive garbage like this. I think it's completely fair to call it "spam" when SO blasts gobs of long-established users with tips on how to use the site because they've asked a question in the last year.
– Cody Gray - on strike Mod
Commented Jun 2, 2020 at 1:48
1

Am I the only one who sees a giant middle finger flipping me off when looking at that graph?
– Eaten by a Grue
Commented Jun 2, 2020 at 11:54
1

@CodyGray That's still not spam. Take a look at this email section of your profile. If you have that second option turned on, then you are going to receive such content; important announcements are in a separate email category. Now, if you have that second option turned off and still got such an email, then that's a valid bug report. But it doesn't make it garbage (keep in mind, one man's trash is another man's treasure).
– TylerH
Commented Jun 2, 2020 at 13:57

Add a comment |

Restore The Data Dumps Again · Accepted Answer · 2020-05-30 01:34:39Z

In response to Andy:

First off, thanks for the heads up on the copy. The "twice as likely" bit is wrong and it's my fault. I'm responsible for the numbers and the numerical statements this year and I just didn't proofread that one closely enough. I'm going to rewrite the summary to that question and have it up on the survey site as soon as possible (it will probably be tomorrow). I'll be more even handed about the results (upside and downside). (Update: The rewrite has been deployed)

The question as asked was...

...and we asked a follow up...

We got 52,683 responses to the main question, and 12,272 people answered the follow up. If we take the people who answered the followup and break them down by their main question answer we get the following...

The answerers to the followup are more polarized than the main question answerers ("Just as welcome" has a lower representation among the followup answerers), and more heavily weighted towards the less welcome side of the spectrum. Respondents were more willing to tell us why they felt less welcome than why they felt more welcome.

At the cost of welcoming people we unwelcomed about 2/3 of that number. Has any research gone into why? Does it matter since "more welcome" out paces "less welcome"?

I wanted to look at the reasons why people felt more welcome and more unwelcome and figure out if your statement was true or not (that we welcomed ~15% of respondents at the cost of ~10% of respondents). I filtered out the responses from the "Just as welcome" and "Not applicable groups" and then I split the remaining responses into the "More Welcome" and "Less Welcome" groups. I broke down the responses into words, calculated the percentage of responses within each group that contained each word, and then the ratio of the percentages. Using those ratios, we can make this graph of the words most likely to come from each group...

On the y-axis are the words most likely to come from one of the groups, and on the x-axis is the multiplicative odds it came from that group. If the bar for a word points to the right, you can look up how many times as likely it was to come from the "More Welcome" group on the x-axis. If the bar for a word points to the left you can look up how many times as likely it was to come from the "Less Welcome" group on the x-axis.

To very, very, (very!) roughly summarize, people in the "More Welcome" group appear more likely to be talking about...

Features (emails, newsletter, "features", badges, surveys, interface, notifications, newsletters, navigate, podcast)
Improvements in either themselves or the site (improved, improvements, compare, skills, confident, learned, learnt, overcome)
Welcomingness (nicer, kindness, collaborate, encourage)
Other? (I don't know what these might mean) (remind, path, depends, relate, visited, projects, signed)

...while people in the "Less Welcome" group appear more likely to be talking about:

The company ("se", "inc", "corporate", "business", "employees", "management", "decisions", "fire" "firings" and former employees whom I cannot and will not talk about)
Monica (whom I cannot and will not talk about) and associated fallout ("Cellio", "Monica", "moderator", "resignations", "left")
Other unhappiness? (My intent is not to diminish anyone's feelings) ("politics", "drama", "alienated", "worse")

While there may be some overlap in some of these topics, I don't think it is particularly clear-cut that the feelings of welcomeness among the "More Welcome" group came "at the cost" of the feelings of unwelcomeness among the "Less Welcome" group. In this (admittedly brief) analysis, the topics each group was willing to talk about appear different.

Welcoming users has been a big deal for two(?) years at this point. What is the goal Stack Overflow was trying to reach with "more welcome"? Is 15% of users within that goal? Is 70% unswayed by the efforts within that goal? Is 10% less welcome within the goal?

We certainly asked the question on the survey (as we did last year, where we measured ~similar numbers despite the fact that most of the things people are talking about on the "Less Welcome" side this year hadn't happened yet), but I can't say that there's a particular percentage or outcome we were looking to get. Of course we'd like more people to feel more welcomed on the site, but I don't think "X% and we're done" is a thought in anyone's mind. This question was also a relative one ("compared to last year"). I'd have to dig in, but there may be a bunch of people in the "just as welcome as last year" group who feel plenty welcomed and it hasn't changed (and similarly in the opposite direction). Finally, if we were planning to plan based on the results of the yearly Dev Survey I think that'd be pretty poor planning to plan. Measuring/learning/changing course only once a year wouldn't be particularly responsive. We ask questions about welcomingness on the Site Satisfaction survey as well (which runs each month).

Emails make people feel welcome?

Cody Gray asks in the comments...

The number-one word used by folks who feel more welcomed was emails? If that's implying that somehow people feel that emails are welcoming, well let's just say that comes as a major shock to me. I find emails to be one of the most annoying things that a site can do to me. I certainly do not see why someone would find spam to be welcoming. This may, therefore, suggest that the "word clustering" method of interpretation is misleading, if not outright wrong.

It's not the number-one word. It's the number one most-more-used word in the "More Welcome" group (among words with 5 or more total usages, excluding stop words). And there's no clustering of anything. These are multiplicative odds ratios, so they're comparisons of how often people said each word in each response group. The emails case is particularly stark because there were zero usages among the "Less Welcome" group (which gets smoothed to 1 usage, so we can calculate a ratio), while about 1% of people in the "More Welcome" group used it. Here's what a random 5 people (first draw, nothing up my sleeve) had to say about "emails" from the "More Welcome" group.

TL;DR: We made some new users happier, some old users angrier, and most users didn't notice a change. — user3657941, Commented May 31, 2020 at 20:00
I don't see the needle moving on the whole Welcoming Wagon initiative, and the blunt reality is that not enough people care about it to make it worth pursuing. Over the last two years, I just haven't seen enough in the way of hard numbers to convince me that this is something that needs energy invested in it at the cost of other areas. Why is this still an emphasis? — Makoto, Commented May 31, 2020 at 20:55
@Makoto: One could argue that the needle NOT moving is already a win. It could well be that without this emphasis on being welcoming things would have degraded otherwise! (Devil's advocate) — Matthieu M., Commented Jun 2, 2020 at 13:03
@MatthieuM.: That's a rather weak argument, IMO. We've had to tolerate the curators/maintainers being labeled something they weren't ever since that initiative kicked off, and I don't count the needle going nowhere as a "win" of any shade. — Makoto, Commented Jun 2, 2020 at 14:59
@Makoto: I think the offenses at curators/maintainers are a separate topic. They would not have been appropriate even if the initiative had moved the needle. I personally think that focusing on nurturing a welcoming and respectful community is important. I do think that without active focus, things tend to degrade -- bullies and trolls thrive in "unregulated" spaces, and continuously find new ways to go skirt the rules. I also think that it's a really hard task, especially because it seems so hard to measure. Whether SE has been helping or hindering... no comment. — Matthieu M., Commented Jun 2, 2020 at 15:35
@MatthieuM. that is the issue. Most would state they want a more "welcoming environment" (I know I've commented on that) but SE's effort to make this environment actually caused the needle to move in the wrong direction with its user-base. The fact that many less people took the survey and the words used by those who found it "less welcoming" points to that type of correlation. Now this would need more data (and reaching out to people who did leave) but there is enough to see that the focus should be elsewhere (though if it really was on welcoming SE would have hired more CMs over firing them) — LinkBerest Left - SO is AI Now, Commented Jun 2, 2020 at 16:37
@LinkBerest: Actually, the fact that many less people took the survey can also be linked to the fact that it was barely advertised on SO itself, unlike prior years, as the team was looking for diversity over quantity -- measuring is hard :x — Matthieu M., Commented Jun 2, 2020 at 17:11
@MatthieuM. ? I got the same email notice & big banner bar I get every year so don't know how "barely advertised" is defined (and on a cynical note, it could just point to not wanting specific people's, meta's, input if it was advertised on SO less than usual - which I would personally believe as this survey always seems more marketing than data driven) — LinkBerest Left - SO is AI Now, Commented Jun 2, 2020 at 17:16
@LinkBerest: In the past years, there were banners on SO about the survey; I don't remember seeing them this year. But don't take it from me, it's mentioned directly in the results: we advertised the survey less on our own channels than in previous years and sought ways to earn responses from those who may not frequent our sites. — Matthieu M., Commented Jun 2, 2020 at 17:53
@MatthieuM. huh, missed that thanks (granted just proves my cynical self was correct) — LinkBerest Left - SO is AI Now, Commented Jun 2, 2020 at 20:12

2020 Developer Survey results feedback

2 Answers 2

Emails make people feel welcome?

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged
discussion
survey
survey-2020
.

Linked

Hot Network Questions

2 Answers 2

Emails make people feel welcome?

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged discussionsurveysurvey-2020.

Linked

Related

Not the answer you're looking for? Browse other questions tagged
discussion
survey
survey-2020
.