Original Data Analysis
- In April 2019 there was an investigation performed that looked at the number of users who regularly post and vote on Meta Stack Overflow, and how answering on Stack Overflow predicted that participation.
- The results were that posting and voting on Meta is done by 200-300 users (on average), and that answering on Stack Overflow had a weak positive correlation with participating on Meta.
- The results also showed that answering on Stack Overflow had a weak positive correlation with participating on Meta: the more answers a user posts on SO, the more likely they are to be active on Meta. However, even at extremes of answering behavior like 10 answers per month, a minority (~10%) of power Stack Overflow answerers vote or post on Meta.
This is the source of the oft-quoted stat that Meta has 0.015% of Stack Overflow's active users. And from this perspective, the statistic is true. However, in retrospect, this does not present the full picture (as we see below). This data painted a picture that contributed to some of the well-intentioned internal thinking related to Meta that continued through the end of 2019.
And please do not try to figure out who is to blame for what here (things are always clearer in hindsight). Everyone who worked on this internally (and there were many people) was coming from a place of trying to make the best product and strategy decisions to benefit the largest number of users to the greatest degree possible given the limited resources of money, developers and other product team members, and time.
Asking a Different Question
Fast-forward to mid-January (nothing much interesting happened in the interim, right?). I asked a question on our internal Teams instance summarized as follows:
- I first queried some internal data sources to look at the numbers of logged-in users who were active (viewed a page other than the home page) on MSO or MSE at least once per month in 2019, and the numbers of users who had engaged (post, comment, edit, vote) on MSO or MSE at least once per month during that time. The numbers looked like this:
- I observed that there were on average 94K active users and ~6500 engaged users per month on MSO or MSE (and this doesn't include anonymous users).
- The engagement percent averaged around 7%, which is significantly lower than most other sites, but one can assume that many people go to MSO/MSE purely to lurk and read, and the lack of engagement there should not be construed as lack of being affected by what is going on there.
- I theorized that, though the relative number of SO users who visit MSE/MSO may be relatively minuscule, the group of users who is active (just visiting) on MSE/MSO is extremely active in the areas of site moderation.
- If true, this would mean that MSE/MSO have an incredibly high degree of influence among the users who contribute to the core areas of site upkeep of Stack Overflow.
- I submitted a request to the data team to ask them to look into the question of: What percentage of content moderation/curation activities on Stack Overflow are performed by accounts that are active on MSO/MSE, specifically looking at flags submitted, review tasks performed and post edits made, and checking these numbers for users who had visited MSO/MSE in the last 30 or 60 days.
Results
Our awesome data team (in this case the data work and analysis was run, checked, and double-checked by Kevin Montrose and Jason Punyon) crunched the numbers, and got back with the following:
TL;DR - a high percentage (50+%) of curation/moderation actions on Stack Overflow come from users who at least occasionally visit MSO or MSE. This is true for the entire range of time we looked at. These graphs give a sense at a glance:
To get more into it:
- 86% / 91% of flags created on SO are created by users who had visited MSO/MSE in the past 30/60 days
- 75% / 81% of review tasks on SO are processed by users who had visited MSO/MSE in the past 30/60 days
- 61% / 66% of post edits on SO are saved by users who had visited MSO/MSE in the past 30/60 days
- 44% / 50% of own-content edits and 84% / 89% of others'-content edits were made by users who had visited MSO/MSE in the past 30/60 days
- While it is still accurate to say that the conversations on Meta are dominated by a relatively small number of users (200-300), the readership of those conversations includes users who perform the majority of the curation/moderation actions on Stack Overflow.
The immediate repercussion of this is that our estimation of the "reach" of Meta (based on the earlier data analysis and a different set of assumptions) had been incorrectly low, as we assumed that viewership correlated with answering activity - which does not appear to be the case.
We also found that there had not been any sign of a change to the amount of moderation activities that were performed by this group on Stack Overflow since our series of crises with the Community starting in September 2019.
Subsequently, the same data analysis was also applied to look at the content moderation activities on SO by users who had been engaged on MSO/MSE at least once per month (the original question looked only at being active - visiting at least once per month):
The TL;DR on this was that a high percentage (30+%) of curation/moderation actions on Stack Overflow come from users who at least occasionally act on MSO or MSE. This is ~50% of the actions performed by users who read MSO or MSE, despite this being a smaller group. This is higher than would seem to be implied by the earlier research.
- 59% / 64% of flags created on SO are created by users who had engaged on MSO/MSE in the past 30/60 days
- 35% / 39% of review tasks on SO are processed by users who had engaged on MSO/MSE in the past 30/60 days
- 28% / 32% of post edits on SO are saved by users who had engaged on MSO/MSE in the past 30/60 days
- 12% / 14% of own-content edits and 50% / 57% of others'-content edits were made by users who had engaged on MSO/MSE in the past 30/60 days
An interpretation of all of this is that MSE and MSO allow us to:
- Reach the people performing a majority of our curation and moderation tasks, and those posting a sizable percentage (if not a majority) of our answers.
- Get feedback from the people mostly doing curation and moderation tasks on SO, but a sizable number (10-20%) of answers are also being provided by these people.
Which brings us full circle to the quote cited in the question above:
Looking at data about Meta that’s eluded us until recently (we promise we’re not ever going to call it metadata), it’s clear that our focus there needs a lot of work and better resourcing.
Many people at Stack are working on these issues. Hopefully this data and history will shed some light on the questions and data that is helping to guide us in this endeavor.
