In a meeting with some moderators last week, I committed to releasing the data sets from our initial studies around the efficacy and false positive rates of ChatGPT detectors to them. Tuesday afternoon, we did so. This post contains as much information from that discussion as we are able to share in public, and represents the joint efforts of my community leadership team and other members of the community team.
We will open this post with a discussion about the baseline error rates in GPT detectors. Then, we are going to discuss some data coming out of Stack Overflow that can rule out many hypotheses about root causes for contraction.
We have done our best to look through the data and understand what hypotheses could hold water. To this end, we are engineers and community managers, not scientists. We will not claim these are formal studies that produce scientific data, nor will we claim that the data sources are perfect or uncontestable. Rather, that they are operational data sufficient to produce engineering decisions: the same standard that we use as a Community Management team. We’d like to be clear: nothing in this post is intended to cast aspersions on the moderator team of any site. We respect the work that they are doing to address this problem, as challenging as it may be.
This post will be a lot to take in. And even then, it’s a fraction of the work put into analyzing this question across multiple teams in the company. However, it’s a piece of the work we believe may help clarify some facts that have heretofore been obfuscated.
The tl;dr: Summarizing the results
- The actual rate at which GPT posts are made on Stack Exchange has fallen continuously since its release, and is now very small.
- The rate of suspension for frequent answerers rose by a factor of 16 from ~0.4% to ~6.6% since the rise of GPT, and has held steady since its release.
- If every GPT poster posted exactly three answers and were suspended within three weeks, this would imply a minimum GPT post rate of 330 answers per week on Stack Overflow; in practice, we would expect a significantly greater quantity. Measurements of GPT occurrences on the platform imply fewer than 100 GPT answers per week, in disagreement with this rate, implying the existence of many false positive detections.
- The suspensions issued appear to have a significant and measurable impact on the demography and volume of answerers on the site, preferentially excluding frequent answerers.
- GPT detectors continue to be ineffective for detecting GPT answers on the platform.
Automated GPT detectors have unusable error rates on the platform
In order for us to consider using detectors of any kind (automated or human) on the platform at these volumes, we’d need to see less than a 1-in-50 false positive rate from them. We’ve selected this rate as a ballpark estimate for acceptability.
At this rate, we would still expect to see around 150 incorrectly-placed suspensions on Stack Overflow in the last six months. This value is still too high for comfort, and ideally, we’d see better rates than this. However, at this level of precision, conversations about how we may put such a system to practice can begin.
HuggingFace’s GPT detector assigns a “threshold” score from 0 to 1 of the post being authored by GPT. Based on a random sample of 500 answers from before GPT’s release, each answer being no less than 400 characters long, the false positive rates follow. At the 0.50 detection threshold, around 1-in-5.5 posts are falsely detected. At the 0.90 detection threshold, around 1-in-13 posts are falsely detected.
Keen readers will note that 1-in-5.5 is a fair bit better than the 1-in-3 false detection rate we originally noticed. This is because we used two detectors during the original survey: ZeroGPT and HuggingFace’s detector. The new value of 18% +/- 3.4% false positives from HuggingFace falls within the 95% CI of the original smaller survey (27% +/- 8.9%).
While it is theoretically possible to achieve better baseline error rates than 1-in-20 by picking higher thresholds, the efficacy of the detector may fall off considerably. A detector that does not produce false positives is no good if it also produces no true positives.
For a 1-in-20 false positive rate, the detector threshold needs to be 0.97. Satisfactory rates (less than 1-in-50 false detections) could not be achieved in this test until the detection threshold was set to 0.9975, but the error due to low sample size at this threshold is considerable. Narrowing this window to a more precise value would require significantly more data points than we collected in this survey. At this point, we can’t endorse usage of this service either as a tool for discriminating AI-generated posts or as a tool for validating suspicions.
We thought it would be helpful to include some more discussion and context. Rather a lot of discussion and context, actually. Over the last few months, folks within the company have been working to answer the question, “What has been taking place in the data coming out of Stack Overflow since GPT’s release?” Considerable changes have taken place in demography: users use Stack Overflow at different rates than they did before; different sorts of users use Stack Overflow; and questions and answers reach different outcomes.
What follows is a single piece of the puzzle. The broader picture informs the company’s operational decision-making processes. This piece has a part to play, but only a part. It is not a full explanation of the question, nor is it a blame-first investigation. Neither can it explain every change in the data on Stack Overflow that we see. However, what it reveals calls into question whether GPT detection is possible or effective on the platform, and by proxy, whether a high false positive rate may be partially responsible for a decline in answerers and answerer retention on the platform.
The volume of users who post 3 or more answers per week has dropped rapidly since GPT’s release
While this value has been dropping slowly over time, it’s been dropping at a well-characterized rate. And, it’s been consistent for many years – since late 2016, with some fluctuation, of course (particularly during the onset of COVID). Nominally we would expect to see this behavior continue; however, after the rise of GPT, the slope inflects and there is no recovery present. In total, the rate at which frequent answerers leave the site quadrupled since GPT’s release.
It is worth noting that, early in the release of GPT, we changed the Stack Overflow rules to require new users to wait 30 minutes between first posts, instead of 3 minutes as was originally set for abuse prevention. If this change were causative, we would expect to see a sudden jump to a new lower level, and a return to the prior well-established rate of decrease. However, we do not see this, a strong indicator of deepening attrition. (We would also expect to see a discontinuity in other metrics not listed – this point is established by a confluence of metrics.)
The number of high-volume answerers has seen a -2.4% average week-over-week decrease since December. In total, there has been a -42% contraction in high-volume answerers since the release of GPT. While users may go to ChatGPT to ask their questions, they are obviously not going there to answer questions. Therefore we can consider that…
The total volume of questions available to frequent answerers continues to rise
The alternative hypothesis for the above chart is that the number of questions available for users to answer has simply fallen, on account of question rates falling. This claim is hard to swallow given current data. What follows is the # of available questions posted per week, divided by the number of users who post 3 or more answers in a week. If this hypothesis were true, then this value should be falling, or at least not rising as quickly, as active users are crunched by a collapsing question rate.
However, even though the question rate is collapsing, there are still plenty of available questions for users to answer. It’s hard to tell exactly what the carrying capacity is for this number, but we certainly know that it is somewhere between 3 and today’s value of ~18. In other words, if users were willing and able to answer more questions, the evidence strongly suggests that they could in fact do so. Of course, on short timescales, factors like % of questions closed and % of questions deleted could cause fluctuations in this value. However, in the long view, the trend is clear and not violated in the post-GPT region.
This leaves one question remaining: Where are they going?
7% of the people who post 3 or more answers in a week are suspended within three weeks
…and this value has held reasonably stationary since these suspensions were enacted.
In the 16 weeks before we enabled moderators to suspend users for GPT users, around 0.4% of users who posted >2 answers per week were suspended within three weeks. After we allowed GPT suspensions on first offense, 6.6% of users who posted >2 answers in a given week were suspended within three weeks, a 16-fold increase.
(Note that suspensions issued for users who posted fewer than three answers are not counted here; nor are suspensions issued three weeks after the user answered several questions.)
In a given week, 32% of the people who post 3 or more answers also did so during one of the last eight weeks. Supposing that these suspensions are distributed across users regardless of GPT usage, we should see (by rough order of magnitude) a ~2.2% decrease in the actual volume of users who post three or more answers in a given week. And indeed, real data (2.4%) are quite close to the theoretical percentage.
Instead suppose that no more than 1-in-50 of the people who were suspended for GPT usage were not actually using GPT. In order for this to be true, a large volume of users would have needed to immediately convert from being regular users to ChatGPT users; and then, a high rate of conversion would have to be sustained over time, long after the release of ChatGPT, in order to sustain present suspension rates.
Regardless of the above, no Community Manager will tell you that removing 7% of the users who try to actively participate in a community per week is remotely tenable for a healthy community. Supposing every suspension is accurate, the magnitude raises serious concerns about long-term sustainability for the site.
It is worth acknowledging that we did give explicit permission to suspend on 1st offense for GPT usage. However, even in the absence of these policies, this value alone rings a deafening number of alarm bells for potential false positive detections and contributor loss alike. If there are false positive detections, even removing users’ content incorrectly could prove harmful.
Users who post 3 or more answers in a given week produce about half the answers
First, a short detour. We are going to focus on users who post 3 or more answers in a given week for much of this post. While it may seem a bit odd to look only at this segment of users (and it is, of course, not the only segment of users we investigated), there is a rationale for doing so.
Users who post answers more than twice a week used to comprise about half the content produced on Stack Overflow. However, since the advent of GPT, the % of content produced by frequent answerers has started to collapse unexpectedly. Given the absence of question scarcity as a factor for answerers (note the above chart), the clear inference is that a large portion of frequent answerers are leaving the site, or the site is suddenly not effective at retaining new frequent answerers.
(It is worth noting that GPT messages and suspensions disproportionately skew towards users who have posted more than two answers in a week, but we can’t discuss this in more detail publicly without revealing the details of how GPT posts are detected on the platform.)
Yet, at the same time, actual GPT posts on the site have fallen continuously since release
What follows is the internal ‘gold standard’ for how we measure GPT posts on the platform. It produces a coarse estimate, and can’t be used to decide whether a given post or person is posting using GPT. However, in aggregate, it can offer us insight into the ‘true’ rate of GPT posts on the platform.
This metric is based around the number of drafts a user has saved before posting their answer. Stack Exchange systems automatically save a draft copy of a user’s post to a cache location several seconds after they stop typing, with no further user input necessary. In principle, if people are copying and pasting answers out of services like GPT, then they won’t save as many drafts as people who write answers within Stack Exchange. In practice, many users save few drafts routinely (for example, because some users copy and paste the answer in from a separate doc, or because they don’t stop writing until they’re ready to post), so it’s the ratio of large draft saves to small draft saves that actually lets us measure volume in practice.
This metric is sensitive to noise, but was validated against other metrics early on at the peak of the GPT answer rate. Additionally, it matches the volume of actual actions taken on the platform early on after the release of GPT.
This allows us to get a good, albeit coarse, understanding of the overall population trends in GPT answers. However, it does not allow us to identify which specific answers were drafted by or with assistance of GPT, because quite a lot of answers are ‘normally’ posted after saving very few drafts.
Between January 2023 and March 2023, it appeared that we were going to hit a floor of around 700 GPT-generated posts per week and stay there, which would have been quite a bad outcome for the site’s general health. However, as time progressed, it became clear that the actual volume of GPT answers was falling precipitously - even as a percentage of total answers (which, yes, is also falling comparably).
Some folks have asked us why this metric is capable of reporting negative numbers. The condensed answer is that the metric has noise. If the true value is zero, sometimes it will report a value higher than zero, and sometimes a value lower than zero. Since we know how much noise the metric has, we know what the largest value for GPT-suspect posts should be.
At the initial holding point, GPT answers reflected around 2.5% of the answers posted on Stack Overflow. Rates hovered around this level for several weeks, but such rates were not sustainable and the percentage of posts authored by GPT began to fall. The following chart shows the expected % of answers posted in a given week that are GPT-suspect.
These days, however, it’s clear that the rate of GPT answers on Stack Overflow is extremely small. In fact, it is difficult to estimate the true number of GPT answers posted on Stack Overflow for this reason. Based on the data, we would hazard a guess that Stack Overflow currently sees 10-15 GPT answers in the typical day, or 70-100 answers per week. There is room for error due to the inherent uncertainty in the measurement method, but not room for magnitudes of error. We can therefore say that the rate of GPT posts is far less than it was two months ago, and then it is less than it was two months before that.
So, under what conditions could it be the case that roughly 7% of frequent answerers on the site are still posting via ChatGPT? If this were the case, the site should be seeing at least 330 GPT answers per week, but the rate estimate is not close. This also assumes every user who posts GPT answers are caught, and that GPT answerers post no more or less than three answers in a given week. In practice, the site should be seeing significantly more than 330 GPT answers per week to support this suspension rate.
It could be possible, either due to severe measurement error or due to an unexpected change in user behavior that obfuscates GPT usage using this method. But the evidence for this viewpoint does not appear strong.
Many of GPT appeals sent to the Stack Exchange support inbox could not be reasonably substantiated
While we could not recover all of the GPT suspension appeals sent to the Stack Exchange inbox, we could characterize some of them.
As a platform, we have an obligation to ensure that moderation actions taken on the Stack Exchange network are accurate and can be verified upon review if we need to do so. We need to be able to see, understand, and assess whether the actions taken were correct. It therefore needs to be said that we are very rarely, if ever, in the position where we cannot do so. In all other areas for which we receive suspension appeals, moderator actions are easily verified and double-checked the overwhelming majority of the time. It is rare and notable if we are ever in the position of overturning a moderator’s decision due to insufficient or contradictory evidence.
So, when we say that many of the GPT appeals we receive could not be substantiated on review, please keep in mind that our baseline value for this is zero, and it’s been that way for years. It is exceptionally strange for us to look at a moderator’s action and find ourselves unable to verify it – yet this is the situation we are frequently in with respect to GPT.
It is worth noting that we don’t believe this discrepancy is due to moderator misconduct or malfeasance. Our goal here is not to accuse moderators of wrongdoing or poor judgment. We respect the fact that they were, and are, working under difficult circumstances to achieve a goal we appreciate. Rather, it is important to remember that the company has a strong need to ensure moderator actions are verifiable and justifiable. And, on this point, we need to seriously consider whether these processes, in whatever form they are taking, do what they should.
True false positive rate of moderator actions
We want to clear up a particularly important point. We are, as of right now, operating under the evidence-backed assumption that humans cannot identify the difference between GPT and non-GPT posts with sufficient operational accuracy to apply these methods to the platform as a whole.
Under this assumption, it is impossible for us to generate a list of cases where we know moderators have made a mistake. If we were to do so, it would imply that we have a method by which we can know that the incorrect action was taken. If we could do this, we would share our methods in a heartbeat in the form of guidance to the moderator teams across the network, and then we’d carry on with things as normal. Instead, the most we can do is state that we just can’t tell. We lack the tools to verify wrongdoing on the part of a user who has been removed, messaged, or had their content deleted, and this is a serious problem.
This is a critical opportunity for us to inspect the processes that are contributing to the outlook for the site, and contextualize them in the overall state and progression of the network.
Stringing it all together
Look, the truth is, there are going to be many hypotheses about what could be taking place in the data on Stack Overflow. Heck, we’ve had several large analytics teams pore over the SO and SE data for any sorts of anomalies or possibilities when it comes to participation contraction. While it is absolutely true that Stack Overflow is losing users faster than it is gaining users, none of the hypotheses generated by the company can explain away the relationship between % of frequent answerer suspensions and the decrease in frequent answerers, in the context of falling actual GPT post rates.
In the Community Management industry, it is a well-known fact that removing a person from a community, even for a short time, has an outsize impact on the contributor community. The scalar factor here varies considerably from community to community, but it must be taken into account here as well. Deleting their contributions for reasons they feel unjust, or commenting on behavior that is not present, appears to have a similar effect to being suspended, and differences are often small when it comes to long-term user outcomes (such as the user leaving, and/or potentially other people leaving with them).
Is it still possible that the proportion of false positives is small? Maybe so – it can’t be completely eliminated at this time. Direct causative data are not possible to obtain for this problem. But for that to be true, it would require some very strange user behavior en masse around answering, by users who were otherwise answering questions normally. These are behaviors we do not have an organic explanation for after months of exploration, even under the scenario where question demand contracts at a rapid pace. Again, it’s possible, but the evidence is not favorable. We are forced to look at all possible exogenous causes of user attrition - including whether logic internal to the network needs to be altered (and we have made changes here, as well).
Finally, it needs to be said that the analysis presented here is far from the complete analysis we have conducted internally. These are pieces of a larger puzzle, selected because they best express our meaning (and contain no proprietary data). Seeking a set of root causes for the contraction in the network’s community size, and an explanation for how it affects different sites/community segments around the network, has been the object of study for dozens of people, and for many months now.
We hope by now the ultimate point here is clear: Suppose we are right in this assessment and GPT removal actions are not reasonably defensible. How long can we afford to wait? To what extent can we continue to risk the network’s short-term integrity against the long-term risks of the current environment? Any good community management policy must balance the risk of action against the risk of inaction in any given situation, and the evidence does not presently favor inaction.
What we know, right now, is that the current situation is untenable. We have real, justified concerns for the survival of the network. We’re not saying this to invoke despair, or to imply that these decisions are overly rushed. There are silver linings here, and there is significant potential for future growth and a more stable long-term path. The onus is on us to find a way forward that protects the communities and network as a whole.
If there is no future for a network where mods can’t assess posts on the basis of GPT authorship (where we would be after this policy), and if there is no future for a network where mods can assess posts on the basis of GPT authorship (where we are today), then there is no future for the network at all. Yet somehow, our adjacent communities are making this work. In this deadlock, something has to give. At the end of the day, it all boils down to this: We have to walk the middle path.
