Since this has been posted, we've done a few things:
- We now cast four flags on posts that pass an accuracy threshold (current 99.9% historical confidence)
- SmokeDetector now casts the first flag on all autoflagged posts. That should help address concerns about transparency in the flag queue & on post flag timelines.
There's a userscript exposing auto flagging activity inline on the post. It's pretty:
We're trying to get an RSS feed of posts deleted with autoflags for chat consumption, but chat RSS is fraught with peril.
TL;DR: Charcoal is the organisation behind SmokeDetector. Since January 2017, we've been casting up to 3 flags automatically on posts that our systems are confident are spam. We'd like to increase that to 5 automatic flags on posts we're even more certain of to reduce the time spam spends alive on the network.
Who are you?
Charcoal is a user-run organisation that is primarily responsible for the spam-detecting bot, SmokeDetector. Over the past four years, with the aid of SmokeDetector, we've looked for spam on the Stack Exchange network to flag and destroy manually. In January 2017, with the blessing of Stack Exchange, we started running an "autoflagging" project, wherein our systems automatically cast up to three flags on a post if they're confident that it's spam. If you missed that happening entirely, we wrote a meta post on Meta Stack Exchange - or there's a slightly more concise explanation on our website.
How's that been going for you?
Good. We currently have 215 users who have opted into the autoflagging system (you can sign up too, if you're interested). We've flagged around 30 000 (29 592) posts, of which the vast majority (29 526) were confirmed spam - that's 99.7% accurate.
What are you proposing?
We'd like to expand our autoflagging system. At present we cast up to 3 automated flags on posts we're confident are spam; we'd like to increase that number to 5 flags on a subset of those autoflagged posts where we're even more certain they're spam.
Why?
Just so we're up-front about this: this is an experiment. Ultimately, we're trying to do these things:
- Reduce the time that spam spends on the sites before being deleted;
- Lower the number of humans who involuntarily have to see or interact with spam.
Increasing the number of flags we cast automatically on spam should accomplish both of these things:
- Automatic flags are near-instant; manual flags take multiple minutes to be cast - that means that increasing the ratio of automatic to manual flags results in a shorter time before 6 flags accumulate and the spam is deleted.
- Automatic flags are not cast by a human. Fewer humans, therefore, are forced to see/interact with the spam.
The data we have backs this up. In terms of time to deletion, we saw a significant drop in the time it took to delete spam when we started our autoflagging project. Take a look at this graph from the meta post on the subject for an excellent visual representation of that. Before we started autoflagging, spam spent an average of 56 hours per day alive across the network; with autoflagging in place, the average is much less, at around 7 hours per day.
What would this change mean for sites?
If this change goes ahead, these things are likely to happen:
- It will only take 1 or 2 manual flags from users to spam-nuke an autoflagged post, instead of the current 3. Posts that are not autoflagged will, of course, still require 6 flags to nuke.
- There may be an increase in posts spam-nuked entirely by Charcoal members, who may or may not be active on the site.
- You will see a reduction in the time spam spends on the site before being deleted.
- Fewer humans will have to involuntarily see each spam post.
The last two of those are indisputably good things. The first two, however, are more controversial, and are the reason we want to have a discussion here on meta before we make this happen. What follows are the major concerns we've seen, and what we can do about them or why we don't think they're an issue - we'd like to hear your thoughts.
The major thing we're looking for out of this is a reduction in time to deletion. The following graph shows how long spam currently spends alive on the top few sites; we're hoping to see a moderate reduction in the average times, and a significant reduction in the top outliers.
The following graph is from an experiment we've been running over the past week, casting between 1 and 5 flags randomly on each post matching the settings we're considering.
In raw numbers, that's this:
PostCount FlagCount ATTD StdDev CommonMax
173 1 191.474 243.63 678.73
166 2 98.7831 127.88 354.55
167 3 69.2814 156.62 382.51
194 4 33.3196 61.46 156.23
177 5 13.5254 12.43 38.39
PostCount is the sample size; FlagCount the number of flags cast on each post in the sample; ATTD the average time to deletion, and CommonMax is the maximum of a 95% confidence interval. The major takeaway from these stats is that we're likely to see a ~5x drop in the average time to deletion, and a ~10x drop in the outliers.
Accuracy & false positives
Spam flags are a powerful feature that need some care in applying correctly. This is a concern that came up when we originally built the autoflagging system, so we already have safeguards built in.
- We only flag a post if we're more than 99.75% sure it's spam. (Technically, the precise certainty varies by conditions set by the users whose accounts we use, but it's always above 99.75% - more detail on that on our website).
- If the system breaks down or bugs out and starts flagging things it shouldn't, all Charcoal members and all network moderators have access to a command that immediately halts all flagging activity and requires intervention from a system administrator to re-enable. Outside of testing, that kill-switch has never had to be used.
- We never unilaterally nuke a post. There are currently 3 manual flags required in addition to the automatic flags to nuke a post; this increase proposal still retains at least one manual flag.
We also make sure that everything has human oversight at all times. While only 3 humans currently have to manually flag the post, there are always more users than that reviewing the system's decisions and classifications; if a post is flagged that shouldn't have been, we are alerted and can alert the relevant moderators to resolve the issue. Again, this is very rare: over the past year, we've flagged 66 posts that shouldn't have been, compared to 29 592 spam posts (that's 99.7% accurate overall). We allow users to set their own flagging conditions, provided they don't go below our baseline 99.75% certainty. We recommend, however, a higher value that has a certainty of 100.00% - those who set their conditions below that are likely to see more false positives flagged using their account.
This proposal decreases the required manual involvement to nuke a post; to compensate for that lower human-involvement barrier, we will correspondingly increase the required accuracy before casting the extra automatic flags. For example, we currently require 99.75% accuracy before casting autoflags; we could require 99.9% accuracy for 4 autoflags, and 99.99% accuracy for 5 autoflags. (For reference, humans are accurate 95.4% of the time, or 87.3% on Stack Overflow - those are stats that jmac (a former Community Manager) looked up for us last year when we started autoflagging).
In the rare event of a legitimate post getting autoflagged, we also have systems in place to ensure it isn't accidentally deleted and forgotten about. Multiple people review each post we catch, whether it's autoflagged or not, and classify it as spam or not; if an autoflagged post is classified as not-spam, the system posts an alert to chat to let us know. That lets us ping the necessary people to retract their flags, and keep an eye on the post to make sure it doesn't get deleted.
To make it starkly clear how accurate this could be, here's a visualisation:
That's a chronological representation (left-right, top-bottom) of every post that would have been flagged under the settings we're considering for 5 flags, and whether they were spam (green squares) or legitimate (red squares).
Community agency & involvement
As I said earlier, this proposal reduces the required manual involvement to nuke a post. Since Charcoal members also cast manual flags on top of the automatic flags cast by the system, that's also likely to increase the number of posts that are nuked entirely by Charcoal members, without involvement from users who are active on this site. Some posts already have 6 flags cast on them by Charcoal (including autoflagging and manual flags), but the proportion of posts that applies to is likely to increase.
We don't think this is an issue in terms of subject matter expertise: the spam we see on the Stack Exchange network is broadly the same wherever you go - you don't need any subject matter expertise or activity on a particular site to be able to tell what's spam and what's not. We do, however, recognise that it's possible that a site's community may want to handle its own spam; if that's the case, we're happy to turn the autoflagging system off on that site or to retain it at its current levels. Just post something on your site's meta and link us to it.
What now?
We want to increase the number of automated flags from 3 to 5 to reduce the time spam spends alive on the network. We'd like to hear your thoughts. We appreciate that quite a lot of the stuff we do at Charcoal is fairly invisible to the sites, so we want to be as open as possible. If you'd like data or specific reports, let us know and we'll try to add them in - we already have a lot of reporting around autoflagging, so it may already exist. If there's other stuff we can do to explain or to help you make an informed decision about whether you want this, drop an answer or a comment on this post. Charcoal members will be hanging around this post to respond to your concerns, or you can also visit us in chat.
