43
$\begingroup$

According to Xander's answer to a previous question on AI-generated content, posting chatGPT-generated answers without disclosing they are generated by chatGPT is a violation of openAI's terms of service.

User therexists, (Edit: account has been deleted) a new user, posted one question on 2023-04-15 at 01:25:32Z. Shortly after that they posted 8 answers, all of which seem to be generated by chatGPT:

  1. At 02:01:29Z
  2. At 02:12:44Z
  3. At 02:17:58Z
  4. At 02:34:12Z
  5. At 02:43:00Z
  6. At 02:48:12Z
  7. At 03:20:09Z
  8. At 03:23:34Z (just deleted as I was typing this).

Should we simply raise individual flags with "In need of moderator attention"? Although they've been identified as chatGPT in comments (mostly thanks to the diligence of JonathanZ), they both seem like rather haphazard responses for this type of systematic misuse. In any case, it would seem that the user merits direct moderator attention. While still a new user, this is a rather strong abuse and in need of corrective action, IMHO.

While we may not (yet?) need a formal ban, I think we are very close or already at the point that we need an official policy of deletion of chatGPT generated answers; and of closing questions of the form "I asked chatGPT for an answer, here is what it told me, could you guys help me understand what it is saying?"

$\endgroup$
33
  • 4
    $\begingroup$ Here's one more deleted answer from that user for your list: math.stackexchange.com/a/4679392 . I was actually surprised that this one was downvoted and deleted, since it looks mostly right to me, up to swapping the variables $x,y$ for $u,v$ in a couple places. $\endgroup$
    – Viktor Vaughn
    Commented Apr 16, 2023 at 3:39
  • 7
    $\begingroup$ @ArturoMagidin Thank you for formally asking this. There were actually 10 of those GPT answers posted by the same user, the ones missing from your list are this and this. I have raised a mod flag on one which scored a perfect 100% AI/GPT generated on ChatGPT detector, which flag is still pending. $\endgroup$
    – dxiv
    Commented Apr 16, 2023 at 6:03
  • 4
    $\begingroup$ @dxiv Thank you for completing the list. The user's account seems to have been deleted. $\endgroup$
    – Arturo Magidin
    Commented Apr 16, 2023 at 6:13
  • 4
    $\begingroup$ How is it possible to identify an AI response? Is there any kind of indication, beyond the fact that it is almost always a wrong answer. $\endgroup$
    – al-Hwarizmi
    Commented Apr 16, 2023 at 10:10
  • 40
    $\begingroup$ I think we need a formal ban. $\endgroup$
    – Alexander Gruber Mod
    Commented Apr 16, 2023 at 12:44
  • 12
    $\begingroup$ A more chilling thought: Some autonomous AI joined math.se, posted a few answers, then closed its account. We have a CAPTCHA system to stop posting bots, but can it catch such a thing? $\endgroup$
    – GEdgar
    Commented Apr 16, 2023 at 12:57
  • 31
    $\begingroup$ @dxiv Beware that ZeroGPT detector yields many false posiitves, e.g. it claims 57, 100, 49, 38, 92, 62,100, 90, 40, 73 percent AI/GPT for said user's answers, but much higher for some of Arturo's recent answers, e.g. it claims these are AI/GPT generated: 93% & 87% & 87% & 84% & 83%. As such we should be extremely careful with such tools to avoid wrongly accusing users. $\endgroup$
    – Bill Dubuque
    Commented Apr 16, 2023 at 20:50
  • 24
    $\begingroup$ Now that's insulting.... $\endgroup$
    – Arturo Magidin
    Commented Apr 16, 2023 at 20:51
  • 4
    $\begingroup$ @Arturo I suspect we'd see similar false positives for most prolific contributors since it was likely trained on SE. Believe it or not, iirc some folks have already begun filing defamation lawsuits against AI bots. $\endgroup$
    – Bill Dubuque
    Commented Apr 16, 2023 at 20:58
  • 8
    $\begingroup$ @BillDubuque "Beware that ZeroGPT detector yields many false positives" $\;-\;$ Absolutely correct, and this should be stressed more in any discussion on possible policies. In the case of the flag I raised, though, I am still confident it was all GPT, but that wasn't based on the ZeroGPT score alone. I read that user's other answers, and the prose I flagged sounded completely out of place for someone who had basic mistakes and obvious blind spots in other posts minutes apart. $\endgroup$
    – dxiv
    Commented Apr 16, 2023 at 21:16
  • 9
    $\begingroup$ @dxiv I confess that I am greatly concerned that these answer analyses are wrongly influenced by unreliable tools like ZeroGPT, and this may lead to false accusations - possibly alienating some users. I think we should judge answers as we have always done - completely ignoring these unreliable AI-verification tools. $\endgroup$
    – Bill Dubuque
    Commented Apr 16, 2023 at 21:31
  • 4
    $\begingroup$ Can we require longer time intervals between posts, or fewer posts per day? meta.stackoverflow.com/a/421846 $\endgroup$
    – mr_e_man
    Commented Apr 17, 2023 at 2:02
  • 8
    $\begingroup$ This recent development on Meta Stack Exchange is frightening. In a nutshell, it plans to bring in "generative AI" to assist users in answering questions, and then claims that humans will develop a relationship with this AI that will allow both to flourish. That's not quite right, because if a human needs an answer without putting effort into their question, CGPT gives it to them : sometimes incorrectly at that. I know that "good faith" is a cornerstone of SE, but with overwhelming evidence that AI won't be used properly, why do this? $\endgroup$
    – Sarvesh Ravichandran Iyer
    Commented Apr 18, 2023 at 10:38
  • 4
    $\begingroup$ "I think we should judge answers as we have always done". I disagree, and I think the point of this post is to disagree with this. Because of both the newly-introduced difficulties handling ChapGPT that dxiv mentions in their comment, and the cost to us that I describe in my answer, I think falling back on existing tools is inadequate. Now, there are a range of changes we could make, and "Run all answers through a detector, and suspend any accounts that score higher than 90%" would be an insane panicked over reaction that nobody wants or advocates. .... $\endgroup$
    – JonathanZ
    Commented Apr 18, 2023 at 15:27
  • 5
    $\begingroup$ But making people aware of such posts, stating a policy of how users can handle them (flagging type, appropriate standards for noting in a comment, making answerers aware that ChatGPT-generated answers are discouraged), and giving information on how to deal with such answers, for those who want to do so, these are all sane and useful new steps that we could take. $\endgroup$
    – JonathanZ
    Commented Apr 18, 2023 at 15:32

3 Answers 3

Reset to default
48
$\begingroup$

It seems like the current approach is a combination of "posting them violates OpenAI's terms of service" and "handle them like other low quality answers". I think we need a new policy, because the current policies ignore the fact that

Handling ChatGPT answers wastes community members' energy, time, and attention. And these are the scarcest resources we have.

In my experience, ChatGPT answers look very plausible, and initially look like correct math. (Heck, plausible and correct text is the corpus it is trained on.) You have to spend an inordinate amount of energy teasing out what they say, and "debugging" them. That's why I bothered to track down all of that user's answers: It bugged me that I'd wasted my time on one of them, and didn't want other members to have to do the same. (Note: I didn't just auto-comment on them. Other users were able to verify that some of them were ChatGPT, and some I read with a bit of attention and they seemed to fit the bill.)

So I'm not sure what the new policy should be, but lots of moderation that happens here (by both mods and non-mods) assume that it's okay that things eventually get taken care of. We've got two examples now of high speed ChatGPT spammers, and I'd love it if we got new tools/policies that will help us deal with them quickly, and prevent wasting members' time.

$\endgroup$
10
  • 5
    $\begingroup$ One of my concerns with simply raising "Needs moderator attention" flags is precisely eating up the moderator's time and attention. It is also my experience those types of flags seem to (understandably) take the longest to be acted on. Just waiting for downvotes and delete votes takes a while. While a formal policy might not deal with them any faster, I think it would eat up less time, much like the policy on closing questions without context does. $\endgroup$
    – Arturo Magidin
    Commented Apr 16, 2023 at 20:16
  • $\begingroup$ @ArturoMagidin - Do you have any ideas for what such a policy would look like? Me, I feel like we can do some stuff from the bottom up, w/o an SE ruling (where "we" = "the people who participate on meta). For example, maybe "submit suspicious answers to a specified detector, and add a comment noting the returned value if it's above X%". This is something anyone could do, and it' has the key advantage of giving others a heads-up. Of course, such a proposal should be posted as an answer, and people should discuss the +s and -s. $\endgroup$
    – JonathanZ
    Commented Apr 16, 2023 at 20:54
  • 8
    $\begingroup$ Be careful depending on AI/GPT-verification tools like ZeroGPT mentioned above since they yield many false positives. $\endgroup$
    – Bill Dubuque
    Commented Apr 16, 2023 at 21:04
  • $\begingroup$ If we don't go the route of a formal ban, then certainly a policy of "downvote/close/delete" for those posts that seem highly likely to be either chatGPT (or the like) generated, or those that are explicit in sourcing chatGPT/Bard/etc. For the latter it's easy to identify. The former may be more borderline, but the precise contours of how we determine it does not have to precede the policy. There is no uniform agreement on what or how much "context" is needed, but I think that policy has been generally successful in its goal. $\endgroup$
    – Arturo Magidin
    Commented Apr 16, 2023 at 21:29
  • 2
    $\begingroup$ @BillDubuque - Thanks for pointing that out. I'm wondering what kind of guidelines we can give for labeling an answer as GPT-generated, even if only informally? I was hesitant to even add those comments, until I had poked around, and seen others raising the same question. Luckily, they all had one thing in common - they were all posted by the same user, way too quickly, across a very wide variety of subjects. .... $\endgroup$
    – JonathanZ
    Commented Apr 16, 2023 at 21:33
  • 1
    $\begingroup$ I don't want the label "ChatGPT generated" to start getting thrown around too much, especially as I'd guess that it'd be disproportionately applied to people with weak English skills or just bad math understanding. $\endgroup$
    – JonathanZ
    Commented Apr 16, 2023 at 21:35
  • $\begingroup$ @Jonathan I think your prior 2 comments are better placed in the comments on the the question - where I raised this matter explicitly. $\endgroup$
    – Bill Dubuque
    Commented Apr 16, 2023 at 21:53
  • 5
    $\begingroup$ In my short experience with the paid version Monica, the chatGPT answer almost always follows a format of an SAT essay - 4 or 5 paragraphs the first being introduction/rephrasing of question and then body paragraphs followed by a "summary" paragraph. Coupled with the fact that chatGPT asserted 2 was a composite number in one of queries and the answers from the banned user wrote mostly incorrect or irrelevant stuff, I think this one was the right verdict. $\endgroup$
    – dezdichado
    Commented Apr 25, 2023 at 5:24
  • $\begingroup$ ChatGPT answers may be better than human for low-quality general questions. $\endgroup$
    – kludg
    Commented Apr 26, 2023 at 7:11
  • 7
    $\begingroup$ We can safely delete both the low quality question and the ChatGPT answer from math.se. $\endgroup$
    – JonathanZ
    Commented Apr 26, 2023 at 14:58
7
$\begingroup$

Below is the little publicized What is the network policy regarding AI Generated content? posted on the main SE meta the other day, followed by ensuing Discussion: Network policy regarding AI Generated content on SE meta, and Discussion: New AI-generated content policy on SO meta (both ongoing, and not copied here). Posting it for reference, my thoughts on this have not changed since the past comments here and elsewhere.

Q.   Earlier this week, Stack Exchange released guidance to moderators on how to moderate AI Generated content. What does this guidance include?

A.   We recently performed a set of analyses on the current approach to AI-generated content moderation. The conclusions of these analyses strongly indicate to us that AI-generated content is not being properly identified across the network, and that the potential for false-positives is very high. Through no fault of moderators' own, we also suspect that there have been biases for or against residents of specific countries as a potential result of the heuristics being applied to these posts. Finally, internal evidence strongly suggests that the overapplication of suspensions for AI-generated content may be turning away a large number of legitimate contributors to the site.

In order to help mitigate the issue, we've asked moderators to apply a very strict standard of evidence to determining whether a post is AI-authored when deciding to suspend a user. This standard of evidence excludes the use of moderators' best guesses based on users' writing styles and behavioral indicators, because we could not validate that these indicators are actually successfully identifying AI-generated posts when they are written. This standard would exclude most suspensions issued to date.

We've also identified that current GPT detectors have an unacceptably high false positive rate for content on our network and should not be regarded as reliable indicators of GPT authorship. While these aren't the sole tools that moderators rely upon to identify AI-generated content, some of the heuristics used have been developed with their assistance.

We've reminded moderators that suspensions (and typically mod messages as well) are for real, verifiable malfeasance only, and should not be enacted on the basis of hunches, guesses, intuition, or unverified heuristics. Therefore, we are not confident that either GPT detectors or best-guess heuristics can be used to definitively identify suspicious content for the purposes of suspension.

As always, moderators who identify that a user has a problematic pattern of low-quality posts should continue to act on such users as they otherwise would. Indicators moderators currently use to determine that a post was authored with the help of AI can in some cases form a reliable set of indicators that the content quality may be poor, and moderators should feel free to review posts as such. If someone is repeatedly contributing low-quality content, we already have policies in place to help handle it, including a suspension reason that can, in those cases, be used.

$\endgroup$
2
  • 2
    $\begingroup$ I have no idea what kind of posts moderators (either on math.se or other *.se sites) have been treating as GPT-generated, but I'm stumped by the "there have been biases for or against residents of specific countries as a potential result of the heuristics being applied". They've gotta be referring to language quality, right? I can't think of what other country information is available. And ChatGPT produces pretty darn good English, at least at the level of grammar. I'm left very confused about this. $\endgroup$
    – JonathanZ
    Commented Jun 3, 2023 at 5:31
  • 1
    $\begingroup$ @JonathanZsupportsMonicaC It is quite confusing, indeed. Or, as another user put it in a comment on the main SE meta: "Painting this all with "in the name of inclusivity" just drives it further home how cynical this all is. Not sure it's worth arguing about the face value of the "arguments" made by the company (for want of a better word)". $\endgroup$
    – dxiv
    Commented Jun 4, 2023 at 0:47
-13
$\begingroup$

I think a good temporary policy would be a requirement to disclose the use of generative AI (like ChatGPT) to assist with (or fully write) answers and questions.

This not only makes them easier to moderate, but also will allow us to collect useful data to track and quantify the problem.

Note that this assumes good faith. However, I don't think this is a problem because:

  1. I think people are honestly viewing this as a tool, not dishonest plagiarism. For example, I often use ChatGPT or similar and share the results with others (though not specifically on stack exchange). When they ask where I found it, I not only reveal that it's generative, but tell them the specific AI and workflow as well!
  2. More importantly, this policy would serve to separate the above two groups. If someone violates this policy and gets caught, it's not longer a debate about what a tool is or about if we're being luddites. It's a blatant case of dishonesty!

To make things easy for the user, I think the base requirement is just to affirm that you are using generative AI, with no further details. This makes it less likely for users to skip the requirement because they are lazy. (Ideally, it would just be a checkbox.) If moderators or other users want more details, they can ask.

I think in practice, many users will share more details anyways to make the answer more credible. For example they may add "I only used ChatGPT for brainstorming" or "I only used Claude for proof checking" or "GPT-4 wrote the entire answer, but I carefully checked it".

I think in the long run, this kind of policy will probably me sufficient. As we get more used to generative AI (and it matures), we will get more efficient at checking their accuracy.

If not, I think that this policy is at least uncontroversial in the short-term and that it will provide helpful data to inform a more long-term policy.

$\endgroup$
1
  • 21
    $\begingroup$ The problem is not "plagiarism". The problem is that it generates crap and wastes everyone's time. $\endgroup$
    – Arturo Magidin
    Commented Apr 27, 2023 at 14:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .