Should AI-generated posts be allowed?

Question

As you may be aware, the artificial intelligence chatbot ChatGPT has been getting some negative attention on Stack Exchange as of late, its usage notably being completely banned on Stack Overflow.

Other sites are now discussing whether to either ban ChatGPT-generated answers or allow them on the condition that they meet quality standards. Two big concerns that have been raised is the issue of plagiarism and citation with respect to chatbot-generated content as well as the potential for a chatbot-generated answer to sound reasonable but contain factual inaccuracies that most honest, knowledgeable human posters would not make. For example, I posed the following question to ChatGPT:

In the original, prime timeline of Star Trek, how does Captain Kirk die?

Many of us know the answer to this question. He dies in the film Star Trek: Generations (1994) while disabling a weapon of mass destruction on the planet Veridian III in the (in-universe) year 2371. Jean-Luc Picard buries him in a makeshift, shallow grave marked with his Starfleet uniform insignia.

ChatGPT's answer was:

In the Star Trek universe, Captain James T. Kirk, played by William Shatner, does not die in the prime timeline. He is a central character in the original Star Trek television series and appears in several Star Trek films.

In the Star Trek films, Kirk is shown to be an aging captain who eventually retires from Starfleet. In the film "Star Trek Generations," which takes place in the prime timeline, Kirk is shown to have died at some point before the events of the film. However, the circumstances of his death are not depicted in the film.

It is worth noting that there are several alternate timelines and continuities in the Star Trek franchise, and in some of these, Kirk's fate may be different. However, in the prime timeline of the original Star Trek series and films, Kirk does not die.

At first this looks like a half-decent middle school essay, but at second glance (more than given to many answers), it is a self-contradictory, factually incorrect piece of garbage. It first claims that Kirk does not die in the prime timeline (factually incorrect). It then mentions the prime timeline film in which the character does, in fact, die, and claims that he died before the film! It claims that the circumstances of Kirk's death "are not depicted in the film", while in fact the film's depiction of Kirk's death is arguably its climax. For its last paragraph, it doubles back and re-establishes its original claim that Kirk does not die.

Answer 1

I propose that we follow other network sites in banning AI-generated copy-pasted content. This stuff is (to use a technical term) dangerous bullshit. As you've noted, it looks plausible and well-written, but to an expert eye (sometimes even a non-expert but humanly intelligent eye) it's clearly nonsense. It's one of the facts of SE that people sometimes upvote incorrect answers that look plausible - not every voter has the expertise necessary to judge whether an answer is true or false. So if we allowed this stuff, we'd run the risk of getting upvoted answers that are not only incorrect but were never even written by anyone thinking they were correct - the computer-generated equivalent of shitposting.

I already posted about this in chat:

If you see something that looks at first glance like an answer but to a knowledgeable eye is clearly nonsense, feel free to mod-flag it.

And, sadly, AI-generated answers have already been spotted recently on SFF.

So yes, let's have none of this nonsense, and please be proactive in flagging apparently AI-generated posts. (I have very little experience with AI-generated text or how to tell if a post is AI-generated or not, but there are mod-only spaces where I can get advice from people smarter than me at this kind of thing. Let's not discuss publicly how to detect these posts, beyond the very simple criterion mentioned above, otherwise the people posting them might make them less easily detectable.)

Using AI as a tool to assist generating good quality content might be reasonable, in some cases. I'm not proposing (yet) an outright ban on any answers where any kind of help from AI was used. But using AI, or other tools, to generate a large quantity of answers, without regard to whether those answers are correct and actually answer the question on which they are posted, is not acceptable. Content copy-pasted directly from ChatGPT output must be attributed to its source, or risk falling afoul of existing rules against plagiarism as well as new policies against ChatGPT specifically.

Answer 2

I decided to get some input from the offending party itself, so I've created an account on OpenAI and asked ChatGPT the following:

Should AI-generated posts be allowed on the Science Fiction & Fantasy Stack Exchange?

To which it answered:

It is ultimately up to the moderators and community of the Science Fiction & Fantasy Stack Exchange to decide whether AI-generated posts should be allowed on the platform. However, it is important to consider the potential impacts of allowing AI-generated content on the quality and integrity of the discussions on the platform.

Some potential considerations might include:

• The quality and accuracy of the content: AI-generated content may not always be of high quality or accurate. It is important to ensure that any content that is allowed on the platform meets the standards of the community and contributes to meaningful and useful discussions.
• The transparency of the source: It may be important for users to know the source of the content they are interacting with, particularly if it is an AI. This can help users to better understand the context and limitations of the content and make informed decisions about how to engage with it.
• The potential for misuse or abuse: Allowing AI-generated content on the platform could also present potential risks for misuse or abuse, such as spamming or impersonation. It may be necessary to put safeguards in place to mitigate these risks.

Ultimately, the decision about whether to allow AI-generated content on the Science Fiction & Fantasy Stack Exchange should be based on a careful consideration of these and other relevant factors, and should be guided by the goals and values of the community.

Make of that what you will.

Answer 3

As technology is moving fast, out of curiosity I just asked ChatGPT-4 the same question:

In the original, prime timeline of Star Trek, how does Captain Kirk die?

to which it responded (mostly correctly as far as I can tell):

In the "Star Trek" franchise, Captain James T. Kirk dies in the film "Star Trek: Generations" (1994). In this movie, which is part of the prime timeline of "Star Trek," Kirk dies while helping Captain Jean-Luc Picard stop a mad scientist named Dr. Tolian Soran. Soran's plan involves redirecting the path of an energy ribbon known as the Nexus in order to enter it and return to a paradise-like reality it contains.

Kirk initially appears to die early in the film during an incident aboard the U.S.S. Enterprise-B, but it is later revealed that he was actually pulled into the Nexus. When Picard, who also enters the Nexus, meets Kirk, he convinces him to leave this idyllic reality to help stop Soran. During their efforts to prevent Soran from destroying a star (and consequently a planetary system) to alter the path of the Nexus, Kirk is fatally injured when a metal bridge they are on collapses. His last words express his contentment, as he feels his life had made a difference.

Surprisingly, it also gets the info on the other captains (Archer, Picard, Sisko, Janeway, Burnham) right!

I also tried to trick it:

In the original, prime timeline of Star Trek, how does Captain Obi Wan Kenobi die?

to which it responded:

Captain Obi-Wan Kenobi is not a character from the "Star Trek" universe; he is a significant figure in the "Star Wars" franchise. In "Star Wars," Obi-Wan Kenobi is a Jedi Knight and Master, and he plays a pivotal role across several of the films and series within the franchise.

I still don't think that AI-generated responses should be allowed here as even though it got these tests right, it could fail on others, but it's a datum.