How do we want to handle AI-generated answers?

Question

New software has been released recently named ChatGPT that is causing quite a stir around the network:

• Could ChatGPT be a viable way to answer people's questions?
• Temporary policy: ChatGPT is banned
• Ban ChatGPT network-wide

Slate, a Stack Exchange Community Manager, posted a comment on the Meta.SE link above:

We have begun internal discussions to identify options for addressing this issue. We’re also reading what folks write about the topic on their individual sites, as one piece of assessing the overall impact. While we evaluate, we hope that folks on network sites feel comfortable establishing per-site policies responsive to their communities’ needs.

This is fairly new territory, so I'm interested in gathering the UNIX & Linux community's thoughts on AI-generated answers. There are several key points that have been raised already, so I'll seed the discussion with some of them:

• You must attribute the answers to the source (sources: Machavity's answer; Makoto's answer)

• Generated answers may not be correct (sources: Nineberry's answer; Journeyman Geek's answer)

Do we want to prohibit AI-generated answers? Do we allow them, with attribution?

Answer 1

I think such answers should be banned entirely, and anyone posting ChatGPT answers without attribution should be banned with prejudice.

If the asker wants an answer from an AI, they can go to ChatGPT directly.

I personally don't think that even ChatGPT answers with attribution should be allowed, but I am willing to compromise on this point providing the attribution is provided up front (not at the end of the answer and certainly NOT in an edit after the answer is first posted), and providing the entire ChatGPT text is in a block quote so nobody can mistake it for a directly written human answer to the question.

Now this raises some questions, "What about people using ChatGPT for grammar and spelling? What about if the answer poster carefully checks the ChatGPT answer before posting?" My answer to both is, that's only fine if the answerer then writes or re-writes the answer themselves in their own words and takes responsibility for every word of it being what THEY actually want to say. This is in alignment with the ChatGPT terms and conditions, and with how we handle any other sort of source of information.

In short, I think we should handle ChatGPT answers the same as we would handle copy-paste from other websites without attribution, but with added prejudice because of: wasting everybody's time, the difficulty of detection, and the need to dissuade other people from the easy "rep-farming" that will occur if we hold a tolerant stance on ChatGPT answers.

Answer 2

I think it is really important to understand the thing that we are banning or not banning.

As far as I'm aware ChatGPT and all other successful AI in this space are NOT doing original research to produce an answer. Eg: they are not running Linux commands they suggest, or writing a proof of concept.

These models are trying to crack the Turing Test with ever higher success rate¹. What's really interesting is the extent to which the Chinese Room argument has proven more meaningful than it first appeared: There is a very large gap between convincing humans that an AI understands something and the AI actually understanding it².

The information provided by ChatGPT is very intelligently collated information from across the internet. But this sets its position in the world similar to that of Wikipedia and Google Search. These are very fine tools, but they should never be considered authoritative sources of information³.

Unlike Wikipedia, ChatGPT answers are very hard to trace. With copy-paste answers from Wikipedia, we can not only trace the origin of bad answers, but actually go and correct it at source!⁴. As far as I know, ChatGPT has no such capability.

The sheer volumes that have been seen make them a real problem that needs to be dealt with firmly.

---

Thanks to Kamil Maciorowski for this comment:

If this answer is true then it's very relevant.

That answer nails it. Discussing with those I know in the field, I believe that answer is very true. These AIs are super smart at word play. Really very smart. But they are not conscious. Not yet.

E.g: The last time I heard "entity linking" across many unconnected sources remains a bit of an unsolved problem. If you see a name "Mickey Mouse" in a document then it's hard to be sure if the document was discussing the Disney character or using it as a euphemism like "Mickey Mouse operation" to mean silly or poorly run.

Besides that, AI has made some amazing advances in recent years with various "models" for various specific tasks: image recognition, image generation, text generation. And logical reasoning has long been relatively trivial in AI. But one thing that remains frustratingly out of reach is a good way to connect these different models into a single system.

In short people should not hold their breath waiting for a really great language model to be connected to a really great logical reasoning engine.

---

To my mind, the idea of allowing AI answers onto SE must wait until AI can take a questions, read some manuals, and then run some tests to prove the solution worked.

I.e.: AI answers must wait until the AI actually understands what it is talking about.

---

• ¹The next frontier is fooling people with subject matter expertise
• ²My own experience with interviewing tech candidates for a role is that even some humans can pass the Turing Test but ultimate show zero understanding of the real subject matter when presented with our trivial tech tests.
• ³Wikipedia even has a ban on original research
• ⁴My only ever wikipedia update came from just this case.

Answer 3

I strongly believe we should ban answers generated by ChatGPT -- or any other AI, for that matter. I actually think (concerning U&L topics) that these answers are bad, except when the question is a very easy one. These answers might look good at a first sight, but are often generic, miss the point, and lack the real-world knowledge and experience of all the intricacies in doing real Linux sysadmin work.

StackOverflow has now empowered moderators to ban users up to 30 days that post ChatGPT-generated content, and added a banner on their site. We should perhaps do the same.

Answer 4

The worst of all is well worded by the OP of the meta-stackoverflow link you suggested :

The primary problem is that while the answers which ChatGPT produces have a high rate of being incorrect, they typically look like they might be good.

In my letter to Santa… I asked for… a bottle of wine. I would not really mind if getting vinegar in return. But… I would hate if the vinegar was packed into a bottle of premier cru.

This would certainly not only mislead the author of the question but also discourage members of the community to contribute and provide their own true knowledge based on their own true experience.
The latter point being in contradiction with the philosophy I understand in a comment from terdon :

I think the idea is we'd rather have more answers than one, accepted one.

I acknowledge that AI-generated answers could be good at answering those questions from students asking the community to do their homework.
But do we really want that sort of questions here ? I don't think so.

---

However, I would strongly support the idea of a bot pushing an IA-generated answer on old && unanswered questions.
This would necessarily provide more value than the disgraceful bump of the community bot.

Answer 5

I vote for a blanket ban on AI-generated answers, just like Stack Overflow.

The primary problem for AI-gen'd answers, as is the case on SO, is their high rate of inaccuracy disguised in a good-looking form. This goes against SE's target as a repository of useful knowledge. We want treasure, not elegant garbage. This is not something AI can reliably generate for us.

Attribution is not even the second problem. What comes next about AI answers is the human incentive behind that. We are expecting users to provide quality answers, or at least, have an intention to add quality answers. Users coming with verbatim copy-paste from AI output are unlikely willing to contribute positively, especially when posting in volumes. They're only coming for rep farming or what have you, and they're more likely to add moderation workload than valuable content.

In this respect, users utilizing AI to improve their answers pose minimal problems to us, if any. These answers are in essence human-composed content, decorated with AI-aided language & expression. This does not violate the intention requirement as described in the previous paragraph.

Finally talking about attribution. More often, lack of attribution alone is a minor issue if it is one after all. If a decent answer lacks proper attribution, we ask for clarification in the comments and fix it up if needed. If it's a bad answer, we're not even concerned whether it's properly attributed. Attribution alone has minimal influence on the quality of the answer.

Answer 6

My personal opinion about this is that I'd like to see correct and well-written answers. If an entity can explain the underlying issue that gives rise to the problem, show with examples that the given solution solves the problem for the user in question, reason about assumptions made, respond to comments in a helpful way, and cite necessary sources, then, if they are an AI or not does not matter.

It is clear from what we've seen on the site so far that AIs are not yet at that point though.

I've removed most of the following paragraph as I no longer think we should allow generated answers at all, not even as attributed wiki answers. Answers written using ChatGPT or any other AI software should be community wiki answers and disclose that they were provided by a bot (no matter if the bot wrote it all or just helped with the technical bits). This is partly so that the answer can be appropriately reviewed by a human for the correctness and partly so that the reputation from votes on the site goes to users who use their own brains to solve issues. Failing to attribute the source of text that is taken from elsewhere is already not allowed on the site.

Artificially inflating one's reputation using an AI tool is a form of deception. This becomes extra serious when users use their reputation on this site to leverage influence in other places (which may well happen in workplaces or other social situations).

ChatGPT or other software should be allowed to help with the grammar of the natural language used in answers, provided the user proofreads the text before it is posted. This is akin to using tools like Grammarly, which I see nothing wrong with.

Summary:

1. Discourage AI-generated answers unless the AI is so good it's indistinguishable from a human (able to respond and correct their answers etc.) There is currently no AI that is that good. If there were, we would not detect them, so it would be pointless to disallow them.

   Note that "discourage" could well mean "suspending users using AI". If they have posted a clearly identifiably AI-generated answer as a community wiki answer, they should not be suspended and the answer should instead be reviewed by humans. The threshold for deleting these answers would be severely lower than ordinarily.

2. Material from external sources (which IMHO includes AI) must be attributed. This is already a site-wide requirement, see e.g. Users are calling me a plagiarist. What do I do?

   In the context of Stack Exchange sites, any copying and pasting of any amount of text or code that wasn't written by you is plagiarism if you try, explicitly or implicitly, to pass it off as your own work.

   Remember, you still have to write an actual answer, in your own words. A post that consists only of copied text, even when attributed, is not your work either. Use quotes sparingly, to support your own words.

   Ignoring this would normally result in a suspension, and possibly a network-wide suspension for repeat offenders.

3. Artificially inflating one's reputation by means of AI should not be allowed. It is already not allowed to artificially inflate one's reputation (or other metrics, such as the "people reached" count).

   Again, suspending users who do this is not uncommon.

Answer 7

Machine-generated content (vice (artificial) intelligence-generated content) is certainly useful as a tool, but machines are fundamentally unable to understand the full context of the question or answer.

Machines are only as good as their programming, and they are unable to defend or justify their results beyond the fact that their algorithm created it. With machine learning/neural network models, the results can even be stochastic vice deterministic.

The value of forums such as StackExchange is the exchange of ideas and the challenging of assumptions or beliefs. Machine-generated content cannot question its own algorithms or neural network training to justify its output, or more importantly, to adapt or update its output when presented with problems or flaws.

ChatGPT and other machine learning/neural network tools are just that--tools. Until a machine can participate in a discourse about a topic and grow from intellectual challenges, it must remain a tool in the hands of a human.

Answer 8

It's important to separate Artificial Intelligence from Machine Learning. ChatGPT is firmly the later, it doesn't do anything we expect from a intelligent creature, but seems intelligent from the emergency properties of the system. Like an ant hill, it's smart in the way it respond to stimuli based on "simple" rules. Given this, it obviously has limitations and do pretty stupid things, like other systems*, which is something that is not unheard of humans doing, yet it tends to be more successful in a specific set of scenarios, either by context or because it has better information.

My interactions with ChatGPT is that it can generate incredibly long response to tasks that I can then refine by hand, consult about circular references and implement algorithms without errors (which is something I usually do not consider myself capable of). And yet, it annoys me to no end that it gets the same things wrong about stuff that most humans get wrong: economic theories and concepts, history, philosophy, etc. The model is, after all, a popularity contest, and no amount of vetting will ever find them all, just the most popular ones.

Instead of any "banning" we should ask SE to actually implement ChatGPT as a pre-ask option for askers. If the askers are satisfied with the results, then we could review that by actual experts and post that as a pair of Q&A. Basically, embrace the thing, since it has the potential to offload much work that we do that we shouldn't have to otherwise, and only focus on verify the information. After all, it's easier to spot a wrong answer than to generate a right one*.

* And despite of this, human players preferred to work with it due precisely lacking those same human characteristics, like pettiness, vengeance, or feeling the blues.
** This is basically the "easier said than done" but adapted to the context