Is it OK to generate parts of a research paper using a large language model such as ChatGPT?

Question

Large language models (LLMs) are complex neural networks that can capture and represent world knowledge present in the human-generated text as their weights. LLMs are trained on a large collection of textual documents; for example, billions of webpages that are crawled from the web by the Common Crawl project's web crawler. Moreover, an application of LLMs is to generate text in response to a query which can be a set of keywords, a question, or even a simple sentence.

For example, the conversation in the image below is generated by an LLM called ChatGPT, created by OpenAI (a startup company working on LLMs).

The text in the box with a slightly darker background is generated by ChatGPT in response to the question on the first line.

As you can see, the generated response is similar (or even indistinguishable) to a reply that a human might have given to the question on the first line.

Now, my question is: is it OK to generate parts of a research paper, for instance, background or parts of the introduction section, using an LLM such as ChatGPT? Naturally, I would give appropriate credit to the creators of the model.

Answer 1

There are some aspects of academic writing that you need to consider if you want to use AI-generated text passages. I'll address four of them in my answer: Licensing, Plagiarism, Argumentation, Referencing.

Licensing issues

Language models are trained on source texts, and these source texts are typically harvested in great volumes from publications (online or print) without much concern for the license under which the text was published. Thus, the input for the language model may containmaterial that is not licensed for this type of processing. Consequently, the status of the model and of the work generated by the model is a legal gray area. Here's an article that discusses this "legal minefield" for AI-generated images, some of which are based on the OpenAI software. Much of the discussion applies directly to AI-generated texts.

Accusations of plagiarism

To my understanding, AI-generated content depends on associations between training data and classification data. Input data might be for instance a collection of photographs together with a classification matrix. This matrix expresses whether each image contains e.g. a car, a bicycle, a staircase, a zebra crossing, a traffic sign etc. Training an AI basically means to develop a mathematical model that expresses the learned associations between the images and the classifications. The resulting model can make a prediction whether an image that was not part of the training data also shows a bicycle. In this way, the model functions as a classifier. But you could also reverse the model logic by asking the model to generate an image that contains a bicycle. The AI model will use the learned associations between image information and classification to produce an AI-generated image of a bicycle.

A potential problem in this process may arise from data scarcity. If the training data contains only very few images of a bicycle, the generated image will be very similar to the input. This means that if e.g. the prompt for your text generator is highly specific, the output text may be very similar to the few input texts that the AI can draw from to generate the text. This is called AI plagiarism.

So, the AI-generated text may be a direct plagiarism of another published text. But to you as the end user, there is no way of knowing how similar it is, and there's no way of checking the actual texts that the model used to generate the text based on your prompt. This means that if you publish an article that contains AI plagiarism, and if somebody realizes the similarity between your article and other published material, they can rightfully charge you with plagiarism. Your only excuse would be that you didn't write that section, the AI did. But in an academic context, this will still ruin your academic reputation.

Argumentation

Academic writing depends on coherent and cohesive argumentative elaboration. While AIs have made huge progress in that regard, this is still something that needs to be checked very carefully. AI models don't understand the argumentation of an input text in any sense that relates to human understanding. At best, an AI model generalizes over argumentative patterns recurring in several input texts, and it uses these generalizations to create a text that resembles these argumentative patterns. It is your task as an author to ensure that the generated argument is indeed valid. You can only do this if you know the subject matter well enough – but you can ask an AI model to generate a text for a topic that you're completely ignorant of. Much responsibility is required at this point.

Your example text illustrates this. To me as someone who isn't an expert in reinforcement learning, the description sounds reasonable. But I can't judge whether it's actually correct. I would never dare to use this passage without reading up on the difference in detail so that I can verify the passage. But then, I could just as well write the section myself.

Referencing

One rule that every first-year student should learn is that academic writing needs proper referencing. If you make claims, you should back up the claims by references. If you report previous work, you need to provide sources.

However, an AI model doesn't know about referencing. It may learn that after particular text patterns other typical text patters occur. So an AI may learn that sentences from the academic genre very often end in sequences like (ABCDE 1234). because very often, the input texts will provide references for a statement in a sentence at the end of the sentence, and the format of this reference is often (Author Year). However, the AI doesn't know what this pattern means. It doesn't know that you can't use the most probable letter-number sequence here, but that it has to refer to an existing source. Obviously, this is less than useless – it's actively invalidating the academic quality of your text.

Alternatively, the AI may not produce any reference, which is also unacceptable and requires your intervention as the author. The generated text in your example doesn't contain references – but it contains many claims for which I as a reviewer would ask for sources to back up the claims. Again, it would be your task as the author to find sources for each of these claims. And since the AI didn't use sources in any sense of term "source" as used in academic contexts, it may even be impossible to provide appropriate references for an AI-generated text that satisfies the very strict standards for referencing.

Summary

AI-generated texts are an interesting development, but it's a topic that is so novel that there are many unresolved issues (both legal and ethical), and also many aspects that ask for a high degree of responsibility and knowledge from the author.

My assessment may change in the future, but as of today I'd advise against using AI-generated texts in academic writing.

Answer 2

No, it is not acceptable to use a language model such as ChatGPT to generate parts of a research paper. Research papers are expected to be the original work of the author(s), and using a language model to generate part of the paper would be considered a form of plagiarism. Additionally, research papers require a high level of accuracy and attention to detail, and a language model may not be able to provide the level of accuracy and precision that is needed in a research paper. It is always best to do your own research and write your own paper.

This is the answer given by ChatGPT.

It doesn't recommend using itself for generating a research paper.

Answer 3

I think, right now, it remains fairly dangerous to do so, at least without extensive oversight. I'd apply similar logic to other software tools that have been around much longer, like grammar or spellchecking software - it's okay to use the software to suggest or assist the author writing, but at the end of the day the author needs to be able to take responsibility for what is written and make the decisions.

LLM AI-generated text has gotten quite good at mimicking real human writing, but it isn't capable of doing the sorts of logical checks that humans need to do when they write. I'm especially worried about writers who use these tools because they do not feel confident in their own writing ability, particularly in a language they're less comfortable in. They may know their topic, but not understand the generated writing well enough to verify it.

Remember that research is meant to be somewhat novel; an AI can only be trained on what already exists. It might generate good summaries of topics that have been summarized many times before, but that doesn't mean it's equipped to actually synthesize knowledge in a forward-looking way. In my experience reading examples so far, I've certainly seen some impressive language constructions, but they feel to me a lot more like a novice university student trying to smash together an essay at the last minute, especially one trying to inflate their word count to get as close as possible to some instructor-mandated length. They have the essay feel down, but not the content. The example you posted seems representative of that - the first sentence, even:

The main difference between (adjective) (noun) and (noun) is (adjective)

Whoa, that's like, profound. Note that none of what is written elaborates on what it means to be "deep" in the context of reinforcement learning. It would be absolutely worthless to have any of this in a paper introduction. Even if the text were better, without citations to where the work came from, you have to admit that you're just trusting an AI's compilation of some library - that's not the same as you personally reading and trusting and vouching for the information. It's not merely the creators of the tool that need to be referenced, but the creators of the knowledge.

As others have mentioned, I do think disclosure and credit is important, but right now I think there are more serious issues to be concerned with.

Answer 4

#GivingCreditstoChatGPT is an act of supporting plagiarism!

I am not familiar with legal/copyright issues, so I will not address them here.

Inspite of its capability to generate well-structured "informative" text, ChatGPT is far from being intelligent (and consistent). It is very dangerous to blindly rely on this tool.

For example:

What is p-value?

A p-value is a statistical measure that is used to determine the likelihood that a given result occurred by chance if the null hypothesis were true. If the p-value is low, it suggests that the null hypothesis is unlikely to be true. On the other hand, if the p-value is high, it suggests that the null hypothesis is likely to be true.

Another answer for "What is p-value?"

‪A p-value is a statistical measure which is used to indicate the probability of a given hypothesis being true. It is calulated by dividing the observed results of a test by the expected value of the test‬. The lower the p-value, the more likely the hypothesis to be true.

The best answer (the 4-th try)

A p-value is a numerical value used in statistical hypothesis testing that represents the probability of obtaining a test statistic at least as extreme as the one observed, assuming that the null hypothesis is true.

If I have a student, and he/she keeps giving me a different answer (ranging from far from being correct to very close) everytime ask him/her the question "What is p-value?", I would say that this student just scrapes the entire internet, and gives me whatever the first result that he/she finds "useful" without knowing what it is. That is the problem of ChatGPT. It does not know. It does not understand. AI is not intelligent.

Also, that also implies that it is hard for others to locate the source that you get information from. The answer you get for asking the question above on ChatGPT is not fixed. Eventually, it will be impossible to do that.

Last but not least, in my opinion, ChatGPT could be an useful tool for productivity, and there is no reason why you should not use it if you found it helpful for your work as long as you acknowledge its shortcomings. Don't trust it blindly.

Answer 5

It's worth mentioning that OpenAI expressly forbids using their API to generate content without clear attribution including how the AI was used. (Note: this is applicable at least to the OpenAI API Platform, and may be applicable to other language models, though it is unclear if it applies to chatGPT.) This is in addition to the ethical and legal issues on the publication side that the other answers discuss.

From OpenAI's website (emphasis mine):

Content co-authored with the OpenAI API policy

Creators who wish to publish their first-party written content (e.g., a book, compendium of short stories) created in part with the OpenAI API are permitted to do so under the following conditions:

• The published content is attributed to your name or company.

• The role of AI in formulating the content is clearly disclosed in a way that no reader could possibly miss, and that a typical reader would find sufficiently easy to understand.

  ....

For instance, one must detail in a Foreword or Introduction (or some place similar) the relative roles of drafting, editing, etc. People should not represent API-generated content as being wholly generated by a human or wholly generated by an AI, and it is a human who must take ultimate responsibility for the content being published.

The website also has a suggested blurb that can be used:

The author generated this text in part with GPT-3, OpenAI’s large-scale language-generation model. Upon generating draft language, the author reviewed, edited, and revised the language to their own liking and takes ultimate responsibility for the content of this publication.

I suspect that other large language models will have similar use policies.

In short: it's not just up to the venue and the authors. By using a large language model published by some company, you are obligated to read and follow the terms of use for that model, which may include proper attribution.

Answer 6

It's difficult to give a conclusive answer. AI-assisted writing is fairly new, and the legal and ethical understanding of it is in flux. I would caution against using AI-assisted writing in published work for a few reasons related to plagiarism and copyright.

AI tools are trained on large datasets, often including published, copyrighted material. You can cite the tool itself, but what about the publications it learned from? Should those be cited as well? Are you certain that it won't produce a paragraph similar enough to a former work to constitute plagiarism? Do authors have a responsibility to ensure that the tools they use aren't plagiarizing or infringing on copyright, or does that responsibility lie solely with the developer of the tools?

At the moment, there are no clear legal answers. It would be prudent to wait until AI technology—and legal and ethical opinions of it—mature.

Answer 7

The example in the question body generates the impression that large chunks of text are to be generated automatically and submitted without editing. However, I'd like to approach the question as asked in the title and more from the perspective of what I consider acceptable usage of AI models as tools.

Summary

• I think such AIs can be valuable tools to help generating initial draft texts, once the current hype is over and they get to be used for what they are good for. Though under much closer human supervision than the question envisions: These models are giant autocompletion engines, and could/should be used as such; they are not trained to predict reality or truth (other than the reality of what may occur in a text).
• We are already using deep models - possibly without realizing - for several tasks during research and paper writing without particular concerns.
• Whether we find it acceptable or not to use such tools to help with proper scientific writing, we'll anyways have to learn how to deal with auto-generated spam papers.

---

TLDR

First of all, I'm in STEM (for this question: as opposed to sciences/disciplines that are more working directly on language such as literature or also law). The science I generate is partly experimental and partly computational (as in crunching the numbers of those experiments). Fields whose core is in the language may need to develop their own standards.
(Also, I'm into machine learning, but not into natural language processing.)

However, keep in mind that we are already using automated tools:

• Automated theorem proving in maths

• Google uses a natural language processing model in their search engine.

• Translators such as deepl come to mind as well. Particularly in case you're not a native English speaker and look for translations of sentences explaining your thoughts in your native language.

• Spell checkers are state of the art. Thesauruses or synonym and antonym dictionaries have been around since ever, nowadays there are also rephrasing tools. Newer tools like grammarly were explicitly recommended in a scientific writing course I lately attended. Note that grammarly can rewrite sentences to make them shorter and more legigible.

• I happily use tab completion, including the variety that predicts what word I want to write next when writing a text.

• Similarly, speech (voice-to-text) or character recognition (OCR) software uses deep learning.

• At my secondary affiliation JKI, colleagues develop cadima for helping with systematic reviews. Behind the scenes, they work with (and on) natural language processing AI models for that tool as well.

AFAIK, using such tools is not considered questionable.

As a side note: when I worked on my very first paper, I was at an institute that still had a secretary/assistant. She drew the illustrations about our methodology according to our instructions and got properly acknowledged similar to how one would acknowledge who helped with field or lab work, but did not contribute intellectually to a paper. There is nothing ethically questionable in this practice, neither.

I'd tend to "acknowledge" models like other software relevant for the paper: either in the text or as reference if there's a scientific publication about it. Since that is done not only for, say, the package that computes a statistical model but also for packages like ggplot2 that produce diagrams and figures, why not for packages that generate (parts/early drafts) of the text.

---

The aforementioned scientific writing course discussed a work flow that has distinct steps for deciding which content, text generation (and recommended freewriting, possibly using voice-to-text software) followed by editing, revising (and reorganizing) the produced text chunks.

GPT-3 and similar models predict how a given text continues. This is IMHO important to understand: It's not answering questions in the sense we'd discuss question and answer, for those models, answers are text that is predicted to come after a question.

It also means that some behaviours that may be considered less desirable are actually what follows from the precise prediction task of those models.

In particular, since the "answers" are in a sense extrapolated from the initial text/keywords, we should expect that this process is unstable (see the different "explanations" of p-values in @edelweiss' answer). (I also wouldn't be astonished if this extrapolation behaviour turns out to be related to the possibility of the model generating abusive or heavily biased language: such extrapolations have a tendency to amplify things)

I think we also need to keep in mind that when talking about writing a scientific paper, we're talking about novel (as in new), but real (not fictional) content. This means we're asking for even more extrapolation: what we want to write about must be outside the body of text known to the model.
And, GPT-3 is not a domain-specific model for predicting how a scientific text goes on, its training base comprises all kinds of text, including invented ones such as fictional or even fantasy texts. Note that the AI generated content has often been described as dream-like - which may be how we humans perceive such instability in the prediction. (It would be interesting whether such a model trained purely on scientific texts and textbooks would produce less fantasy)
(We also should not wonder about "fake news" being produced; and it should by its set-up be able to predict also how fake news stories go on.)

This suggests to me that these models may best be used as very sophisticated auto-completion tools (with tight control by us of the content - and that possibly tighter control is needed for generating/auto-completing parts of a scientific text than for, say, a web page or blog post on some well-known topic), and maybe as a tool for the free writing stage, generating (even a variety of) sentences or paragraphs from maybe a list of bullet points.

I'd gladly use an AI tool with a voice-to-text of my talk as starting point, applying a "scientific paper" language style to generate an initial pre-draft version of a manuscript. ...*dream*...

When writing a scientific paper, we provide (and are responsible for) the knowledge content. I care less about "who" puts the thoughts into nice-to-read English, and personally find the use of also very sophisticated tools entirely acceptable.

---

relevant blog posts and papers

• Artificial intelligence is getting better at writing, and universities should worry about plagiarism

• Using Internet based paraphrasing tools: Original work, patchwriting or facilitated plagiarism?
  I'd expect the observed indicators (which are mostly rather clumsy) to vanish rather soon as the tools become more sophisticated

• In an AI world we need to teach students how to work with robot writers

• OK computer: to prevent students cheating with AI text-generators, we should bring them into the classroom

• ‘Tortured phrases’ give away fabricated research papers and the original Tortured phrases: A dubious writing style emerging in science. Evidence of critical issues affecting established journals
  From the nature news (haven't yet read the arXiv preprint) it is not clear to me whether spam papers were newly generated (by AI) or whether plagiarism was tried to hide behind automated forth-and-back translation.

(One of these posts mentions that also plagiarism detection software will become more sophisticated, establishing "writing fingerprints" of people. Unless these fingerprints are transparently computed, and based on proper science, I'd have the same type of concerns with plagiarism accusations on such a basis that I have about predictive policing, in particular based on intransparent (and likely not well validated) models.)

---

I don't want to speak (much) on the licensing issues:

• If ChatGPT is found to have been trained on material that was not properly licensed (see @AustinHemmelgarn's comment on the respective lawsuit) it will likely be taken down. A different AI model trained on a legally available body of text will likely be developed (possibly by someone else) and provide similar functionality - so the rest of these answers is not affected.

• As Schmuddi discusses, copyright of content generated by AI is not clear.

• However, I'd expect for an AI model (software) that's used basically as autocompletion tool, i.e. under close (intellectual) supervision of humans, that the copyright lies with those humans. This is IMHO the more relevant and realistic scenario for a proper scientific text being written using an AI.
  (Consider: if Microsoft were to loose a lawsuit for copyright infringement for something incorporated in the Word software. Whatever happens to the future of Word, your text produced with the help of Word stays yours.)

Answer 8

As long as your text is clear and your research is sound, you may use any tool you prefer to aid you in the writing. Indeed you'll have to proofread the text and make sure that everything is correct. You'd have to check if the license of the particular model you wish to use permits you to use it in such a way or if there are some limitations/attributions required.

If you believe that the text generated by such models is good enough, I don't see why you should not use it.

EDIT: as pointed out by other people these kinds of models may have licensing issues with the data used to train them, this is indeed a big problem. Also as pointed out by others, these models may plagiarize other authors if they have little information about a topic. This said, I stand by the idea that if the final text is a publishable text, with a sound research and that satisfies all the requirements to be publishable, you may use any tool you prefer to write it.

Answer 9

An answer to the literal question in the title, "Is it OK to generate parts of a research paper using a large language model such as ChatGPT?", is probably yes. At least if your research paper happens to be about LLMs, trying to understand how and why they work and how to improve them.

A fairly recent and somewhat extreme example is this article, mostly written by GPT-3. The (co)authors asked the LLM to write a paper about itself (using multiple queries). As a consequence, that paper contains a lot of sections written entirely by the LLM.

Whether or not something like that really is acceptable (or should be) is, just as mentioned in the other answers, still up for debate. This particular paper is still in peer-review (or it was at least in June 2022, when the (co)author wrote an opinion piece on their own paper and the ethical and moral questions it poses in Scientific American).

Answer 10

I'll add to the other answers that some conferences explicitly forbid some use of language models, so if you plan to use some language models you should make sure it doesn't violate some policy of your publication target.

E.g., https://icml.cc/Conferences/2023/llm-policy:

The Large Language Model (LLM) policy for ICML 2023 prohibits text produced entirely by LLMs (i.e., “generated”). This does not prohibit authors from using LLMs for editing or polishing author-written text.

Answer 11

Times have changed. These days the answer is broadly "yes", but 1) you can't list the AI as an author, 2) you must disclose where you used AI, and 3) you are still responsible for the content of your paper, even if it was written by AI.

The use of artificial intelligence (AI) tools such as ChatGPT or Large Language Models in research publications is expanding rapidly. COPE joins organisations, such as WAME and the JAMA Network among others, to state that AI tools cannot be listed as an author of a paper.

AI tools cannot meet the requirements for authorship as they cannot take responsibility for the submitted work. As non-legal entities, they cannot assert the presence or absence of conflicts of interest nor manage copyright and license agreements.

Authors who use AI tools in the writing of a manuscript, production of images or graphical elements of the paper, or in the collection and analysis of data, must be transparent in disclosing in the Materials and Methods (or similar section) of the paper how the AI tool was used and which tool was used. Authors are fully responsible for the content of their manuscript, even those parts produced by an AI tool, and are thus liable for any breach of publication ethics.

Answer 12

No, using language models like ChatGPT to generate parts of your research paper is not advisable. Several issues arise from the direct usage of artificial intelligence services such as OpenAI ChatGPT, BingAI, BardAI, and so on in the education sector. Some of the common issues are:

1. Plagiarism Accusations, 2. Licensing Issues, 3. Referencing Issues, 4. Originality of the Sources Issues

If you are confused about any concepts, you can ask these services to explain the meaning. However, after grasping the concepts, you should do your own research, learn from it, and use it in your research papers.

One more thing. Someone once said, "In your life, all kinds of troubles will come looking for you all the time. You don't need to go out and look for them."

A legal team in New York City recently got reprimanded by a Federal Judge for writing a legal brief with the help of ChatGPT. Turns out, none of these cases that ChatGPT had cited to the lawyers actually existed.

NYC judge scolds lawyer for using ChatGPT to write brief full of ‘legal gibberish’; attorney swears robot ‘duped’ him | Yahoo! News | Friday, June 9, 2023

To conclude, I think that it is okay to use ChatGPT within ethical and code of honesty constraints, but beyond that, you are going to attract all kinds of negative attention and consequences you never expected to receive.

Good Luck to you.