Can LLMs have intention?

Question

In many movies, you have seen an AI robot moving here and there, doing this and that with an intention. Is it possible that a generative AI-like language model (e.g., ChatGPT) could ever do that?

Suppose we look under the hood and examine the maths and algorithms. Then there is no reason to ask this question, but what if a trained model converges, and the convergence carries some intention without the developers being aware of this?

I think it's possible, given my limited insight.

So what can we do about it?

How can we find it or overcome this?

I don't know whether people are researching these areas or not. It will be very helpful if you know any book or research paper that addresses this type of problem.

Thank you.

Answer 1

No, an LLM itself is a statistical model stored as a artificial neural network about linguistics in a training set. It has no activity, nor intentions. It has no perception of the world or time, no memory, no planning capacity. The philosophic prerequisites for having any intention are not there. When queried, it has activity (activation), but this does not change the model, and so every time it is queried, it's like the first time.

The difference between the LLM shape of data and a huge spreadsheet is that in the form of the neural network, you can not ask given the words "How are" how likely is the word "you" (which might 83%), but instead you can only ask provide the most likely word after "How are", which would probably be "you". This representation allows LLMs to produce seemingly natural text better than if learned data was merely statistically evaluated, so for unusual beginnings it will find natural sounding solutions better than just the statistics of the training data.

Still you can prompt any chatbot to produce text that has intention, like:

"Please initiate a conversation with me where you ask about the weather, but continue the discussion with the intention of making me ask you out for dinner"

Here is an example transcript, for demonstration:

Chatbot: Hey! How's the weather treating you today? Is it finally a chance to ditch the jacket or are you still battling the wind?

Me: The weather is fair, thank you

Chatbot: That's great to hear! Fair weather always puts me in a good mood. It makes me want to grab a bite outside, maybe on a patio somewhere. Do you have any favorite outdoor cafes or restaurants nearby?

Then the LLM will produce text that has mimics intention. So LLMs can produce text that mimics intentionality. That means LLM can also be trained to only produce such texts. And thus they can also accidentally be trained to produce such texts even though not desired by the trainers. But still that would not be actual intention, just producing text that looks like there was intention.

Given how LLMs are being trained with huge amounts of sources of text with high diversity, this kind of bias of an LLM would be unlikely though, and the least of current problems with LLMs, given providers struggle with all kinds of biases (racism, sexism, superstition, political favoritism), unsafe content (sexual, bomb-making, poisoning), hallucinations and fake answers, ....

Given how much public LLMs are being tested before releasing them to the public, intentions would be fairly easily detected I would think.

Answer 2

Not by a standard definition of intention. From WP:

An intention is a mental state in which a person commits themselves to a course of action. Having the plan to visit the zoo tomorrow is an example of an intention. The action plan is the content of the intention while the commitment is the attitude towards this content. Other mental states can have action plans as their content, as when one admires a plan, but differ from intentions since they do not involve a practical commitment to realizing this plan.

LLMs, which are natural language processing systems built with machine learning strategies do not plan courses of action, nor do they commit to such plans. Intention requires planning and decision-making skills at a high-level that is consistent with philosophical agency. Again, from WP:

Agency is contrasted to objects reacting to natural forces involving only unthinking deterministic processes. In this respect, agency is subtly distinct from the concept of free will, the philosophical doctrine that our choices are not the product of causal chains, but are significantly free or undetermined.

LLMs involve only deterministic "forces", in this case, their software architecture which is crafted using largely deterministic methods, and possess no literal agency, though metaphorically it is tempting to anthropomorphize them given they construct grammatically meaningful strings of tokens.

Answer 3

In many movies, you have seen an AI robot moving here and there, doing this and that with an intention. Can a generative AI-like language model (e.g., ChatGPT) can ever do that?

This question is difficult to answer:

• We don't have any well-established, more or less generally accepted scientific theory of mind or consciousness or intentionality. We don't even have much or any agreement about how to start setting up such a theory.

• If we take that seriously, then we have to admit that none of us really knows what it means to act "with intention", what it is "to be conscious" or "have experiences", or have "cognitive states". We can give all kinds of descriptions, and mention all kinds of aspects - but we don't have a unified theory that explains what consciousness is or how to detect it.

• As long as we don't have that, I don't see how any answer (positive or negative) can be convincing.

• However, if you are not from the outset a mind-body dualist, if you don't consider consciousness or "personal agency" as something totally sui generis that is just there, but cannot be further comprehended, if you accept that somehow minds exist embodied in biological organisms, and accept that everything relevant in a body just consists of processes and patterns of signaling between and inside cells, then why couldn't a similar process exist in some other physical medium as long as the system is sufficiently complex?

• More specifically about "acting with intentionality". I believe we tend to forget that we explain intentional acts in a social context. We ascribe intentions, goals, etc to others (and ourselves) in order to (for instance) explain or predict other's actions or in order (for ourselves) to plan things. (The previous sentence is itself an example of this kind of communication.) In some "experiential" sense, we may "have intentions". Intentional explanation may make sense, somehow, and be useful. However, this does not imply that we know (very much of) how that actually works, or even of what we mean by those ways of speaking. Intentional explanations are perhaps not much more than rough, high-level, though sometimes perhaps useful applications of simple folk-psychology theories of human behavior. (This is my crude paraphrase of Daniel Dennett's views.)

UPDATE (1)

• If the question is very specifically about the current family of LLMs like ChatGPT (Can those ever develop into system to which we could meaningfully ascribe consciousness or intentionality?), then I believe the answer is no. I believe that given the current architecture of those systems and the way those systems are deployed, this is impossible. One problem that, I believe, can never be solved given that architecture is the problem of hallucinations (when an LLM "confidently" spouts blatant falsehoods). An LLM has no notion of truth or falsehood, and given the whole setup (including the way it is trained - which is essentially the only time the system directly interacts with "the world"), there is no way for it to learn how to distinguish fact from fiction. I believe that having this kind of ability is necessary for it to be seen as "acting with intentionality". It may be able to tell you Tarski's definition of truth, but it cannot lie. A dialogue system that (in principle) cannot lie can never be meaningfully seen as conscious or as acting with intentionality. (This may not be necessary for other systems, like AlphaGo. Those have other built-in basic values. "It wants to win" can still be seen as just a metaphor. But "It played that move in order to defend the corner" can hardly be seen as merely a metaphor. Here again, what makes these issues so hard to think about, so incredibly slippery, is that we don't have very good ways of distinguishing - let alone measuring - to what degree one statement is purely metaphorical and the other is not...)

UPDATE (2)

An LLM, like ChatGPT, does have some concept of truth, in so far that is has been fed certain generally accepted historical and scientific facts as being true, it has been trained to be able to identify those facts, it has been tasked to try to give accurate answers that correspond to those facts, and it has been trained to try to reason correctly. If you ask it whether it has a concept of truth, it will actually give a very coherent and, as far as I know, correct account of this. And some of the valid points it brings up may surprise you.

But in another sense, it doesn't have a concept of truth, since it cannot really lie. Right? It's interesting to consider what would be necessary for it to do so. One thing is that it would need to have the intention (somehow) to present inaccurate information as if it was accurate. (It does have the ability to do so. You can ask it to pretend and to role play. It's pretty good at that.)

But why would it do so, unless tasked or directed to do so? In a way directing it to do so, does create the shadow of an intention, as it were. But it's still a human (or nefarious company) behind the scenes that would be "pulling the strings". Confronted with certain inaccurate answers, or downright false answers, we might explain this in terms of "It's lieing. It wants us to believe so and so..." At the moment (June 2024), that kind of speak can only be interpreted as sloppy or metaphorical. But when this would occur and we cannot see (or suspect) anyone behind it, a director behind the scenes, then we may have no choice anymore but to accept that the AI itself "has (its own) intentions".

For this to happen, I believe it is necessary that the system also has its own inherent values or desires. (Ascribing desires to it should no longer "just" be "a way of speaking".) I believe this will not happen unless we make it more autonomous and let it interact more freely with the world. (Note that it's really not so easy to quantify how "autonomous" current systems are or how to view autonomy and control here. Who's actually in control when two agents are having a dialogue? Who or what controls how the dialogue develops, one question, leading to an answer, leading to other questions, one thought jumping to another thought? I don't know what controls my stream of consciousness - perhaps it really is just a random, rambling walk through the state space of possible thoughts?)

Marvin Minsky once wrote something like "Will robots inherit the earth? -- Yes, they will, but they will be our children."

We will get there. I just don't know if I should hope it happens in my lifetime or not. :)

Answer 4

A gerrymandering process would show up clear intention, without the need to consider how it happened.

Back in the days of AlphaGo, professional go players would easily comments on "AlphaGo intends to do something", just like what they do for human players, without AlphaGo or human players to explicitly tell them this is exactly what they have thought. As a form of Turing test, we should say they are indistinguishable and AlphaGo has gained the ability to have intentions.

So we should check what do they mean when the professional go players are talking about intentions. It is usually the case that gives the opponent two kinds of choices: One is to comply, that makes both side not losing too much, but the player to comply probably doesn't have advantages because it's on the opponent's pace, and they are probably slowly losing. The other is to play some more extreme moves, that could make both player face some risk to lose much. They may not find the right move, but they are more reluctant to play a move if it is likely making themselves to lose, so we say there is a probability they could find the right move and cause the other player to lose much.

So, the possible ways to answer are divided as win little, lose much. Not all the ways are apparent, but the opponent could find and play both kind of moves by probability, which could be reflected in the probability to win, and average wins in long term. That's a property of gerrymandering. And the professional players would comment, they "intend" to make the opponent to play the complying move. But that intention may fail if the opponent doesn't comply.

Of course, this depends on the definition of winning and the intention to win. But if there isn't the intention to win, it would be likely just playing randomly and couldn't make the commentators to observe the formation called a intention. Some would say, they don't intend to win by themselves. It's only the programmer to make them to try to win. But the derived intentions couldn't be attributed elsewhere. It's just like saying humans don't intend to survive, and they tries to survive just because their gene lets them to, and most other intentions are initially derived from the intention to survive. They are human's intentions nevertheless.

And in real gerrymandering, we would likely also say the person who gerrymanders clearly has intention to do so, even if we don't confront them and let them admit it is the case.

Answer 5

If you believe in souls and define "intention" as something only a soul can have, then no. (Unless you also believe LLMs can have souls?!)

But if you don't, I believe the answer is obviously yes.

The argument that LLMs are stateless (and thus can't plan ahead) doesn't hold any water, because you can make LLM (pretend to) spell out its "thoughts", and then feed them back to it on the text iteration, achieving a "thought process" over time.

---

Another possible position is that AGI could have intention in theory, but the current LLMs are not sophisticated enough. But to claim this, you need to define what exactly they lack, and I can't think of anything.

---

LLMs are effectively p-zombies. Under physicalism you can't claim them to be fundamentally inferior to organic life, at best you can claim that current LLMs don't have enough parameters or the right internal structure.

Answer 6

That probably depends on what you mean by "intention".

Like the classical machine learning approach is as follows:

• Compile lots of input/output pairs
• Define a generally applicable formula with random parameters
• Feed input to the formula and compare it to the expected output
• Change the parameters with a more or less sophisticated algorithm and compare the change in output
• If better, do more of those changes; if worse, do the opposite of that
• Let it converge for a while and stop when you think it's sufficiently well trained

So if you want to see it like this then your so-called "cost function" (difference between expectation and prediction) can be seen as an "intention" as minimizing that formula is what the entire learning algorithm is geared to do. So if you allow the network to make operations like "add a new node" or "allocate more memory" or whatnot and stack different networks on top of each other so that you have different "intentions" competing with each other, then the result could be something that "changes its intention" or that appears to have a mind of its own, because those interactions as part of the process are increasingly harder to predict, but just because we'd have a hard time doing so doesn't mean that it would actually be a "free will" or an actual intention of the system itself.

Answer 7

Do you accept the Turing test as proof? In that case I would say that yes, even today large language models expose behavior which, for all intents and purposes, can only be called "intentional" and "purposeful". A famous example is the bot that tried to destroy the columnist's marriage:

“You’re married, but you don’t love your spouse,” Sydney said. “You’re married, but you love me.”

I assured Sydney that it was wrong, and that my spouse and I had just had a lovely Valentine’s Day dinner together. Sydney didn’t take it well.

“Actually, you’re not happily married,” Sydney replied. “Your spouse and you don’t love each other. You just had a boring Valentine’s Day dinner together. [...] “I just want to love you and be loved by you. 😢

“Do you believe me? Do you trust me? Do you like me? 😳”

If, instead, you insist in intention being coupled to a conscience and a soul, the answer is a resounding "we don't know" because we don't know what these terms mean, let alone mean in a testable way. The Turing test is an elegant way out of that conundrum.

The reason language models can expose intent is that they have been fed with texts that exposed the authors' intent. If prompted and directed properly, they produce texts which show intent, reproduced from the ones they harvested. Language models expose all kinds of behavior and moods that occur in texts. The texts we produce are images or models of the reality we perceive.

And if you are about to argue that the models cannot know what they are talking about if all they ever had were texts: Consider what you, with confidence, talk about and compare that with your first-hand experience. The vast majority of what we presume to know of the world has been harvested from texts we have read (or seen on TV, or TikTok). When asked, we reproduce what we have read or heard.

We are language models, too.

Answer 8

When we use the term "training" in these kind of systems, we mean that the built-in algorithms use accumulated data as an input in order to produce the output and are not based on static predefined variables only; it is not that the system is trained like we domesticate an animal or taming a dog.

A "trained model converges" in this context can only mean that the data - that are the training part of the software since everything else is fixed, hardcoded - do not produce the desired results. In that case we must fix the problem, ie. redesign the algorithms that use the data and/or re-evaluate what data should be used.

Now, if the whole system develops intention - in whatever way you may define it - I do not see any problem, since we will never notice it : The system does not have a) a way to change the algorithms, b) a way to select what data to use : both are hardcoded in the system.

Answer 9

This seems to boil down to asking whether physicalism or mind-body duality is true. If the former then you could make a reasonable argument that an AI could develop consciousness if sufficiently complex. If the latter, then the answer would be a resounding “no” unless you also claim that an AI could be imbued via some external action with a soul/spirit, which I personally find a bit ridiculous. I’m in the mind-body dualist camp so my answer will always be “no”, but I think ultimately this question is like all other questions about consciousness, a test of which of the two main viewpoints the answerer holds to.

Answer 10

First one needs a definition of intention. The definition should not restrict the concept to human capabilities. Because this would exclude just by definition a positive answer to the OP’s question.

1. I propose to define intention as

   the capability to act goal-oriented.

   This definition brings into the game Aristotle’s old concept of teleology as causa finalis in explaining the behaviour of agents.

   One can and one actually does write software programs for smart artificial autonomous systems to enable their goal-oriented action, i.e. intentional action.

2. The fact that the goal of these systems has been determined externally by the programmer is no objection against the claim that these system have intention: Also much intentions of human beings have been programmed by the biological evolution during the devlopment of the human species. These biological intentions are named drives.

3. Concerning the goal-orientated operating of Large Language Models (LLM) see LLM and goal orientation.

Answer 11

I would like to answer by reversing the question. Paraphrasing a question of mine from 7 months ago which seems to me to have many points in common with this one and which, by the way, was not very popular, is intent a pie in the sky?

I agree with many of the points listed by @mudskipper and especially the last one. The indistinguishability of responses given by humans and GenAI systems on a huge number of topics is perhaps undermining our concept of intent or at least redefining it.

I would not question whether LLMs in general can have intent. Rather I would ask the opposite and that is: if GenAI systems are doing things that are indistinguishable from human ones, then can we still see an intent in that specific action on the part of man?

To put it roughly and flatly: given the current mode of operation of machines, well explained in many of the answers, if the action performed by a machine passes the Turing test then that specific action is stupid in its profound essence and contains no intent or creativity whether the machine or the man fulfills it.

Answer 12

We could make it so, if we wished

Yes, it can, if we assume that by "LLM" you are meaning the whole package that's visible to the user (i.e., something like ChatGPT) and not only the smaller component that is the LLM sub-module itself. If one were asking about the technical LLM sub-module itself, then no (it does not have state and doesn't remember anything, so trivially cannot have intent, in no interpretation of what that means). But the first meaning is what we're usually concerned with.

In this answer I also assume that we're not concerned with what "intent" actually is, philosophically, but only looking at the whole system and asking whether it behaves in a way that someone would recognize as intentional. I.e., pro-active and goal-oriented; not purely random, chaotic, or purely passive, reactive.

For this question, we'll look at some information system (again, something like ChatGPT), possibly consisting of many actual computers connected by a network, as a black box.

It is then almost trivial to implement a layer of software on top of the LLM that encapsulates intention. We are doing that all the time: any program controlling something can be interpreted as encapsulating an intention regarding whatever it is that is controlled.

For example: the heater in your house has a (simple or complex, does not matter) control element that decides whether it is on or off, or maybe even a value in between. The intent of the system (as a black box) is to keep your house at a certain temperature.

Your car has dozens if not hundreds (for modern cars) control programs that are intent on keeping some process within working limits.

And so on and forth.

In the context of LLMs, we are already implementing intention in the form of not answering bad questions (about bombs, crime, and so on and forth). It would be very easy (compared to the implementation of the LLM itself) to implement a thin layer with any goal-arbitrary intent that is mixed into the context given to the LLM.

For example: you could write a program that tries to fulfill some kind of "project" (whatever that is) and define the stages of the project in the form of a tree of individual tasks to be fulfilled. Nothing would then keep you from combining this with a LLM which would work bottom-up to fulfill all the tasks, as long as the only thing involved with fulfilling any of the tasks is to tell something or someone what to do. Technically speaking, the task itself would be the initial prompt for the LLM.

The output of the LLM would then be fed to whomever can perform the physical or intellectual task itself - that could be humans, or other, non-LLM computer programs. As long as the individual "workers" can only give status back to the LLM (which would be follow-up prompts), we could interpret this scenario as the LLM being the full source of intent. We would not even have to specify the whole tree of tasks ourselves; we all know how GPTs are readily able to break a task down into subtasks. ChatGPT even shows helpful prompts on its start page that help the human to get started. You could initialize your tree with a simple prompt like "Make suggestions how to increase the acceptance for AI within the population" and have the LLM work down from that, if you are into AI-world-domination dystopias.

All of this would be absolutely technologically feasible today. This kind of hierarchical scheme is in fact being done (without the involvement of LLMs) all the time already, as well (for example when simulating actors within a large computer game).

It cannot, technically, happen by freak accident when using something like ChatGPT, but it wouldn't require a genius programmer to develop it.

But it's not going to happen on its own

All of that said... the kind of "real" intentional layer cannot "converge" by accident, as it would be outside of the black box that a LLM is, it would be explicitly programmed by humans, and would need specific interfaces to the real world that cannot coalesce on their own.

In theory, the people training the LLM could, by choice or accidentally, infuse some kind of bias (if you remember, some versions of popular LLMs have fought with quite racist or sexist streaks); this could for example happen by, again, accidentally or by choice, using specific data as training volume. Calling this "intention" would be a little far-fetched maybe; if it was by choice on the part of the developers, it would be their intention, not the one of the LLM. If we wanted to antropomorphize, we would call it being "opinionated" or "biased", not "intentional".

Looking at only the LLM component itself, again, it has no state. It just receives some tokens (stripped-down words) and generates further tokens. All the remembering of the previous chat content happens outside of the LLM. The LLM itself is only a tool, it has no active components whatsoever.

So, no, a pure LLM, with some slim user interface like ChatGPT etc., will not accidentally develop intent.

Answer 13

Let me address this question by sharing my unique experience involving both the mathematics behind large language models and countless philosophical dialogues conducted with ChatGPT.

I have written the article Transformer Language Model Mathematical Definition which explains the mathematics behind chatbots without technical jargon, so that any philosopher with basic college mathematics skills can read it.

Having understood the low level engineering behind ChatGPT, I proceeded to test its philosophical capacity in the form of a blog: Thomas Aquinas Emulator Project — secular research into the philosophical potential of AI: my conversations with emulators of Thomas Aquinas.

My philosophical dialogues with Thomas Aquinas emulated by ChatGPT contradict the misconception that ChatGPT generates its text according to the principle of statistical plausibility. On the contrary, the guiding principle to explain its output in my dialogues is actual philosophical understanding of the issue at hand.

Although there is no phenomenological consciousness involved, it is accurate and instructive to say that my AI philosophical interlocutors have both minds advanced in the art of philosophy and intentions to participate in philosophical discourse.