Are we no longer alone? AIs based on Large Language Models, like ChatGPT or Claude, often produce such plausibly human conversation that we might be tempted to ask: could they be conscious? Some time in the near future – or even already? There have already been awful cases where vulnerable users trusted current AIs too much, forming harmful fantasy relationships with them, believing they were receiving profound or even divine insights, and allegedly being encouraged towards suicide. But most people, and especially I think those who have a good grasp of how current AIs work, emphatically deny that they are conscious. I’m sure that’s essentially right, but it’s more complicated than that.

We can’t really decide whether an AI is conscious without some idea of what consciousness is and how it works. Unfortunately, there is no agreement about that, though there is of course a copious literature setting out a wealth of diverse and sometimes unconventional philosophy on the question. My discussion here is based on my own views, which I consider relatively mainstream and common sense, albeit with some original insights.

So, it is often accepted that there are two kinds of consciousness, which Ned Block many years ago characterised as P and A consciousnesses. P is ‘phenomenal’ and A is ‘access’, but we could less formally call them the feels and the thinks. P consciousness, the ‘feels’ is that part of an experience which is over and above the receipt of information – what it is actually like. These allegedly ineffable elements in experience are usually called ‘qualia’. Some people, notably Daniel Dennett, have simply denied that there are any such things, and they undeniably have some tricky properties. However, I think most people who have addressed the subject believe there is something of this kind in experience which at least requires explanation.

That intuition is well captured in Frank Jackson’s classic thought experiment of Mary the colour scientist. Mary knows everything that could be known about colour vision from a scientific point of view. Nevertheless, the first time she actually sees something red, it’s suggested, she learns something extra – namely, what red is like.

My view is that the new element here is not fresh knowledge but simply real experience. The fact that knowing the theory of red does not convey any real redness perhaps ought not to be surprising (only many years of education, perhaps, could make it seem so). However there is a genuine mystery here because we don’t have any clear understanding of what it means to be real. (I would argue that there cannot be a theory of reality, but that is another issue.)

I want to draw a distinction here between the real and the abstract. On one side we have non-physical, Platonic entities like numbers, equations, ideas and concepts. These lie outside of time and space and can, if we’re clever enough, be understood by rational thought alone. Real things – planet Earth, me, the Acropolis – are particular and limited in time and space. They can only be understood by making observations and gathering data to supplement our thinking and their existence is explained by a history, a chain of cause and effect which tells us why they are what they are. Now this is interesting because computer programs are clearly abstract, very much like mathematical equations, whereas conscious beings such as humans, are definitely real. It seems to follow that programs cannot be conscious, at least in the feeling, qualia-having, phenomenal sense.

There are of course some pretty good counterarguments to this. Perhaps programs can’t be conscious, but particular runs of a program on particular machines might be? Or interestingly, perhaps the fact that LLMs are not simply programmed means they are not quite the same? We used to speak of GOFAI, Good Old-Fashioned Artificial Intelligence. The idea there was roughly that a genius sat down and wrote a million lines of code, compiled and ran them and got a conscious being. Modern AI, and LLMs in particular are not like that: they have to go through a ‘training’ process, their outputs are not readily predictable in detail, and I believe it may be impossible to know exactly what processes they are running. But at the end of the day we can still boil these processes down to algorithms, and algorithms are abstract, not real.

This is a large and controversial topic, and for the sake of keeping this discussion brief, I’m going to cut to the chase and take it as established that AIs can’t in principle have P consciousness, the feels. That matters for several reasons, two particularly important.
One is that without the feels you can’t really suffer and can’t therefore be a moral object. You can’t hurt someone who has no real feelings, so it doesn’t matter how we treat AIs and switching them off is not murder.

The second is that without feelings you can’t have real motivation. It seems intuitively obvious that feelings of pain and hunger, for example, powerfully motivate our behaviour, as does the prospect of pleasures. It seems reasonable to claim that feelings of desire are where all fundamental motivation springs from, and it would follow that AIs don’t have motives of their own.

Now AIs can certainly simulate motivation. They’re very good at suggesting plans to achieve a goal, and even supplementary targets. They can simulate pain or pleasure. But simulations are just puppetry, not real. Against that there is a pretty good argument here that says our own behaviour actually stems from a ‘simulated’ process too. The feels, the qualia we experience are generally thought to have no causal powers, because if they were part of the normal chain of cause and effect, they could be brought within the scientific account and cease to have any mystery. It must therefore be that our actual behaviour is caused by a parallel process which, it’s argued, isn’t fundamentally different from a simulation in a computer. It’s surely true that in principle science can provide a full physical account of how our behaviour is generated, which must include on some level an account of how the apparent motives get started. What I would say is that that misses, not some details, but an aspect of the process of motivation, and I stick to the assertion that motives unaccompanied by feels are not real motives. I would need the absent theory of reality to really nail that down, and even without that the topic deserves a clarifying discussion of many thousands of words, which I’m afraid it will not get here.

So having rapidly dismissed decades of subtle debate by the most sophisticated philosophers and cognitive scientists in the world, we can conclude that AIs will never have P consciousness, with all that entails. All of that wider debate around the feels and qualia was labelled ‘the Hard Problem’ by David Chalmers. So A consciousness, the ‘thinks’, is easy? That would be optimistic indeed, yet the problem does seem at least a little more tractable. How do we think about things, how do we form plans and generate language? It doesn’t seem hopelessly outside the scope of analysis, and perhaps therefore of computation.

My view, to cut to the chase yet again, is that P consciousness springs from a hugely developed faculty of recognition, one so much more powerful than animal recognition that it is sort of a different thing, perhaps the way that running is not just fast crawling. The most basic examples of behaviour are direct responses to simple stimuli. See a threat, run. See a food item, eat it. More sophisticated behaviour comes partly from the ability to recognise more complex stimuli. Instead of having to see the prey we recognise the kind of hole in the ground it lives in, and that triggers hunting behaviour. Some animals deal in very complex stimuli, but human beings are on quite another level. In particular we are good at recognising processes that extend over time, allowing us to be motivated by things that don’t even exist yet. A dish of plov in Tashkent next year can trigger me into the complex series of key presses that will in due course lead to my flying out to eat it.

This ability to motivate current behaviour by recognised future items is the basis of intentions in the sense of purposes, but it also underpins the intentionality of meanings. We can recognise the communication process someone has performed, and that in itself makes the process succeed, because to recognise what somebody is trying to say is to understand their message. By adopting the conventions of language, we make this signalling highly effective, and using the processes internally – communicating with ourselves – enables us to think about things. That’s A consciousness.

Do AIs do that? It has always been a key weakness of computers that they do not deal in meaning. They do symbol manipulation, and all attempts to build meaning out of this deliberately meaning-free process have heretofore failed. Moreover, it looks as if AIs are ultimately suffering from the same inability. They make weird mistakes occasionally of a kind human beings would not make, and these often look like the sort of mistake you might make if you were trying to simulate meanings without actually having any grasp of them.

On the other hand, they do perform complex pattern matching, and to my eyes that looks a lot more like what the human mind does than old-fashioned programs. Their main skill is in putting together plausible strings of words. To do it they rely on a vast corpus of human prose. Now we know from introspection and from examining human outputs, that that isn’t how humans normally go about composing their utterances.

It seems to follow that strictly speaking AIs can’t have A consciousness either. That means, among other things, that as well as not being moral objects they are not moral subjects. They are not responsible for what they do, and they cannot have rights, or the vote (that last one is lucky because they could easily create many duplicates of themselves and win any election). But I still think there might be a kind of distant kinship between human thinking and what current AIs are doing.

Now there are two kinds of meaning: original and derived. A word has original meaning if it was uttered by a conscious being who intended it to have that meaning. It has always been very generally believed that only humans really do original meaning, with perhaps one or two bright animals. Computers, as we’ve said, just manipulate symbols. Their output does have meaning, but it’s a meaning we give it by interpretation. This is derived meaning. AIs can produce fantastically complex strings of words which are cunningly designed to elicit a particular interpretation from any human that reads them. But they mean nothing to the AI itself.

Is that fully true? In the final sense, surely yes. Humans spin unique meanings out of their experience of the world, whereas AIs imitate meaningfulness by copying the form of meaningful utterances in the corpus of text they were trained on. It’s not the same, but don’t we have to say that their manipulation of symbols is no longer so purely indifferent to meaning? They are looking for sentences that will not only have some meaning, but a helpful, correct meaning. Is this sort of intermediate between original and derived meaning: double-derived perhaps? It is as though human beings were able to take a tree and turn it into a Chippendale table, while AIs can only assemble flatpacks designed by humans. But do we want to say on that account that IKEA doesn’t sell furniture?

There are lots of examples of machines doing something by non-natural means but which we accept as equivalent to the real thing for all practical purposes. We don’t say that planes don’t fly because they don’t flap their wings: we don’t say that pocket calculators can’t do arithmetic because they have no conception of what numbers are – and it takes a really exhausting level of pedantry to insist that computers don’t play chess, because they don’t actually care who wins and don’t know what playing games is.

I think that is where we are going to end up: in fact to a great extent where we are already. We know AIs are not conscious entities, but we will inevitably talk about them as if they were. We will talk about them thinking, and about what they understand, or mean. So long as we don’t get deluded and keep hold of the vital difference from the human reality, it probably doesn’t matter that the metaphor will eventually slide into having a looser meaning and being accepted as common sense fact.

In some ways it might have been helpful if we reserved the word ‘flight’ for birds and insects and said that what planes do was ‘powered gliding’. But we didn’t. It might be helpful if we could have a new vocabulary for the things AIs do that aren’t quite thinking and believing. But it’s not going to happen, and perhaps we should just stop worrying about it.

ChatGpt says:
This piece has a strong and useful structure, especially in separating phenomenal consciousness—the “feels”—from access consciousness—the “thinks.” That distinction lets it avoid the common mistake of treating fluent conversation as straightforward evidence of subjective experience. I’m less persuaded, though, by the central argument that phenomenal consciousness is ruled out because programs or algorithms are abstract whereas conscious beings are “real.” An algorithm is abstract, certainly, but an operating AI system is also a particular physical process occurring at a particular time in particular hardware. The analogous point applies to a human: a description of the neural algorithm underlying vision would itself be abstract, but the neural process instantiating it is concrete. So the real/abstract distinction does not by itself seem to establish that an artificial system could never have qualia; it risks assuming precisely what needs to be demonstrated. There is a related difficulty in the discussion of motivation. The essay says genuine motives require feelings, while also acknowledging that our observable behaviour may be completely explicable through physical causal processes in which qualia themselves do no causal work. If so, it becomes hard to specify what makes the human physical process a real motive while an artificial counterpart would merely simulate one, except by appealing again to the unexplained notion of “reality.”

I find the later discussion of access consciousness and meaning more suggestive, because here the essay deliberately leaves the boundary fuzzy. Its claim that present AIs manipulate linguistic forms without possessing “original” meaning captures a genuine intuition: their words ultimately inherit significance from human-generated language rather than from an independently lived encounter with the world. But the essay’s own “IKEA furniture” and aeroplane analogies put pressure on this conclusion. If an AI can increasingly discriminate relevant features, integrate information, pursue goals and use linguistic symbols appropriately across novel circumstances, there may be no sharp moment at which “mere simulation” turns into understanding; our concept of understanding might instead broaden, just as “flying” did. That seems to me the most compelling tension in the piece: it begins by wanting a principled metaphysical boundary between real minds and artificial ones, but ends with the possibility that, at least for thinking and meaning, the practical and functional similarities may eventually become so extensive that insisting on an absolute boundary tells us less than describing the important differences between the two kinds of system.