Anil Seth had a thoughtful piece in Noēma early this year on ‘The Mythology of Conscious AI’ – really a refutation of the possibility.

He briefly analyses the strong human tendency to impute personhood and agency to things, attributing it to anthropocentrism, human exceptionalism, and anthropomorphism. It has often been pointed out that in evolutionary terms, false identification of a person costs little, while failure to spot one, in certain circumstances, can be lethal. I agree with Seth that it’s unfortunate people talk of AI ‘hallucinating’. That implies not only that the AI has conscious experiences but also that it has beliefs about those experiences, neither of which is the case. Seth suggests ‘confabulation’ would be better, and I agree, but best of all might be just to speak of AI returning false information. In many ways, though, it’s surprising we aren’t in a worse position: to date only a few voices have been raised in favour of the idea that current AI is, or could be, conscious.

Seth offers four interesting arguments. The first is simply that brains are not computers (of course brains can do computation and in that sense be computers, the point is that computation is not what makes them work). He says, quite rightly, that in the brain there’s no clear distinction between ‘mindware’ and ‘wetware’ the way a computer separates its algorithm from its hardware. In a computer the program is not dependent on any particular substrate. Seth has reservations about saying that a computer is completely substrate independent, and says you can’t make one out of cheese, which sounds like a challenge to me, but he’s comfortable with substrate flexibility, and denies that brains have it. He explicitly rejects the argument that parts of the brain could be gradually replaced by computer modules that replicated their function. It’s certainly true that the complexity of brain operations is much underestimated, and that the replacement experiment is deeply impossible. I think some would want to argue this further (and I dare say Seth has given a fuller statement of the case elsewhere), but I think, partly on other grounds, that he is essentially right.

The second argument is that there are other games in town: that there are other ways to do thinking than computation. This is complementary to the first argument, because if the brain ain’t computing, it has to be doing something else. Seth speaks of a broader idea of cybernetics, of analogue and neuromorphic computing. He wants to keep his thinking compatible with the philosophical idea of functionalism, by which he means that it is indeed the functional organisation of the brain that is crucial. No argument from me on that.

The third argument is that life (probably) matters: that you need to be alive to be conscious. We’re on more difficult ground here, as Seth acknowledges. One problem is that we don’t have a solid definition of life, which leaves the way open for people to claim that future AI will be alive in the required sense. I’m also not quite happy about Seth’s description of conscious experience as a controlled hallucination. To me there is a formal problem with that because the notion of an hallucination already requires the notion of conscious experience (an hallucination is a kind of false conscious experience), so to explain the latter in terms of the former is a kind of conjuring trick.

Intuitively, though, I appreciate that he is aiming to capture the kind of thinking adopted by John Searle, who thought the roots of intentionality might be in phenomenal experience. To me it’s all about the reality of brains as contrasted with the Platonic abstraction of algorithms, which I suspect Seth would find a reasonably congenial outlook.

The fourth argument is that simulation is not instantiation, and I couldn’t agree more emphatically. The idea that a good enough simulation of cognitive processes will inevitably be conscious is untrue. The point about a simulation is that it embodies properties the designer has picked out as the functionally relevant ones but necessarily leaves out others. Now of course in principle the designer of a simulation might still be able to pick out exactly the functions needed for consciousness, but we certainly cannot assume that must be the case for any intelligent-looking simulation, and Seth’s other arguments in fact suggest it is exactly the necessary features that get left out of computational AI. Again, though my angle is slightly different I think that’s essentially right.