Your Undivided Attention - How to Think About AI Consciousness With Anil Seth

Episode Date: July 4, 2024

Will AI ever start to think by itself? If it did, how would we know, and what would it mean?In this episode, Dr. Anil Seth and Aza discuss the science, ethics, and incentives of artificial consciousness. Seth is Professor of Cognitive and Computational Neuroscience at the University of Sussex and the author of Being You: A New Science of Consciousness.Your Undivided Attention is produced by the Center for Humane Technology. Follow us on Twitter: @HumaneTech_RECOMMENDED MEDIAFrankenstein by Mary ShelleyA free, plain text version of the Shelley’s classic of gothic literature.OpenAI’s GPT4o DemoA video from OpenAI demonstrating GPT4o’s remarkable ability to mimic human sentience.You Can Have the Blue Pill or the Red Pill, and We’re Out of Blue PillsThe NYT op-ed from last year by Tristan, Aza, and Yuval Noah Harari outlining the AI dilemma. What It’s Like to Be a BatThomas Nagel’s essay on the nature of consciousness.Are You Living in a Computer Simulation?Philosopher Nick Bostrom’s essay on the simulation hypothesis.Anthropic’s Golden Gate ClaudeA blog post about Anthropic’s recent discovery of millions of distinct concepts within their LLM, a major development in the field of AI interpretability.RECOMMENDED YUA EPISODESEsther Perel on Artificial IntimacyTalking With Animals... Using AISynthetic Humanity: AI & What’s At Stake

Hey everyone, it's Aza. Welcome to your undivided attention. When we go out and talk to people about the AI dilemma and the risks we're going to see from AI, there's one question we always very predictably get asked, and that is, will AI become conscious? Will AI wake up and then kill us all? We're not alone in fielding these kinds of questions. People who think about AI for a living get questions about machine consciousness all the time. And it takes up an outsized space in the public discourse.

This question of AI becoming conscious exploded into the news the summer of 2022 when a Google engineer named Blake Lemoyne publicly claimed that Google's large language model was sentient. Google has a policy against creating sentient AI. And in fact, when I informed them that I think they had created sentient AI, Is it, no, that's not possible. We have a policy against that. Le Moyne stuck to his claims and was ultimately fired from Google. Most experts agreed he was wrong.

In my opinion, though, is AI conscious? That's the wrong question to ask. After all, AI doesn't have to wake up to cause massive damage. AI doesn't need to be conscious to overwhelm our democracies or build a bioweapon. Instead, the right question is, will AI persuade people that it is conscious, and will people then form relationships so strong with AI that they're willing to make dangerous sacrifices for them? In the years since, more prominent figures in AI have argued that AI actually is on the path to consciousness, though, and companies

like Open AI have taken great efforts to make their chatbots feel like a conscious, sentient human. In the end, perhaps the question of, is AI conscious, is so endlessly fascinating because of our own self-obsession. Each one of us has that spark of awareness, and fundamentally we want to know, are they like me? And so that's why we're so excited to have Anil Seth on the show to help us explore these questions. Anil is the professor of cognitive and computational neuroscience at the University of Sussex, an expert on the science of consciousness, and the author of Being You, a new science of consciousness, a book I remember laying in bed and reading front to back, cover to cover. So Anil, welcome to your undivided attention. Thank you, Aza. It's a real pleasure

to be here. The question of whether AI is conscious as not has been a nonstop debate for the longest time. That question that I'm sure you always get that we always get is, you know, conscious is going to wake up and is going to kill us. Isn't that the thing to worry about? So why do we get asked this question? Why is the topic of consciousness almost like irresistible catnip for our minds? Well, I'm not surprised it's catnip. I mean, it's been catnip for my mind for a few couple of decades now. I've always been interested in this question of consciousness. and I think most of us are. But you're absolutely right that it's come to the forefront

in the discourse about AI and perhaps, I think, very likely to an outsized degree. And there are many reasons for that. I think one of them is just science fiction. I mean, science fiction has given us this whole history of stories about machines that are not only intelligent but also aware. And of course this goes back a long time in culture as well.

way before people even had the first glimmerings of artificial intelligence there's always been this tendency for human beings to try to create things in their own image and to project human-like qualities into the things they create or imagine I mean in literature this goes back Frankenstein I remember reading the myth of the Jewish golem Yossal or Yosef which was created out of mud from the banks of a river and under a magical incantation became

conscious and started rampaging around and doing all kinds of crazy things. So I'm not surprised people are interested in it, but this idea, this very quick association that is AI going to wake up and kill us all, I think that's just a very misconceived way of phrasing the worry, phrasing the question that owes more to these kinds of dystopian terminator style science fiction scenarios than to what the actual technology, the actual science, and the actual philosophy has to say about these issues. maybe we should dive in a little bit into that like what are the right questions to be asking if the wrong question is like will it wake up and kill us then what are the deeper philosophical questions that should begin this conversation i like to think of it in terms of a few related questions there aren't good answers to any of them i think the point is to lay out the questions and and see whether they help shed light on

the issues just in virtue of the way they are. So the first question is, why would we even think that AI is conscious? We are building systems that are supposed to do things, that are supposed to have capabilities of particular sorts, intelligent capabilities we call them. So why would I even think that this would result in something that is also conscious? Consciousness and intelligence are different things.

I mean, we think we're intelligent and we know we're conscious. That is we think we're smart, and we also experience. the world and the self. We have conscious feelings. But they are different things. And then the second question, I think, is what would it take for an AI system to actually have consciousness to our best guess because there is no consensus out there scientifically or philosophically about the sufficient conditions for consciousness to arise in a system? We don't have a consensus theory. And of course, this isn't a question just for AI. We face this question in many, many really

really pressing areas of society and technology now with non-human animals, with newborn human infants, or people at the end of life, the humans after brain damage, and emerging neurotechnologies, things like brain organoids. I think this question applies differently to all of them. And then the last question is, what should we do? What are the ethical and moral concerns about AI systems that either are conscious? That is, they have experiences. There is something it is like to be an AI system. Or I think more likely, because we're almost already there, at least for some people, AI systems that give us the convincing and perhaps irresistible impression of being conscious, whatever is going on under the hood.

I mean, that's a situation we're running headlong into, if not already there. And that poses its own ethical and moral concerns. That's an incredible table of conflict. That's an incredible table of And maybe a good place to start is just sort of where you're ending, which is without the need for AI systems to be conscious. And I want to get us to definitions of intelligence and attempted definitions of consciousness and things like that. But even before we get there, you know, I look at a cloud and I see a face. I look at a car grill and I see a face. Like the human mind projects intimacy and emotions into almost everything. You know, I grew up in the era of Tomogacchis, Furbys, Nintendo dogs.

Like, we love to project ourselves, as you said, onto others, make the world, remake the world in our image. And as you also said, humanity has been writing stories about machines becoming alive and having consciousness forever. And because what AI does is it mimics the patterns and the styles of how human beings. Right. It's actually very good at persuading us that is becoming conscious. So what are the implications of that as AI chatbots, things we don't even know are chatbots, become more and more able to convince us of the stories we've been telling ourselves all along? Yeah, so let's set aside for now the question of whether they actually are conscious or what leads us to think they are. I think you're right that we project these qualities into them when they're not necessarily there.

just as our brain works in this way all the time. It's not just things like animacy and consciousness and mind, but colors into the world and time. But at least that's how I think about how the brain works. It's always engaged in this projecting the things to give them a sense that they have an essence, which is really always a collaboration between our minds and our brains and what's going on in the world and the body.

But anyway, what are the consequences? Well, if we have a language model, especially language, because as you say, language is very, very key to our image of what it is to be human. Things that speak to us seduce our intuitions in an extremely powerful way. And so this is why we've seen people much more willing to claim that language models are conscious than other forms of AI. And so the first thing is we shouldn't be sanguine. Even those of us who are predisposed to believe that language models are not conscious, we may get to the point that we're unable to resist feeling that they are.

And I think this is a really important point. Illusions of consciousness in language models could be cognitively impenetrable in the same way that some visual illusions are. So there are some visual illusions that even when you know what's going on, you still can't help seeing them. And this might be a case like that. So even if we believe that they're not conscious,

we might still feel that they are. So we can't be complacent and say, oh, you know, just when we understand how things work better, that'll change how we intuitively respond to them. It really might not. And my suspicion is that it will not. So what is the problem with that?

Well, I think there's a number of problems. One of them is we become potentially much more psychologically vulnerable. If we think that the thing we're interacting with really understands us, in the sense of having a conscious experience of understanding or empathy if it says something that seems to empathize with us, then, you know, we may open up to it in ways that we would otherwise not do.

We may also take its advice more seriously than we might otherwise do. And there have already been some tragic cases like this. There was, I think, in 2003, in Belgium, the newspapers reported a case of a man who took his own life after interacting with a chatbot that encouraged him to do so or certainly discuss the topic with him and of course we don't know exactly what went on there

but it's very plausible that it played some role so I think psychological vulnerability is one important issue abrogation of digital privacy goes along with that we may divulge aspects of our lives that we wouldn't otherwise do there's also a much more subtle issue, which is something I worry about quite a lot, which comes down to the ethics of treatment

of these systems. So if they're not conscious yet we feel that they are, we have a choice. On the one hand, we either treat them as if they are conscious because we feel that they are, even though we believe that they're not. Or we treat them as if they're not conscious, even though we still feel that they are. And there's no good outcome here. In the former case, if we treat them as if they are conscious, even though we think they're not, I think plausibly we end up caring about hunks of silicon and code, which have nothing going on, and caring proportionately less about other things that really do warrant our moral consideration, other human beings and other non-human animals and so on. So that's a problem. But the other option is also bad. If we do not care about things, even though we feel

they are conscious, we end up brutalizing our own minds. This is an argument that goes back to Emmanuel Kantan's lectures on ethics. It's why we don't rip up dolls in front of children, even though they're just made of plastic. It's psychologically very bad for us to do that because of the way it makes us feel, it changes the way we would interact with other conscious creatures. So these are the questions, and they're pressing because how we design language models will really make a difference to how these issues play out.

If we just keep designing them so that they're maximally human-like, especially when we start coupling them with generative images and generative video, generative sound, then we're going to be in real trouble, I think. Yeah, it erode some fundamental part of what it is to be a good human being. We wrote an op-ed piece with Yuval Harari, I guess now, almost two years ago. And in it, one of the points that we made is that language is the operating system of humanity, of civilization. When AI starts to hack language, it hacks all of us.

It's cognitively impenetrable. And if we form intimate relationships with things that we cannot help but feel our conscious living, breathing, sentient beings, even if we know that they aren't, that then sort of at the very deepest level hacks us. It does. I mean, this gets back to this point that language is exceptionally powerful at seducing our intuition.

Of course it's useful. I mean, there's a line to walk here. Language models are really powerful, really impressive. It's great to be able to engage with things in a very natural, fluent way. But it is not without its trade-offs, and this is one of the big trade-offs. I guess this gets into the definition of consciousness, maybe intelligence, because it appears like we're not going to be able to answer the question of whether something is conscious by looking at its behavior.

Like a language model may perfectly replicate the way that a conscious system might answer. So how do we tell looking from the outside what kind of mechanisms or architectures by which we could even understand if something is conscious? And that gets us to the definition, maybe starting with the definition in your book. So consciousness is everyone says it's really. really hard to define. And of course you could spend hours, careers, talking about definitions. But in a simple, straightforward sense, we all know what consciousness is. It's what goes away when you go under general anesthesia or fall into a dreamless sleep. And it's what comes back

when you wake up or start dreaming or come around from anesthesia. The philosopher Thomas Nagel, I think, has a very useful working definition in which he says, for a conscious organism, there is something it is like to be that organism there is something it is like to be that organism now by that what he means or what most people take him to mean is that for some things it feels like something to be that thing it feels like something to be me it feels like something to be you it feels like something to be a kangaroo

or a bat as in Thomas Niggles famous paper what it's like to be about but it doesn't feel like anything to be a table or a chair. And so the question here is, does it feel like anything to be a language model? Is there something it is like for the language model itself to be a language model? The reason I like this definition is because it's very, very basic and it doesn't conflate consciousness with all sorts of other things that might contingently come together with consciousness in us humans, but which are not necessary for consciousness in general. So in this definition, there's no necessary relationship with intelligence or with an explicit sense of self,

you know, interpersonal identity or agency, or anything. It's just the presence of any kind of experience whatsoever. When we open our eyes, look out through the window, as I'm doing now, my brain does more than just respond to it as a physical system. There's an experience of blueness in the sky, whiteness on the wall. There's something that it's like to have that experience. And that's different from intelligence, which, again, there's many definitions, perhaps no consensus, and you'll know this better than I,

but broad definitions are things like doing the right thing at the right time or solving complex problems flexibly or various things like that, but they're all defined in terms of the functional capabilities of a system. They're not defined in terms of whether there's anything it's like to be that system. Now, one of the most famous tests in the space of the most famous test, by a country mile in AI, is, of course, the Turing test. And the Turing test is often rolled out as a test of all sorts of things. Is it intelligent, but also it's been used as a test or interpreted as a test of consciousness too.

But it isn't. It's a test, or is supposedly a test of machine intelligence. Interestingly, it's also been argued that the Turing test, is really a test of what it would take to convince a human than a machine is intelligent. It's rather a test of human gullibility or what our internal criteria are rather than the objective test of whether the system actually is conscious. And in this scheme, a Turing test is a reverse test of intelligence, is a test of what it would take for a human being to attribute intelligence. But what we want is a forward consciousness test.

So I think for me, this way of putting things makes it very clear that we can't use things like Turing tests as tests of whether a system is conscious. We need other ways of doing it. And just clarifying it like that, I think, is already potentially quite helpful. Now, the question then comes out of it, well, what would be a forward consciousness test?

Well, we don't have one because nobody knows what makes anything conscious. Now, for forward tests of consciousness, we just have to do the best we can. And that means we try and condition out the biases that we bring to any situation. So we think, okay, I'm not going to rely on just linguistic fluency because that's more reflective of my anthropomorphic and anthropocentric biases. The things that I think as a human are super important.

So we'll try and not rely on those. And ask, well, what properties are shared by the system that we think, are important in consciousness in general, whether it's in humans or in other animals, things like that. I think that's the best strategy. And if we go that way, that strategy will cash out differently, let's say, for an AI system

than for a bumblebee or a parrot or a human infant. Because what we share and what we don't share are very different in each case. With a non-human animal, we share an evolutionary history, we share a brain made out of the same kind of stuff. We share many things that seem to be plausibly important in consciousness.

With a language model, we don't share any of these things. So I think language models are more similar to us in ways that will turn out to not matter and less similar to us in ways that turn out to matter when it comes to consciousness. So I guess one of the questions we should start asking is this question of, like, how do we know? The way I model AI is that it is very good at writing in the style of. That is to say, it doesn't have empathy.

It's just very good at writing in the style or producing text in the style of empathy. It doesn't have chain of thought reasoning. It just can produce text in the style of chain of thought reasoning. It doesn't actually have theory of mind. it just produces text in the style of theory of mind and I mean well this opens up a very interesting philosophical question which is if you can perfectly emulate the style of it does in the limit style become substance

and how do we tell the difference between imitation and the thing itself? Yeah I mean there's very deep issues here in philosophy that go many issues come together so one is simulation of something the same as real of that thing. If we have a system that is able to perfectly simulate what a conscious human would say

or even simulate at a more detailed level, just simulate the brain mechanisms involved, simulate every neuron involved in how we do what we do. Would that be the same as actually giving rise to it? And this is one of the central questions, I think, that is at issue and is perhaps not given the promise, that it ought to in contrast to many things in this area now,

this distinction between simulation and realization. Because there are some things for which it doesn't matter. So if we have a computer that plays chess, it plays chess. But there are other things for which it's clearly an important difference. I mean, this is the issue or one of the issues with Nick Bostrom's simulation hypothesis. Serious philosophical attention has been given to the idea that we actually do live in a computer simulation already. one of an uncountable number.

And interestingly, when Bostrom talks about the simulation hypothesis and simulation argument, related but different things, he'll go through the sort of statistical arguments why he thinks it's at least a plausible thing to consider. And then also says, oh, by the way, the other thing we've got to assume is that consciousness is a matter of computation, just getting the right computations, as if that's a consensus view

and it is not or it should not be a consensus view it's one option among many options about how consciousness fits into the universe so I found in discussions with people in AI and machine learning and to be fair it's also the probably the most common view in neuroscience and philosophy as well that yeah computation is sufficient for consciousness

this I think stems from the fact that the computer as a metaphor for the brain has been extremely powerful over decades. But metaphors in the end are metaphors, and we often go wrong if we confuse the metaphor for the thing itself, if we reify the metaphor, take the map for the territory. It could be, and it's a very compelling, at least for me, alternative. And it's only one of many alternatives, which is that consciousness is a property of living systems and living systems only. In the same way that metabolism is a property of living systems. It's definitional in some sense.

You can simulate metabolism, but that's just a simulation. This is called biological naturalism. It goes back to John's cell. And I think there are some good reasons why this might be the case. I certainly don't think you can just dismiss it and say, no, we just assume that consciousness is a form of computation. And that, by the way, I mean, that is, That's the assumption on which any discussion of AI actually being conscious is grounded.

If that assumption doesn't hold, then we're always in the space of systems that at best can give us the irresistible impression of being conscious without themselves actually being conscious. Just a couple of threads to pick up here, you're talking about metaphor hazards, which is to say whatever is the latest technology, beings always then use that as a metaphor for the mind when we invent clockwork the mind works like clockwork when we invent steam power the mind works like steam power when we invent computers works like a computer when we invent the internet the mind is now a network and you know there's the risk is yes the map is not the territory but even worse than that the map can terraform the territory based on like what we believe about our maps we end up changing the world and you know i also work, help run this nonprofit Earth species project, which translates animal language.

And ironically, the goal there is by demonstrating rich, symbolic, interior-aware communication, you sort of have to assume some kind of sentience that you get a window into perhaps what is it like to be that being. And as a designer, I think I've always been obsessed with the question of what is it like to be another. person and that just expanded as we can expand our perception and hence our understanding with AI to like what would it like to be another being? I mean I always wonder what would like to be a tree like a sequoia where like every in and outbreath equivalent is like a decade. There are some of these trees like bristlecline pines that around when the ancient Egyptian pyramids like rose and fell like what would that experience of being be like? And I, I think

think in some sense goes back to the question of is consciousness a property of you know information processing or matter if the only way they're going to be able to tell if a system is consciousness by having some kind of forward tests some correlate from being able to peer you know at the human brain from outside the human brain see what's different when it's undergoing general anesthesia when it's not when it reports having a conscious experience versus when it doesn't report having a conscious experience. And I know you have some great experiments in your book of how you might be able to tease those things apart. But doesn't this always end up with, we'll never be able to fully cross that first third person divide. We'll never really know whether something has a conscious experience.

I don't really know that you, Anil, is having conscious experience. I can just make good suppositions because we share the same hardware, the same wetware, the same sensory history. And so we can make a good assumption, but we're always going to be looking for outside sort of like correlates, things that sit next to, but don't actually get us all the way there. I partly agree with you. I think there's a couple of places where I think it's less pessimistic than you say. So one thing is, I think you're right that we can never cross, entirely cross this divide between the first person and the third person. So I will never know what it's really like to be anything other than who I am now.

I don't even know what it's going to be like to be me tomorrow morning or you or any other human being. But I'm totally convinced that you are conscious. I'm totally convinced that tomorrow morning I'll be conscious too. So that limitation, I don't think, is a relevant limitation, right? I am totally sure that you're conscious.

Now, take a slightly more tricky example, someone who's suffered brain injury. If they're able to speak, even if it's through imagining things so that it sets off there's a beautiful series of experiments pioneered by my colleague and friend Adrian Owen where he got people in a brain scanner

who could not behave. They couldn't say anything, they couldn't even move. So they looked from the outside as if they're unconscious, right? But if you ask them questions or give them instructions, imagine walking around your house or imagine playing tennis, different parts of the brain light up

in the same way that would happen for me or you. This seems to be extremely good evidence in this case that these people are conscious because it's very, very difficult to imagine a human being doing that without being conscious, understanding the instructions voluntarily following them for extended periods of time.

It makes sense in that context, so we can be pretty sure in that context. It's very easy to get a language model to say anything, but that doesn't mean what it says relates to what it is in the same way, because it's a language model. So the shared background is not there. So I think the challenge we face is, it's really the challenge of consciousness science. It's going to be very difficult to figure out what the sufficient conditions for consciousness are

until we have a better understanding of how consciousness happens in those places where we can be reasonably certain that it exists. We learn bit by a bit. We gradually extrapolate. These days, most people are pretty convinced that all mammals have at least some consciousness. experience. Not everybody, but most people. That allows us to generalize beyond the human, get a sense of what's going on in brains that's relevant to consciousness that's not specific to humans, gets us away from language, for instance. And as we do more of this, I think we'll be

able to make better inferences about the presence of consciousness where we're less sure on our footing. I'm getting from you, Anil, that you don't think that, machines will become conscious, although there is plenty of room for disagreement. So what are the consequences of getting it wrong? I'm glad you raised that, because you're right. From my own perspective of consciousness being very closely tied to life, I think we're not on a path to conscious AI. I think we're on a path to AI that seems conscious, but that's a very different thing.

But, of course, I might be wrong and probably am wrong. I think most people who have theories of consciousness are also probably wrong. there's a real need for a certain humility here. I always think if you hear a confident pronouncement from somebody that AI could never be conscious or AI is conscious, then I think we should be pretty skeptical. There are no grounds for extreme confidence either way here.

The consequences of being wrong about the fact of the matter about AI actually being conscious are huge, is another reason we need to respect this humility. If AI is on a path to being conscious or already slightly conscious, as Ilyos Setskeva puts it, then we face a moral and ethical catastrophe of kind of unprecedented proportion, and that sounds very dramatic, but I think it's warranted. As soon as something is conscious, it has moral considerability. It has its own interests. It plausibly has the potential to experience suffering and it may be suffering of the kind that we won't or constitutionally unable to recognize because of the very different

constitution of these systems. If we artificially bring new forms of suffering into existence through developing real artificial consciousness, all that is with capital letters a very bad thing indeed. So I think it's a really ethically crucial but epistemologically which is to say how will we know highly uncertain situation. And what do you take out of this mess? I think the first thing you take out of it is that it should not be a goal to build conscious AI. Nobody should be

trying to build AI that actually is conscious. That is ethically, morally a highly irresponsible thing to be doing. Now, of course, some people would say, well, any AI system, you're saying we shouldn't do any research in cognitive neuroscience or AI? No, no, no, no. I think there's again a line to walk. I think it's very reasonable that we can build systems that are useful, that maybe leverage some of the functions that we associate with consciousness. This is a conversation I had a number of times with Yoshio and others. We were talking about this in terms of where AI should go. And this idea came up that there are things that we do

that we associate with consciousness that are very useful that AI systems don't do or don't do very well yet. Things like learning from one shot or very small amounts of data, generalizing out of distribution to novel situations. And having insight into their own accuracy, we intuitively have a degree of confidence about things we're conscious of. These are all very useful things, and so it makes sense to think, can we abstract the principles by which brains do these

and build them into artificial systems? I think that's a reasonable thing to do. It's walking the line. What I think is unreasonable to do is have an explicit goal to build a machine that has conscious experience. We don't know what it would take to do that, but critically, we also don't know how to guarantee not doing that. we don't know what it wouldn't take.

I can see the strong argument in here for the best way to understand our own consciousness is going to be to make sort of surrogate models AI that we can turn on and off its consciousness to see how its behavior or other systems might change. And so the best way to understand what it'll take to not build conscious AI is to try to figure out what it would take to turn it on and off. I'm curious how you react to that. It's very similar to like the greatest accelerant to AI risk has been people who've been focused on AI safety trying to race there to get there first. Yes. No, it's a dangerous line of thinking. And yeah, it's also there in, let's say,

animal research in neuroscience too. Okay, so how do we best understand how to avoid suffering in human beings and other animals? We need to understand the neural mechanism, so we need to do experiments, there's always a line to tread here. So personally, because I think that consciousness is not just a matter of computation, personally, I'm quite comfortable with computer simulations and models of the brain as ways of understanding the brain mechanisms involved in consciousness. In just the same way that I'd be comfortable with computer simulations of protein folding or weather systems as ways of understanding these things. I think the chances of actually these things being conscious is so low that I'm happy with doing that.

But if I thought that building a model of, let's say, a global workspace or whatever your theory of consciousness preferred might be, would actually generate a little bit of consciousness, I would be much less happy about doing that. So it does depend on what your starting assumptions are, which is another reason I think we should be very curious about the assumptions we make about what it would take for something to be conscious.

I'm just noticing it is like almost a contradiction inside of myself, which is I really want the answer to what it means for a system to be conscious when it comes to animals, for us to treat them with a greater respect, dignity, to displace humanity as like the center of the world. And yet when it comes to machine consciousness, I want the answer to go the other way. I sort of wanted to be like, well, that doesn't really count somehow. It's a different substrate. Therefore, we don't have a moral obligation to something that I can't pet and touch or hold or feel like a kind of empathy in its physical form. And I guess it gets to this next question, which is like, all right, what steps do we need to take to orient society around the future we're actually already starting to live in? this is a very difficult question of course which is why we're talking about it right and I don't have all the answers at all

I do feel strongly that number one thing already mentioned we should make it morally unacceptable to have as a goal to build conscious AI real conscious AI I mean that's just not a good idea hard to know what that means in practice again for the reasons we've just discussed but at least if we have that as a north star then as we learn more about consciousness and more about AI. We should revisit this constraint. Then there are a whole set of other questions,

which I think come down to how do we learn more. And here, I think, is where we do need to walk the line. So the more we do know about consciousness in those places where we're sure it exists or reasonably sure, the more informed our inferences will be in other cases. So the real urgency is to accelerate research into consciousness. So this is a bit like your acceleration example.

But I do think that consciousness, we are learning a lot more about it. And there are ways to do this, which I don't think are particularly morally problematic. They just require resources and scientifically upping the game in all sorts of ways. But that's perfectly possible. So I think that's something that we should do more of, not only for AI's sake, but also when it comes to animal welfare, and your work with in the Earth Species Projects. A beautiful example of, again, exactly how do we use knowledge to finesse to change our behavior

with the systems, creatures we share our world with? But also, you know, human infants and organics. So many cases where it's really important to understand, even in the law, it's important to understand more about consciousness, to understand when we hold people responsible for their actions. It's not an armchair philosophical endeavor. Consciousness science is possibly one of the most practically urgent things. things that we could be doing. And then I think the last thing, back to AI, is systems that

seem conscious, because this is happening. And here we need to decide, well, what kinds of systems do we want? Do we really want to just design AI systems that maximally seduce our intuitions are as convincing as they possibly could be? I don't think so. I see some advantages, and some of them may be slightly uncomfortable. The idea that we could, best, benefit from therapy talking to a language model that actually doesn't care about us at all because it can't care about anything is disquieting. But if there's evidence that it's useful and there is some, then it's hard to dismiss it as a bad idea. But it doesn't mean that this race to the bottom, wherever you want to go, race to complete incorrigibility of systems

appearing conscious is a good thing. One of my mentors who sadly died recently, Daniel, Dennett, I think he said something, well, he said many very wise things, but one of the things he said that stuck with me was that we should always remember that AI systems are tools and not colleagues. We should not kid ourselves otherwise. And I think we should design them with that in mind too. They design systems that best complement us as humans and that don't mimic or try to replace us as humans. Yeah, I think that's a really great point, but I also think we need to be really careful because we can't forget that while humans wield tools, incentives wield humans.

So when people hear that AI is just a tool, they sometimes hear that it's just up to us to decide how to use them, but that's not really true. It's the incentives that shape how the tools end up being used. And thinking back to the releases of OpenAIs GPD-40, it's really, designed to feel eerily human. It jokes and it laughs, it sings, it flirts. There's a clear race, a scramble for the appearance of artificial consciousness. Yeah, there is. There's an appeal to it. And I think this does go back to this 100,000 year history of humans wanting to create things in their own image. But that history of literature, if it's taught us anything, it's taught us

the things that can go wrong if we buy into that goal without thinking critically about it. I recently read Frankenstein, Mary Shelley's novel. The thing about Frankenstein was that when I read it now, thinking about things we've been talking about now, instead of it being a cautionary tale against the hubris of creating life, which I think is often how it's interpreted, you know, the lightning bowl and I'm alive and in the dark tower and midnight. It's not that the creature was alive that mattered.

It was that it was conscious. It was the ability of Frankenstein's creature to feel, to experience sadness, rejection, envy that led to all the carnage that followed. So reading it now, I think it does what great literature often does, which is it's always relevant. And it seems now to be a cautionary tale against the hubris of the desire to create artificial consciousness

and we should avoid making that mistake whether in reality or even in appearance I just want to come back because it's because I know how the human mind works is that intelligence and consciousness the ability to have such a thing as the feeling of being are different. And what the labs are doing right now is they are racing towards the ability to automate AI research. That is the ability to sort of spit out an AI that acts like a intelligent human agent that can program at the level of an AI researcher to discover new algorithmic gains and new

ways of running AI says that AI gets better. And so, you know, the thing they're heading towards is, intelligence explosion, and of course, once you start being able to create AI that makes AI which makes AI better, you end up in this like double exponential curve. And none of that asymmetric power, which comes within an asymmetric ability to persuade among many other things, none of that requires whether you are conscious or not. But where it intersects with the question of consciousness is the rate at, which new technology will be coming out,

the rate at which scientists or labs can try out, hey, maybe this thing is conscious, that's going to get faster and faster and faster. And so the ability for us to have the moderating effect to even be able to do this kind of philosophy on a deadline gets shorter and shorter and shorter and shorter as the consequences go higher and higher and higher. Yeah, this notion of the double exponential is troubling.

I wonder about it a bit, though. There's also arguments that things might be plateauing and flatlining in some respects. I mean, psychologically, again, our ability to deal with exponential curves is notoriously bad. We always feel like we're on a threshold, but that's just how exponential curves look

wherever you are on them. Things are moving fast, but there are things we can do. I mean, the first thing is just to continually remind ourselves, because it's hard because we're psychologically programmed to think otherwise, but just to continually remind ourselves that consciousness and intelligence are different things, that consciousness won't just come along inevitably as systems get smarter and reach some threshold. It's not just going to happen. And also that there's a difference between systems

that are conscious and systems that seem conscious. And they have these different ethical consequences. One is more about the problems of bringing into the world massive new forms of suffering. The other is what it would do to us as human beings, brutalizing our minds, making a psychologically vulnerable. Different situations, different risks, one very unlikely uncertain, one already with us. So seeing the landscape clearly, I think, is really the first step, and that can get us a long way.

We're in this trouble in the first place because we don't have a sufficiently good understanding of the nature of consciousness. It was always one of the biggest problems in the world. It's been perplexing thinkers since people have been thinking. What's been amazing to me over the 25, 30 years I've been doing this is that progress is being made. We don't have the full answer,

but what is known now about consciousness and how to think about it has changed. And these changes have implications already for how we think about consciousness in other systems, animals, machines. So really, and this is a little bit of a self-serving, prescription but honestly more effort more resources to better understanding consciousness is really really important and then the last word is humility for me no one knows how consciousness happens no one knows how to guarantee to avoid creating something that's conscious so we need to take all these things with a bit of a pinch of salt and recognize as a plurality of views but not throw

our hands up and realize because anything we can still say things we can still have have informed guesses, but let's beware of certainty. Anil, this has been fascinating and even more so, I think, has drawn out a lot of the consequences for what we get wrong when we think about AI and consciousness and what its implications are going to be. So thank you very much for coming on the show. Thank you, Asa. It's been a delight speaking to you. Really enjoyed it.

Thank you very much for having me. Your undivided attention is produced by the Center for Humane Technology, a non-profit working to catalyze a humane future. Our senior producer is Julia Scott. Josh Lash is our researcher and producer. And our executive producer is Sasha Fegan. Mixing on this episode by Jeff Sudakin. Original music by Ryan and Hayes Holiday.

If you like the podcast, we'd be grateful if you could rate it on Apple Podcasts because it helps other people find the show. And if you made it all the way here, let me give you one more thanks for giving us your undivided attention.