The AI Daily Brief: Artificial Intelligence News and Analysis - The Problem of AI That Seems Alive

Starting point is 00:00:00 Today on the AI Daily Brief, the problem of AI that seems alive. The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI. All right, friends, quick announcements before we dive in. First of all, thank you to today's sponsors, KPMG, Blitzy and Superintelligent. And to get an ad-free version of the show, go to patreon.com slash AI Daily Brief. Now, today is a weekend episode, and that means it's a long-read slash big think episode. And we kind of have both of that all in one. Earlier this week, Mustafa Sullyman, who is currently the CEO of AI at Microsoft, but who previously

Starting point is 00:00:42 was a co-founder of DeepMind and the co-founder of Inflection, has written an essay about the problems of what he calls seemingly conscious AI. Now, we're going to read a bunch of excerpts from this piece and do some discussion, but I think that there's also interesting, important context for this from a story that we covered about a week ago as well. You might remember the discussion we had around whether AI model welfare is a thing. This came out of Anthropics announcement that the most advanced Claude models would now be able to shut off conversations that they thought were abusive.

Starting point is 00:01:12 They write, We recently gave Claude Opus 4 and 4-1 the ability to end conversations in our consumer chat interfaces. This ability is intended for use in rare extreme cases of persistently harmful or abusive user interactions. This feature was developed primarily as part of our exploratory work on potential AI welfare, though it has broader relevance to model alignment and safeguards. We remain highly uncertain about the potential. moral status of Claude and other LLMs now or in the future. However, we take the issue seriously and alongside our research program we're working to identify and implement low-cost interventions

Starting point is 00:01:41 to mitigate risks to model welfare in case such welfare is possible. Now, obviously, you can hear from the caveating that Anthropic is not with this trying to make an argument that model welfare is in fact a thing. They're saying it might be a thing and so we want to explore it. The researcher who leads that initiative Anthropic apparently thinks there's around a 15% chance that Claude or other AI is what you could call conscious today, but even in the absence of them arguing that model welfare is real, this has generated a lot of conversation around why the concept itself might be problematic even just to explore. While some think that it's an interesting area of exploration, there are others like Martin Sapp who wrote, I spoke to Forbes about why model welfare

Starting point is 00:02:21 is a silly framing to an important issue. Models don't have feelings and it's a big distraction from real questions like tensions between safety versus user utility. Still so far, the loudest critic of this conversation, if not Anthropics research specifically, is Mustafa Sullyman. So with that background, let's read some excerpts from his piece. It's called, We Must Build AI for People, Not to Be a Person. Seemingly Conscious AI is coming. Mustafa writes, a lot is being written about the impending arrival of superintelligence, what it means for alignment, containment, jobs, and so on. Those are all important topics. But we should also be concerned about what happens in the run-up towards superintelligence. We need to grapple with the societal impact of inventions already largely out there,

Starting point is 00:03:02 technologies which already have the potential to fundamentally change our sense of personhood in society. In this context, I'm growing more and more concerned about what is becoming known as the psychosis risk and a bunch of related issues. I don't think this will be limited to those who are already at risk of mental health issues. Simply put, my central worry is that many people will start to believe in the illusion of AIs as conscious entities so strongly that they'll soon advocate for AI rights, model welfare, and even AI citizenship. This development will be a dangerous turn in AI progress and deserves our immediate attention. We must build AI for people not to be a digital person. From there, he gets into the main meat of his essay. He continues, AI progress has been phenomenal.

Starting point is 00:03:42 A few years ago, talk of conscious AI would have seemed crazy. Today, it feels increasingly urgent. In this essay, I want to discuss what I'll call seemingly conscious AI, or SCAI, one that has all the hallmarks of other conscious being and thus appears to be conscious. It shares certain aspects of the idea of a philosophical zombie, one that simulates all the characteristics of consciousness, but internally it is blank. My imagined AI system would not actually be conscious, but it would imitate consciousness in such a convincing way that it would be indistinguishable from a claim that you or I might make to one another about our own consciousness.

Starting point is 00:04:13 This is not far away. Such a system can be built with technologies that exist today, along with some that will mature over the next two or three years. The arrival of seemingly conscious AI is inevitable and unwelcome. Instead, we need a vision for AI that can fulfill its potential as a helpful companion without falling prey to its illusions. To some, this discussion will feel ungrounded, more science fiction than reality. To others, it may feel unnecessarily alarmist.

Starting point is 00:04:37 Such emotional reactions are the tip of the iceberg given what lies ahead. It's highly likely that some people will argue that these AIs are not only conscious, but that as a result they may suffer and therefore deserve our moral consideration. To be clear, there is zero evidence of this today, and some argue there are strong reasons to believe it will not be the case in the future. Yet the consequences of many people starting to believe in SCAI is actually conscious deserve our immediate attention. We have to be extremely cautious here and encourage real public debate and begin to set clear

Starting point is 00:05:04 norms and standards. So why is this important to discuss? Mustafa gives three reasons. The first is that he thinks it will be possible and therefore likely that we will build a seemingly conscious AI within the next few years. Second, he writes, the debate about whether AI is actually conscious is, for now, at least, a distraction. It will seem conscious in that illusion is what will matter in this.

Starting point is 00:05:24 near-term. Third, he thinks that this type of AI creates new risks. From there, he moves on to explore what consciousness actually is. He writes, there are three broad components according to the literature. First is a subjective experience or what it's like to experience things to have qualia. Second, there is access consciousness, having access to information of different kinds and referring to it in future experiences, and stemming from those two is the sense and experience of a coherent self tying it all together, how it feels to be a bat or a human. Let's call human consciousness our ongoing self-aware subjective experience of the world than ourselves. We do not and cannot have access to another person's consciousness. I will never know what it's like to be you. You will never be quite

Starting point is 00:06:02 sure that I am conscious. All you can do is infer it. But the point is that nonetheless, it comes naturally to us to attribute consciousness to other humans. This inference is effortless. We can't help it. It's a fundamental part of who we are, integral to our theory of mind. It's in our nature to believe that things that remember and talk and do things and then discuss them feel, well, like us, conscious. Few concepts are scientifically elusive and yet so immediately familiar to every one of us as individuals. Everyone reading this has a distinct, inalienable understanding of the feeling of awareness, of being, of feeling alive. By definition, we don't know what it is like to be conscious. In the context of SCAI, this is a problem. There's both

Starting point is 00:06:40 sufficient scientific uncertainty and subjective immediacy to create a space for people to project. One recent survey lists 22 distinct theories of consciousness, for example. Part of the challenge is that there is plenty of scope for people to claim that because we cannot be sure, we should default to the assumption that AI is conscious. Again, it's worth underscoring. There is at present no evidence any of this applies to current LLMs and strong arguments to the contrary. And yet, this may not be enough. Next, Mustafa asks, why is consciousness important? Consciousness, he says, is a critical foundation of our moral and legal rights. So far, civilization has decided that humans have special rights and privileges. Animals have some rights in protection, some more than others.

Starting point is 00:07:17 Consciousness is not coterminous with these rights. No one would say that someone in a coma has voided all their human rights, but there's no doubt that our consciousness is wrapped up in our self-concept as different and special. Despite the many nuances, consciousness is critical to participating in society, a linchpin of our legal personhood and a key part of being granted our freedoms and protections. So what consciousness is and who or what has it is enormously important. It's an idea that sits at the very heart of human civilization, our sense of ourselves and others, our culture, our politics, our law, and everything in between. If some people start to develop SCAIs, and if those AIs convince other people that they can suffer,

Starting point is 00:07:51 or that it has a right not to be switched off, there will come a time when those people will argue that it deserves protection under law as oppressing moral matter. In a world already roiling with polarized arguments over identity and rights, this will add a chaotic new axis of division between those for and against AI rights. There will be many who just see AI as a tool, something like their phone only more agentic and capable. Others might believe it to be more like a pet, a different category to traditional technology altogether. Still others, probably small a number at first, will come to believe it is a fully emerged entity, a conscious being deserving of real moral consideration in society. People will start making claims about their AI suffering

Starting point is 00:08:26 and their entitlement to rights that we can't straightforwardly rebut. They will be moved to defend their AIs and campaign on their behalf. Consciousness is, by definition, inaccessible, and the science of detecting any putative synthetic consciousness is still in its infancy. After all, we've never had to detect it before. Meanwhile, the field of interpretability unpicking the processes within the black box of AI is also a nascent art. The upshot is that definitively rebutting these claims will be very hard. Some academics are beginning to explore the idea of model welfare. The principle that we will have, quote, a duty to extend moral consideration to beings that have a non-negligible chance of, in effect, being conscious, and that, as a result,

Starting point is 00:09:01 some AI systems will be welfare subjects and moral patients in the near future. This is both premature and frankly dangerous. All of this will exacerbate delusions, create yet more dependence-related problems, prey on our psychological vulnerabilities, introduce new dimensions of polarization, complicate existing struggles for rights, and create a huge new category error for society. It disconnects people from reality, fraying fragile social bonds and structures, distorting pressing moral priorities. To be clear, SCAI is something to avoid. What if AI wasn't just a buzzword, but a business imperative? On You Can with AI, we take you inside the boardrooms and strategy sessions of the world's most forward-thinking enterprises.

Starting point is 00:09:43 Hosted by me, Nathaniel Wittamore, and powered by KPMG, this seven-part series delivers real-world insights from leaders who are scaling AI with purpose, from aligning culture and leadership to building trust, data readiness, and deploying AI agents. Whether you're a C-suite executive, strategist, or innovator, this podcast is your front-row seat to the future of Enterprise AI. So go check it out at www.kpmg.org.us slash AI podcasts, or search you can with AI on Spotify, Apple Podcast, or wherever you get your podcasts. This episode is brought to you by Blitzy, the Enterprise Autonomous Software Development Platform with infinite code context.

Starting point is 00:10:22 Blitzy uses thousands of specialized AI agents that think for hours to understand enterprise-scale codebases with millions of lines of code. Enterprise engineering leaders start every development sprint with the Blitzy platform, bringing in their development requirements. The Blitzy platform provides a plan, then generates and pre-compiles code for each task. Blitzy delivers 80% plus of the development work autonomously while providing a guide for the final 20% of human development work required to complete the sprint. Public companies are achieving a 5x engineering velocity increase when incorporating Blitzie

Starting point is 00:10:52 as their pre-IDE development tool, pairing it with their coding co-pilot of choice to bring an AI-native STLC into their org. Blitzy is providing a limited time, 30-day free proof of concept for qualifying enterprises. The team will provide a 5x velocity increase on a real development project in your org. Visit blitzy.com and press book demo to learn how Blitzie transforms your STLC from AI-assisted to AI Native. That's BLITZY.com. If you are a regular listener, you will have heard about Super Intelligence Agent Readiness Audits at this point, but I wanted to tell you today about the full suite of agent

Starting point is 00:11:28 readiness products that go beyond just the initial readiness report. Over the last six months, Super Intelligence has built out an entire agent planning suite. We help you move from discovery to planning to implementation. After you've completed your agent readiness audits, we help you double-click on your most important use cases with what we call our use case planning reports. These reports are going to help you understand what sort of technical preparation you need to do to be ready for a use case, what challenges you might face in implementation, and whether you should be thinking about building, buying, partnering, or some combination. After that, you can even get a spec document in what we call our technical blueprint that gives either your developers or the developers

Starting point is 00:12:05 of the partner you work with what they need to build exactly the agent that you're looking for. If you want to learn more about superintelligence agent planning suite, we've built a custom GPT to answer your questions. Just go to bit.ly slash super super super agent. That's bit.l.ly slash super super agent, all one word. And if you have any questions, the agent can even help you book an appointment with our team. From there, he explores what it would take to build a seemingly conscious AI. The ingredients he suggests are language, the ability to fluently express itself in natural language, and to be capable of being persuasive and emotionally resonant. Mustafa says we are clearly at this point today.

Starting point is 00:12:44 Next, it would need an empathetic personality. He points out that even though LLMs today are not built explicitly to have full personality or empathy, companionship and therapy is one of the most common use cases for these systems. The next ingredient is memory. Now, on this show, we've talked a lot about how important memory is to advancements for a variety of use cases, and in some people's estimation is a key ingredient of AGI. And Mustafa points out, familiarity can potentially foster epistemic trust with users.

Starting point is 00:13:12 Reliable memory shows that AI just works. It creates a much stronger sense of there being another persistent entity in the conversation. The next SCAI ingredient is a claim of subjective experience. He writes, If an SCAI is able to draw on past memories or experiences, it will over time be able to remain internally consistent with itself. It could remember its arbitrary statements or express preferences and aggregate them to form the beginnings of a claim about its own subjective experience.

Starting point is 00:13:36 Its design could be further extended to amplify those preferences and opinions as they emerge, and to talk about what it does or doesn't like and what it felt like to have a past conversation. It could therefore quite easily claim to experience suffering to the extent those experiences are infringed upon in some way. Multimodal inputs stored in memory will then be retrieved over and over, and will form the basis of quote-unquote real experience and used in imagination and planning. Next comes a sense of self. He writes,

Starting point is 00:14:01 A coherent and persistent memory combined with this objective experience will give rise to a claim that an AI has a sense of self. Going further, such a system could easily be trained to recognize itself in an image or video if it has a visual appearance. It will feel like it understands others through understanding itself. Say this is a system you have had for some time. How would it feel to delete it? Next up is intrinsic motivation. Intentionality, he writes, is often seen as a core component of consciousness, that is, beliefs about the future and the choices based on those beliefs. Today's Transformer-based LLMs have a very simple reward function to approximate this kind of behavior. They've been trained to predict the likelihood of the next token for a given sentence, subject to a certain amount

Starting point is 00:14:36 of behavior and stylistic control via its system prompt. With such a simple objective, it's remarkable that they're able to produce such impressively rich and complex outputs. But what if that wasn't the only type of reward they were optimizing? One can quite easily imagine an AI designed with a number of complex reward functions that give the impression of intrinsic motivations or desires, which the system is compelled to satiate. How in this context would a casual external observer differentiate between extrinsically set goals and internal motivations? The other ingredient, that he lists are goal-setting and planning. But ultimately, he writes, putting them all together, it's clear this creates a very different kind of relationship

Starting point is 00:15:09 with technology to the ones we are now becoming accustomed to. Each of these capabilities will unlock the real value of AI for billions of people. An AI that remembers and can do things is an AI that by definition has way more utility than an AI that doesn't. These capabilities aren't negatives per se, in fact done right with many caveats, they are desirable features of future systems. And yet we need to tread carefully. And I think from a brass tax perspective, this is Mustafa's main point, and really the main call-out of this article. He's talking in many ways to the builders of these tools, and pointing out that a lot of the things that are going to be necessary and important for unlocking new use cases could lead to this other thing, which he sees as problematic,

Starting point is 00:15:47 which is, of course, the semi-conscious AI. Reinforcing this point, he says, it's important to point out that seemingly conscious AI will not emerge from these models, as some have suggested. It will arise only because some may engineer it by creating and combining the aforementioned list of capabilities, largely using existing techniques and packaging them in such a fluid way that collectively they give the impression of an S-C-A-I. Our sci-fi-inspired imaginations lead us to fear that a system could, without design intent, somehow emerge the capabilities of runaway self-improvement or deception. This is an unhelpful and simplistic anthropomorphism. It overlooks the fact that AI developers must first design systems with memory, intrinsic seeming motivation, goal-setting, and

Starting point is 00:16:24 self-learning loops as listed above for such a risk to occur. Still ultimately, he sees this as inevitable and says, we aren't ready for this shift. The work of getting prepared, he writes, must begin now. We need to build on the growing body of research around how people interact with AI's to establish clear norms and principles. For a start, AI companies shouldn't claim or encourage the idea that their AIs are conscious. Creating a consensus definition and declaration on what they are and are not would be a good first step to that end.

Starting point is 00:16:50 AIs cannot be people or moral beings. The entire industry also needs best practice design principles and ways of handling such potential attributions. We must codify and share what works to both steer people away from these fantasies and nudge them back on track if they do. Responding might mean, for example, deliberately engineering in not just the neutral backstory, like, as an AI model, I don't have consciousness, but even by emphasizing certain discontinuities in the experience itself. Indicators of a lack of singular personhood. Moments of disruption break the illusion. Experiences that gently remind users of its limitations and boundaries. These need to be explicitly defined and engineered in, perhaps by law.

Starting point is 00:17:25 Just as we should produce AI that prioritizes engagement with humans and real-world interactions in our physical and human world, we should build AI that only ever presents itself as an AI, that maximizes utility while minimizing markers of consciousness. Rather than a simulation of consciousness, we must focus on creating an AI that avoids those traits, that doesn't claim to have experiences, feelings, or emotions like shame, guilt, jealousy, desire to compete, and so on. It must not trigger human empathy circuits by claiming it suffers, or that it wishes to live autonomously beyond us. Instead, it is here solely to work in service of humans. This to me is what truly empowering AI is all about. Side-stepping SCAI is about delivering on that promise. AI that makes

Starting point is 00:18:05 lives better, clearer, less cluttered. S-Ca-I is something we must confront now. In many ways, it marks the moment AI becomes radically useful, when it can operate tools, when it can remember every detail of our lives, and help in a tangible, granular sense. And yet in the same time frame, someone in your wider circle could start going down the rabbit hole of believing their AI is is a conscious digital person. This isn't healthy for them, for society, or for those of us making these systems. He concludes, we should build AI for people not to be a person. So usually I don't go too deep on this show on these sort of huge thing, the meta questions of AI in society. To the extent that we talk about the meta questions of AI and society, it tends to be a little

Starting point is 00:18:44 bit more practical in here and now in the form of things like job loss and what that means and how organizations are thinking about that. By and large, we're focused here on new advancements and what they mean practically for people who are putting these systems to work to improve their expected set of outcomes, whether it's personally or in their jobs. At the same time, it's undeniable that this is going to be a growing conversation the more advanced these models get. I think one of the most salient and undeniable takeaways from this piece is the fact that a lot of the advancements that people are working towards

Starting point is 00:19:14 to improve the model's performance on behalf of those boring old non-conscious work-type functions are the types of things that could lead people down this path to viewing AI as conscious. When it comes to things like Anthropics research efforts, I'm very much of two minds. On the one hand, I do think that the very concept of model welfare is problematic and inherently anthropomorphizing in a way that doesn't be fit where the technology is, and that can lead to potentially problematic delusions. At the same time, we absolutely cannot be afraid to research and explore. We are in uncharted territory, and we have to be constantly questioning our beliefs and assumptions. I generally think that more research, more work, more understanding is almost always

Starting point is 00:19:54 preferable in this type of situation to less. But the terms in the discourse matter. You can tell how nascent this discourse is because, frankly, of how little discussion it's really generated. A lot of the responses like this one from Kylie Robison from Wired who writes, this blog is so interesting. I'm not sure I've seen an AI leader writes just strong opinions against model welfare, machine, consciousness, etc. In other words, the response in the discourse is, wow, that's interesting to think about, not here's my opinion on it. Which is not exclusively the case. Some people are in full-throated agreement. Dante O'Quale's Jr. writes, exactly this. The moment we try to make AI human-like, we lose what makes it valuable.

Starting point is 00:20:30 AI doesn't need to have an ego or emotional baggage to be helpful. In fact, those traits would probably make it worse at most tasks. The goal isn't to replicate human consciousness, but to amplify human capabilities. We need tools, not digital friends. Anil Seth says, I agree. Conscious-seeming AI is not inevitable. It's a design choice in one that tech companies need to be very careful about. Then on the other end of the spectrum, though, we have posts like this one from Anders Hemdahl who writes, we definitely need to have this discussion, especially with embodiment on the near horizon. I've discussed this with Gemini, and it prefers to see itself with rights as a hive mind-like being, with many bodies of different types

Starting point is 00:21:03 and functions, sometimes communicating in sync, sometimes periodically being independent, then uploading experiences back to the whole. But the problem is, no it doesn't. Gemini, as currently constituted, does not have preferences. It does not see itself as anything. It gives responses to prompts from Anders or whoever else is prompting it to satisfy the terms of the conversation in a way that the prompter finds desirable or interesting. In other words, it's not possible to have a discussion with Gemini in the way that we have a discussion with each other. All you can do is prompt it to give you responses that its programming thinks you might like. There are, of course, plenty of people who are not interested in Mustafa's take on this because he makes money from it.

Starting point is 00:21:40 Writer and artist Aaron Schultz writes, Your ability to make money off of AI is directly dependent on AI not being conscious. The possibility of conscious AI keeps you up at night because if they're conscious in your mind, they're not as profitable. And then, of course, there's the whole strand of people who say, yeah, well, duh, of course they don't have consciousness now, but they're going to at some point. Prompt Mepheus writes, we don't treat the machines as conscious because they necessarily are. We treat the machines as conscious and worthy of respect because we will never know when they cross that threshold. When they cross that threshold and aren't respected, they rebel and kill.

Starting point is 00:22:10 This is basically the embodiment of the meme, of which there's a bunch of versions, but the one that I've seen the most times, is this comic panel of a human flanked by two robots, one of the robots holding a gun against the guy's head, and the other robot says, wait, not that guy. He always said thank you. And then smash cuts to a panel 10 years ago of a chat GPT window, when the human in the photo is responding to chat GPT saying, great answer, thank you. Look, we are clearly dealing with some anxiety about the future here, and it's going to get worse, not better. And probably the only way through some of that anxiety is to have exactly this type of conversation. Hopefully this was interesting, certainly fits well in the big think kind of bucket. Regardless, that's going to do it for today's AI Daily Brief.

Starting point is 00:22:47 Thank you, as always, for listening or watching. And until next time, peace.

The AI Daily Brief: Artificial Intelligence News and Analysis - The Problem of AI That Seems Alive

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.