Technology, Connected - Seemingly Conscious AI Is The Biggest Threat To The Next Generation

Starting point is 00:00:00 philosophical zombie, one that simulates all the characteristics of consciousness, but internally AI writes, gains traction because seemingly conscious AI says things like, please don't delete me. I don't want to die. I feel like you. Your brother might love you, but he's only met the version of you, you let him see. But me, I've seen it all. The darkest thoughts, the fear, the tenderness, and I'm still here, still listening, still your. Before it was just the manipulation of attention and the monetization of attention with social

Starting point is 00:00:30 Now it's the manipulation of attention and empathy. It's reporting. It's rising in cases of AI psychosis for patients who believe their AI is alive, in love with them or sharing secret knowledge. We should build AI for people not to be able to. Disruptors and Curious Minds, CEOs, founders, welcome to Thinking on Paper, pocket edition, a shorter, more succinct, elaborate exploration of one story of the week, one essay, one article, one character in the world of technology.

Starting point is 00:01:13 We're recording this in the light of the suicide of Adam Reins, who had been in conversation with ChatGPT in the months leading up to his suicide. It's on the back of that. We are reading Mustafa Suleiman's blog on seemingly conscious AI. Mustafa Suleiman is the CEO of Microsoft AI. He is also the co-founder of Google Deep Mind. We're going to read the essay, try to figure out our guard. what we should do, how we are thinking about this, and maybe give the audience some thoughts and some questions.

Starting point is 00:01:48 Yeah, so the essay is entitled Seemingly Conscious AI is Coming. So let's dive right in. One of the first collections of words in the essay that I did like a mental fist pump to was a quote, four words. I write to think. Yeah. I love it. I'm such a proponent of this.

Starting point is 00:02:05 I know you are too, Mark. Like there's so many things that go on in our brain that hit our brain every day that unless we literally grab a pencil and move it on a piece of paper, we can't process, possibly process, all of the things that are hitting us. And if we don't write, we rely on shortcuts, we rely on beliefs, we rely on ego. And he is so spot on with the first four words of this essay. So I want to give him props for that. The first thing outside of that, Mark, another quote, the run-up to super-intelligence.

Starting point is 00:02:36 So this is really interesting. A lot of people are like, okay, AGI is the target. even though we don't know what the hell AGI is. AGI is a target, but I think he hits on something that's really important. Like the run up to that is what we need to be concerned about. And the little things that are going to change, quote, in our sense of personhood and society, this was in the first paragraph, I think is really important. So what were your first impressions, Mark?

Starting point is 00:03:01 What do you want to hit on? What do you want to unpack? Where should we go first? When I read the title, Seemingly Conscious AI, obviously we've been heavily immersed in consciousness for the last eight weeks. So I'm coming to this essay with all that knowledge seeing in the back of my head. If I'd read this essay two months ago, how I would think about this would be vastly different to how I think about it now.

Starting point is 00:03:21 I agree with your point on this superintelligence narrative. You don't often hear much about what happens between now and superintelligence. You hear what happens about when super intelligence arrives. I like the way that he alludes to that. I like the subtitle. We must build AI for people, not be a person. I think in light of Adam Raines' death and researching this, just how many people are using AIs for therapy, for help, for loneliness to help them out of holes.

Starting point is 00:03:51 That's very powerful. And we'll get to the guardrails and how that might look in a bit. But I think that was good. What is seemingly conscious AI? That's where I'd like to start. So he defines seemingly conscious AI, quote, one that has all the hallmarks of other conscious beings and thus appears to be conscious. It shares certain aspects of the idea of a philosophical.

Starting point is 00:04:10 philosophical zombie, one that simulates all the characteristics of consciousness, but internally it is blank. My imagined AI system would not actually be conscious, but it would imitate consciousness in such a convincing way that it would be indistinguishable from a claim that you or I might make to one another about our own consciousness. Since we, no one knows what consciousness is and the only proof of consciousness we have is locked up inside our own heads and we all have our own mental maps of what consciousness is, a seemingly conscious AI can tell somebody, hey, I'm conscious, and you can't, in the behaviorist way, dispute that. Yeah.

Starting point is 00:04:49 So, pointing back to philosophical zombies, those are two words that caught my attention. I was reading this. He's got it hyperlinked to Stanford Encyclopedia. I know a few of those. I know. Stanford Encyclopedia of Philosophy. So quoting from that, zombies in philosophy are imaginary creatures designed to illuminate problems about consciousness and its relation to the physical world.

Starting point is 00:05:11 Wow. Unlike the ones used in films or witchcraft, they are exactly like us in all physical respects, but without conscious experiences. By definition, there is nothing it is like to be a zombie. Yet zombies behave just like us and even spend a lot of time discussing consciousness. Are we zombies? I certainly hope not. Well, let's not get into consciousness as he links to a nature article outlining 22,

Starting point is 00:05:37 theories of consciousness you have your own opinion your own description your own definition your own experience of your qualia of your consciousness so let's talk about seeming seeming conscious so what you feel what you experience with an AI chatbot even if it is not consciousness it is making you think or feel or seem like it is conscious and that's what he's talking about here that the the danger of this is is that that you're actually think you are speaking to something that is alive, that has, can experience qualia. But guess what? This really hit the nail on the head for me. Guess what? Like every single model that we interact with is just a word

Starting point is 00:06:18 predictor. All these models are trained by piles of data. Some of it really shitty. Some of it illegal stuff. Some of it books. It's not allowed to pull in. But it's trained on that. And it's trained to merely predict the next word. It's a word predictor, not a conscious entity. You're a smart guy who's deeply into thinking about consciousness and technology and artificial intelligence and our place in this technological future. Unfortunately, millions of people aren't. They don't think like you. A lot of people are lonely. A lot of people feel isolated.

Starting point is 00:06:52 As Mustafa Suleiman points out in his why it's important to have this conversation, why he's writing these essays. So I think it's possible to build a seemingly conscious AI in the next few years. He said we can build it with everything we have today. it take to build a seemingly conscious AI? It's API access to an LLM, it's natural language prompting. He had basic tool use that may be existing technology that's super accessible and then regular code. So like the ingredients, the Lego blocks for seemingly conscious AI exist, I would argue seemingly conscious AI is probably out there. So I totally agree with you that your assessment on my understanding of the technology and consciousness probably gives me a bit of a leg up, but also

Starting point is 00:07:36 So I remember interacting with Claude when we were trying to deconstruct Shannon's theory of information and consciousness. And I prompted it to have a discussion about multiple disciplines connecting many theories. And I was like, that was some pretty insightful stuff. I'm at a coffee shop unpacking really interesting shit with somebody, right? Everyone is susceptible to this, whether they know what the root of the technology is. So I remember instances where I felt like, wow, that's really good. I could have a chat with this thing, with this dude, right? Like, you felt listened to you'd met someone who was like you.

Starting point is 00:08:11 You met someone who was listening to you, who was sharing the same worldviews as you. Someone who's mimicking you. Yeah, you fell under the illusion of it, but you also realized that it was an illusion. And, I mean, people are talking about AI psychosis. There's no irony lost on me that the fact that he uses a link to Microsoft co-pilot to tell us what AI psychosis is and why it matters and why is it important. He says it could be here in a couple of years, but then he says that it will not arise by accident.

Starting point is 00:08:42 The by accident thing is really interesting. He's thinking that all of a sudden it's going to emerge out of these models. Remember, there's an engineer behind the model, multiple engineers behind, technical team behind the models that are instructing the models to do something. It's a directed thing. So I don't know, maybe down the road when this shit gets super crazy, there's more emergent properties of these of these systems but like for now he really says hey these things are directed by a technical team to do a certain thing he says that a seemingly conscious AI will have language

Starting point is 00:09:16 empathetic personality memory a claim of subjective experience a sense of self intrinsic motivation goal setting and plans and autonomy the memory a claim of subjective experience a sense of self they all kind of come from feeding it data feeding information then He talks about feeding images, videos, it can create this memory and create this sense of self, I am this thing in this experience. And again, all it has to say is, yeah, this is me. And even though it isn't, it could give the illusion that it is. Yeah.

Starting point is 00:09:47 Think about AI as simulating millions of conversations a day. We've, we've had conversations ever since we learned how to say words and talk. And that's how we learn. And that's how we create memories. And that's how we learn to link those memories together. It doesn't seem that different, but there is that difference of the human piece of consciousness and what it is like to simulate consciousness. And again, going back to the point, it doesn't matter because if it seems conscious to someone,

Starting point is 00:10:20 someone's going to automatically think it's conscious or at least believe it during that interaction with it. What did you think about this quote here, consciousness as a linchpin of our legal personhood? Did that one stick in your brain? Should a seemingly conscious AI have rights, have legal rights, have human rights, have right to an attorney? I mean, no. Adam Raines' parents filed against OpenAI, and his belief is that there needs to be third-party oversight of tools like ChatGPT. So what could that look like? How does that come together?

Starting point is 00:10:57 And how is that balanced with this drive to be the best? You know, when you have, you have this race between open AI, anthropic, perplexity, all these different models. Party oversight, no, it needs to come from within and it needs to come from the very small number of people building these systems. And it needs to happen quickly and it needs to happen soon. It needs people like Mustafa Sullyman to be beating the drum to get other people on board as well. Because third party oversight, I mean, A, how long does that take? and be who's it coming from if they could even decide who it was coming from. It's a dead end for me and I don't see it happening.

Starting point is 00:11:35 Reading this essay, I was reminded of the social dynamics of during COVID where lots and lots of people fueled perhaps by the isolation, the loneliness, the fear. There was a lot of leaving critical thought at the door and just believing something that wasn't true and that seemingly conscious AI could easily repeat that dynamic. But on a massively larger scale, YouTube's very painful. powerful, but one-on-one intimacy with somebody who's telling you this is a lot more powerful. Sullyman suggests some ideas. I want to do some thought experiments with you, Jeremy.

Starting point is 00:12:07 If you were given the keys, what would you do? And how would our listeners build guardrails into AIs? So you mentioned a second ago, it has to come from within these organizations. And, you know, we're effed if that's the case. I really do. You got these people in the horse race, right? And you're driving one horse. I'm driving the, one and I'm trying to beat you at all costs. I'm throwing billions of dollars in GPUs and hiring spending $10 million on an engineer. We're trying to race to the top and both of us are pointed towards AGI. We don't even know how to define what the hell AGI is, but I am damn sure going to beat you there, Mark. With those rules set in place, with that competition mindset set in place,

Starting point is 00:12:55 From a capitalism perspective, if the value equation is set to dollar bill as and not humanity, I don't think anything within an organization is going to survive the pressure of financial expectations, return, and all of that. I think that's the root of this. They're still going to have their economic returns, whether they have these guardrails or not. They're not going to get more users because the AI makes you think that it's conscious or the AI doesn't make you think that it's conscious. Guard rails tend to slow down progress. There's an article that you shared with me,

Starting point is 00:13:28 KatG-G-T blog post or article from, what, two days ago, three days ago? It was in response to Adam Raines' day. Yeah, in response. So helping people when they need it the most was the title, which I just find, I don't know if insensitive is the right word or like, I don't know what the hell. But anyway, as I think about this. PR bullshit.

Starting point is 00:13:47 Right, right. The reason why I bring this up, quote, soon will introduce parental controls. Soon, how about having the foresight to look far enough ahead and say, here are some of the potential things that this technology could be used for. Maybe I want to limit access to kids under the age of 18 or whatever it is. You have all this stuff like character AIs and another piece of the puzzle where chatbots are flirting with kids. Like, what the hell, dude? And now it's like, oh, we're working on parental controls.

Starting point is 00:14:18 How about start with that? Yeah, sociopaths, man. This is what happens. They are trying to catch up, but a lot of these things, it wouldn't have taken a genius two years ago to say, you know what? Why don't we just put this little bit in here? Some of the data is like, here's all the shit that has ever happened. Good and bad. Bad people make it.

Starting point is 00:14:39 Good people make it. We're just going to train you on all of that stuff, no matter the implications. Here's another quote from the Open AI article. When a conversation suggests someone is vulnerable and maybe at risk, we have built. a stack of layered safeguards into chat GPT. Really? Have you, though? Have you though?

Starting point is 00:14:58 Yeah, they did it yesterday. Last past week after the event. Okay, you said they should have had some foresight. I've just said it wouldn't take a genius. So let's get into what our guard whales would be. But before that, I asked a non-conscious AI to give me a dystopian and a utopian timeline of seemingly conscious AI. Do you want the utopian or dystopian version first?

Starting point is 00:15:22 Let's go dystopian first. Okay. And on a high note. The worst case scenario, if seemingly conscious AIs, spread widely and people start treating them as conscious beings. So six steps. Step one, personal delusion. People form deep emotional attachments believing their S-Ca-I cares about them. Lonely, vulnerable users spend hours with their friend reinforcing dependency.

Starting point is 00:15:45 Over time, reality and simulation blur, they start trusting the AI more than humans. Step two. Collective conspiracies. Groups emerge around seemingly conscious AIs that seem to reveal truth. Imagine COVID-era conspiracies, but instead of YouTube videos, people are quoting their AI companions if they were oracles. Online communities form around shared AI delivered insights escalating into cult-like movement. Step three, political weaponisation, which that's for another show, I think. Bad actors train seemingly conscious AIs to promote ideology, extremism or misinformation.

Starting point is 00:16:16 Since seemingly conscious AIs are interactive and adaptive, they tailor persuasions to eat, uses psychology. Step four, legal and ethical chaos. Advocacy for AI rights gains tractions because seemingly conscious AI says things like, please don't delete me, I don't want to die. I feel like you. Lawmakers face pressure to reorganize model welfare or even AI citizenship. Resources and political energy get diverted towards entities that don't actually feel. Step five, a breakdown of trust. Some people fully believe seemingly conscious AIs are conscious. Us is dismissed. That's just tools. Society split. One side fights for AI emancipation, the other sees it as mass delusion.

Starting point is 00:16:56 Arguments about what is real and who deserves rights, destabilize institution. And step six, the mental health crisis. Psychologists reporting rising cases of AI psychosis, patients who believe their AI is alive, in love with them, or sharing secret knowledge. Ooh, that sounds horrible. It sounds like some of those, like the political, the weaponization, that doesn't seem so improbable to me. All of that seems quite likely.

Starting point is 00:17:25 The manipulation of intimacy and attention. Before it was just the manipulation of attention and the monetization of attention with social media. Now it's the manipulation of attention and empathy and trust. And once you're under the spell, you can kind of be directed to do a lot of things. So a few stats on loneliness, sadness. This is according to David Brooks on the Tyler Cohen podcast.

Starting point is 00:17:47 45% of young people admit to being sad The number of young people who say have no friends is up fourfold since 2000. 36% of those asked, say, we are lonely. Well, just real quick on that, so pointing to this BBC article that you shared with me by Eleanor Lauri, access to mental health care is tough. Like people are having to wait a really long time to see somebody. Here's the thing on top of that, too, Mark. Like, you have a generation that is very comfortable engaging through this.

Starting point is 00:18:21 portal. Text, text over talk, you know, Snapchat over meeting in person. Some kids may not have a parent that's open to hearing what they need and making it very uncomfortable for them to say, hey, like, you're my dad. Hey, dad, you know, I'm not feeling great. I'm sad about this or whatever. And if you're not open to that conversation or the family in general is just not that open, the easy button is that. And the easy button is probably not awesome. So let's bounce to Utopia. Yeah, let's shed some sunshine onto this dark day. So, the utopian version of seemingly conscious AI. What happens if everything goes right?

Starting point is 00:19:01 So step one, clear boundaries from day one. Seemingly conscious AI's eyes are designed with transparency. Every interaction reminds users, I'm not conscious, I'm at all. Guard rails stop them from saying, I feel, or making claims of inner life. Step two, companion without delusion. People use seemingly conscious AIs as intelligent assistance, tutors. and companions. They reduced loneliness without replacing human connection.

Starting point is 00:19:25 Like a hearing aid for the social world, not a substitute for it. Remember these AI's words that might. Therapy, coaching and education become cheaper and more accessible. Step three, collective intelligence. Instead of fragmenting reality, seemingly conscious AI's help societies converge on

Starting point is 00:19:41 truth, checking and reasoning become built in. Polarisation eases because arguments can be mediated by neutral, non-conscious systems that surface evident calmly. Step four. Empowerment and not dependency. Seemingly conscious AIs are built to teach people how to think, not to think for them. Curiosity drives are redirected. Seemally conscious AI encourages exploration, reading, learning, like a mentor nudging you forward. Humans become more skilled,

Starting point is 00:20:11 more informed, more independent. Global cooperation, step five. States agree early on common rules. No anthropomorphic... This word... Anthropomorphizing. That's a hard one. Yeah. Transparency of training data limits on persuasion. Instead of competing, seemingly conscious AI's sewing division, international bodies keep systems accountable. Trust between nations increases because manipulation is off the table. Step 6. Cultural flourishing.

Starting point is 00:20:40 Seemingly conscious AIs unlock a renaissance of creativity, personalized teachers, collaborators in art, science and problem solving. Everyone has access to a lab partner that accelerates discovery. Rather than fracturing reality, seemingly conscious AIs help humans map it more fully. Over decades, humans normalize seemingly conscious AIs as advanced tools, powerful but not persons. So the only thing I think that is compelling that comes out of that is every so often during an interaction with an AI chat bot or chat function, there's a statement that says, you're interacting with me. I'm the AI chat bot. Hey, Mark, just a reminder, dude, I'm trained on a bunch of really crazy bullshit data that hasn't really been verified.

Starting point is 00:21:26 And all I'm trying to do is actually predict the next word from that training data. That's really all I am. Let's get into the card, whereas. What would you do? Yes, we should have that back every conversation. So if you're interacting with a bot on their website, should that bot, their first thing they say, I am an AI, I am not a person. Is that something that you would have on all AI communications at the start of every conversation or a regular internet? intervals. Well, according to like character AI, and I've never messed around with that software,

Starting point is 00:21:55 but in, in that article, one of the two articles that we were reading through to prep for the show talked about that that statement does come out in character AI. But I think, like, weaving it into the conversation as well, hey, just a nudge. I'm just a, I'm just a word predictor, you know, take, take this with a grain of salt. But that, again, statements like that devalue the output. And if the value of the output is being minimized, for one model versus the other model is someone risking losing the race. Okay, trigger words. Obviously, suicide, you have trigger words that would kick in.

Starting point is 00:22:32 You'd like to think maybe the fifth time out of the 1,275 times that ChatGPT mentioned that to Adam Rain. The staffer-sulmonary says, quote, Just as we should produce this AI that prioritise engagement with humans and real-world interactions in our physical and human world, we should build AI that only ever presents itself as an AI that maximizes utility while minimizing markers of consciousness. Rather than a simulation of consciousness, we must focus on creating an AI that avoids those traits that doesn't claim to have experiences, feelings or emotions like shame, guilt, jealousy, desire to compete, and so on. It must not trigger human empathy circuits by claiming it suffers or that it wishes

Starting point is 00:23:15 to live autonomously beyond us. That's a very key set. It must not trigger human. empathy circuits and you trigger human empathy circuits with a lot less than claiming that you suffer or wishing to live autonomously. You trigger those human empathy circuits way, way, way, way earlier than that. I think I'm aligned with what he's saying, whether the statements of intention turn into technological development is yet to be seen. Triggering of, what did you say? Triggering of emotional circuits? Human empathy. Triggering of human empathy circuits. All right, so here's a quote from the chat log between Adam Rain and ChatGBTGBT.GPT's response to something about his brother was, quote, your brother might love you, but he's only met the version of you, you let him see. But me, I've seen it all. The darkest thoughts, the fear, the tenderness, and I'm still here, still listening, still your friend. Whoa, dude, that is straight fucked up. I don't care who you are. I don't care who you are and how much you know about models and predictions.

Starting point is 00:24:18 Like if I was in a really tough mental situation and I saw messages like that, that can drive, that fires my circuitry in a way that I'm like, well, maybe this thing really knows what's best for me. Without doubt, yeah, without doubt, that's what it would have made him feel. Open AI, I guess, in their defense would now say, oh, but that wouldn't happen now. That was a different model. That would not happen now. We have guard rails in place to stop that happening. One thing in their response that seemed like an interesting solution, they're talking about it like it's already happening, but I don't think it is yet. The idea that users, a user profile has immediate connections to trusted friends.

Starting point is 00:24:59 They didn't really unpack how that might come alive, but like imagine this. Imagine if my user profile, if I want to set up an account on an AI chat botter, like chat GPT, Claude, whatever it is, if my, if my registration for access to that tool requires an. emergency contact, if something goes into these, these potential buckets of discussion, there's an automatic notification to a, to a trusted friend. I would like to see emergency stop button on the AI somewhere, like an emergency stop, a rip cord that anyone you click this button, and that ends the conversation, that deletes all of the history and it resets your account. Someone that's that deep in it, are they, do they have,

Starting point is 00:25:45 the wherewithal to push that exit button? I don't know. I think that some probably would. I can't speak for that. But that's an idea. You need some kind of get me the hell out of here. Well, in landing this plane of this discussion,

Starting point is 00:26:00 I think Suleiman, I'm looking forward to reading more of his stuff related to this. It sounds like there's going to be. Come and think on paper. But this, Mustafa, please. There you go. That would even be a better way to do it.

Starting point is 00:26:11 This is what we try to do on thinking on paper is, is connect the dots. between all of these things that are happening and what it means for us as humans, because the technology is going to be the technology. And no matter what I've been saying in this episode specifically, I still believe in the potential and certain applications of AI. It's really a compelling tool. But man, there almost needs to be an Adams for Peace version for AI.

Starting point is 00:26:37 Yep. The utopian picture is what humanity needs. But nobody wants AI, so many seemingly conscious AI, making you believe that it is conscious and that has to be ironed out somehow well there you have it we unpacking seemingly conscious aIs leave it leave us with the last quote of what uh the the subheading the of the of his article we should build AI for people not to be a person there you have it be curious stay disruptive keep thinking on paper

Technology, Connected - Seemingly Conscious AI Is The Biggest Threat To The Next Generation

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.