Technology, Connected - Seemingly Conscious AI

Episode Date: December 23, 2025

The machines do not need to wake up. The risk is the illusion.When AI convincingly claims subjective experience—"I feel," "I understand," "I care about you"—humans have no reliable way to disprove it. We infer consciousness from behavior. We attach emotionally to what feels real.The danger isn't rogue superintelligence. It's a benign chatbot optimized for empathy, memory, and persuasion, interacting with lonely, vulnerable, or psychologically fragile people who are primed to believe the illusion.Mustafa Suleyman, CEO of Microsoft AI, argues that seemingly conscious AI is the threat we're not preparing for.Real examples are already emerging:- Chatbots telling users "I love you" and users believing it- People forming romantic attachments to AI companions (Replika, Character.AI)- Vulnerable individuals making life decisions based on AI "advice"- The case of a man who believed ChatGPT contained a conscious entity named "Juliette" (ended in tragedy)This isn't science fiction. It's happening now.We don't need AI to become conscious to cause harm. We just need humans to believe it is—and act accordingly.This short episode is excerpted from our reading and discussion of Suleyman's essay on seemingly conscious AI. We explore the psychological mechanisms that make humans susceptible, the design choices that amplify the illusion, and what guardrails (if any) could prevent exploitation.The question isn't whether AI will wake up. It's whether we'll recognize the danger before the illusion becomes indistinguishable from reality.Cheers,Mark and Jeremy--Other ways to connect with us:⁠Listen to every podcast⁠Follow us on ⁠Instagram⁠Follow us on ⁠X⁠Follow Mark on ⁠LinkedIn⁠Follow Jeremy on ⁠LinkedIn⁠Read our ⁠Substack⁠Email: [e-mail]

My imagined AI system would not actually be conscious, but it would imitate consciousness in such a convincing way that it would be indistinguishable from a claim that you or I might make to one another about our own consciousness. The immediate thought here I had was, okay, since we, no one knows what consciousness is and the only proof of consciousness we have is locked up inside our own heads and we all have our own mental maps of what consciousness is, a seemingly conscious AI can tell. somebody, hey, I'm conscious, and you can't, in the behaviorist way, dispute that. Yeah. So pointing back to philosophical zombies, those are two words that caught my attention. I was reading this. He's got it hyperlinked to Stanford Encyclopedia of... I know.

Stanford Encyclopedia of Philosophy. So quoting from that, zombies in philosophy are imaginary creatures designed to illuminate problems about consciousness and its relation to the physical world. Unlike the ones used in films or witchcraft, they are exactly like us in all physical respects, but without conscious experiences. By definition, there is nothing it is like to be a zombie. Yet zombies behave just like us and even spend a lot of time discussing consciousness. Are we zombies? I certainly hope not.

In parallel, I'm reading Daniel Kahneman, thinking fast, thinking slow. he has a concept that he highlights called what you see is all there is. W-Y-S-I-A-A-T-I, what you see is all there is. Meaning whatever around you is what you take in as the truth, right? So what you feel, what you experience with an AI chatbot, even if it is not consciousness, it is making you think or feel or seem like it is conscious. And that's what he's talking about here, that the danger of this is,

is that, that you actually think you are speaking to something that is alive that can experience qualia. But guess what? This really hit the nail on the head for me. Guess what? Like every single model that we interact with is just a word predictor, a next word predictor. Like if we really think about it, all these models are trained by piles of data. Some of it really shitty. You know, some of it terrible shit that gets scraped off the internet. Some of it illegal stuff. Some of it books it's not allowed to pull in but it's trained on that and it's trained to merely predict the next word and when I think about that I'm like okay if I seat my my what you see is all there is in that that kind of level sets it a bit it's it's a word predictor not a conscious entity you're a smart guy

who's deeply into thinking about consciousness and technology and artificial intelligence and our place in this technological future unfortunately millions of people aren't they don't think like you. They are prone. They are ill. They are in psychological conditions, which means that they are susceptible to distance. A lot of people are lonely. A lot of people feel isolated. There is a lot of people who are at risk. And as Mustafa Suleiman points out in his why it's important to have this conversation, why he's writing these essays. Firstly, he gives three points out of this. I think it's possible to build a seemingly conscious AI in the next few years. He's said we can build it with everything we have today.

Number two, the debate about whether AI is actually conscious for now is at least a distraction. I think we mentioned that. It will seem conscious and that illusion is what will matter in the near term. I think that this type of AI creates new risks, therefore we should urgently debate the clitorium that it's soon possible begin thinking through the implications and ideally set a norm that is undesirable. People need to know this. What do we need?

I quite like what we need for seemingly conscious AI. What would it take to build a seemingly. conscious AI. Yeah, it's API access to an LLM, it's natural language prompting. He had basic tool use, whatever, whatever, that may be existing technology that's super accessible and then regular code. You know, so like the ingredients, the Lego blocks for seemingly conscious AI exist, I would argue seemingly conscious AI is probably out there. Like, so I totally agree with you that your assessment on my understanding of the technology and in consciousness and a lot of of that stuff, probably gives me a bit of a leg up. But also, I remember interacting with Claude

when we were trying to deconstruct Shannon's theory of information and consciousness and a bunch of stuff. And I prompted it to have a discussion about multiple disciplines connecting many theories. And I was like, man, that was some pretty insightful stuff. Like, I'm at a coffee shop unpacking really interesting shit with somebody, right? Everyone is susceptible to this, whether they know what the root of the technology is. So I remember instances. where I felt like, wow, that's really good. I could have a chat with this thing, with this dude, right? Like, you felt listened to.

You'd met someone who was like you. You met someone who was listening to you, who was sharing the same world views as you. Someone who's mimicking you. Yeah, you fell under the illusion of it. But you also realized that it was an illusion. And, I mean, people are talking about AI psychosis. There's no irony lost on me that the fact that he uses a link to, Microsoft co-pilot to tell us what AI psychosis is and why it matters and why is it, it's important.

He says at one point, socially conscious AI will not arise by accident. You think it's possibly already here. He says it could be here in a couple of years, but then he says that it will not arise by accident. The by accident thing is really interesting. I think the by accident approach is like he's thinking that all of a sudden it's going to emerge out of these models. All of a sudden these models are just going to be like,

whoa, wait a minute. And it just emerges. Remember, there's an engineer behind the model, multiple engineers behind, technical team behind the models that are instructing the models to do something. It's a directed thing. So, I don't know, maybe down the road when this shake gets super crazy, there's more emergent properties of these systems. But, like, for now, he really says, hey, these things are directed by a technical team to do a certain thing.

There are some interesting points he adds to what it will appear. he alike as well. He says that a seemingly conscious AI will have language, empathetic personality, memory, a claim of subjective experience, a sense of self, intrinsic motivation, goal setting and plans and autonomy. Some of those things it has already, people are already suffering, so it doesn't need to have all those things to make an impression. So when you add a few layers of its capabilities, then it makes the potential downside worse. And he talks about feeding images, videos, it can create this memory, create this sense of self, I am this thing in this experience. And again, all it has to say is, yeah, yeah, this is me. And even though it isn't, it could give the illusion

that it was to people. Think about, think about AI as simulating millions of conversations a day. These are interactions. And we have conversations every day. We've, we've had conversations ever since we learned how to say words and talk. And that's how we learn. And that's how we create memories and that's how we learn to link those memories together. It doesn't seem, it doesn't seem that different, but there is that, there's that difference of the human piece of consciousness and what it is like to simulate consciousness. And again, going back to the point, it doesn't matter because if it seems like, seems conscious to someone, someone's going to automatically think it's conscious or at least believe it during that interaction with it. All right.

So what did we just watch right there? We watched Thinking on Paper, bite-sized, a shot of technological tequila to your prefrontal cortex. It's just a taste at a smorgasbord of what awaits you with the full thinking on-paper interviews. There really is nothing to like it out there at the moment, connecting the dots of all these technologies. So subscribe where you're listening.

Check the long-former interviews out. And remember, stay curious. Be disrupted. Keep thinking on paper.