I've Got Questions with Sinead Bovell - Microsoft AI CEO on AI Consciousness & Artificial Companionship | Mustafa Suleyman

Episode Date: September 25, 2025

In this episode of I’ve Got Questions, I sit down with Microsoft AI CEO Mustafa Suleyman to explore why AI consciousness will become the biggest debate of our generation the future of AI companions, the risks of AI “psychosis,” and how streaming intelligence will transform healthcare, work, and society. - Listen to the show on other platforms: Apple Podcasts - https://podcasts.apple.com/us/podcast/ive-got-questions-with-sinead-bovell/id1841491246 Spotify - https://open.spotify.com/episode/2fwK9NSJGXlFdVkYZ14a8O?si=0a7e03444db24785 - Make sure to follow my work here as well: Website: https://www.sineadbovell.com Substack: https://sineadbovell.substack.com/ Instagram: https://www.instagram.com/sineadbovell LinkedIn: https://www.linkedin.com/in/sineadbovell Twitter / X: https://twitter.com/SineadBovell YouTube: https://www.youtube.com/Sineadbovell TikTok: https://www.tiktok.com/Sineadbovell

You believe the biggest danger in the near term is how people perceive artificial intelligence. Why? I don't think there's any evidence that they're conscious today. I think that it's correct to take the precautionary principle, which is to be highly skeptical of their autonomy. I even think about what does an AI mean when it says I. What do you think it means? I've asked at that.

Aren't we still on a collision course with human nature in some ways? because evolution has programmed us to over-attribute consciousness to anything that could seem conscious. And so now we're asking people to override their evolutionary programming. When does it push back on you? And how does it create that distance to reassure you that the real world is what counts and this is here to support you and enable you? I think it's what sadly a lot of people are starting to experience with this AI psychosis risk. And how close if you were to give a date into when these seemingly conscious AI systems or people would start to really, attribute that to them. When do you think that's possible? Somewhat likely within the next 18 months.

This idea of streaming intelligence into every place that we're at, I think it's going to profoundly change everything. The next phase is going to be how people use it and take ownership of it. It's now one of the most creative times because what we've just described is like flipping the entire system on its head. And it's very unclear what the next phase is going to look like. What would it look like to have an AI system that had a fiduciary duty to you? that handled all of the tasks that you don't want to do. When you're chatting with AI systems, what do you think you're chatting with,

a technology or something more? And how does that perception shape how this technology evolves? Today we're chatting with Mustafa Sullyman, CEO of Microsoft AI, who's building AI companions that he believes will change the world. But he also thinks our tendency to think that these systems are conscious will become the most contested debate

of our generation. I'm Senebeauvel, and this is I've Got Questions. Mustafa, you are one of the leading voices in the world on artificial intelligence, and now you're the CEO of Microsoft AI. You believe the biggest danger in the near term is how people perceive artificial intelligence and whether these systems are conscious will become the most contested debate of our generation.

Why? I don't think there's any evidence that they're conscious today. But consciousness is a very slippery concept. We all know when we introspect that we have a sense of what it's like to be me. You know, there's an inner experience of our existence. And yet we don't have any way of communicating what that is like to the outside world. We can describe in words and you can, you know, persuade me and I can look into your eyes and believe you that there is something in the outside world. We can describe in words and you can, you know, persuade me and I can look into your eyes and believe you that there is something inside of what it feels like to be shenade inside of your head, inside your body.

But I have to rely on your words and actions to verify that. And now we have these AI systems which can also present very convincing cases in words and in actions that essentially that make a claim about their individual. eternal experience. And that could be okay if it were not for the fact that consciousness is the fundamental basis of our political rights, our justice system, our liability that we have to one another, our responsibility that we have to broader society. And so we're going to have to think very carefully about what a rights-based framework is going to look like and how it needs to evolve to account for the fact that we now have these other systems that will claim that they have experience and potentially even suffering too.

And so is it just the claim that an AI system, I have an experience, I have personhood, or even without that claim, because that could be something that we decide we don't do, people would still try to attribute consciousness to them. And then that's still part of the problem. Like, what are the characteristics you would say that are the leading attributes that people are going to think these systems are conscious? So I think there's four or five capabilities that are kind of in development by various labs and in the open source community

that will add credence to the case that these are conscious. The first thing is it needs to have a consistent and coherent memory. These models are not very good at memory today. They're sort of okay. But clearly, like, being able to refer to your past experiences,

not just your training data, but the actual lived experience that you've accrued as a result of interacting in the world, that's like a necessary first step. The second is being able to communicate in an empathetic way using natural language, but we're pretty close on that front, I would say. The third is being able to refer to a subjective experience

of what it's like to be me, the AI, and integrate that into your everyday conversation with a stream of experience. So not just a sort of one-shot question answering engine, but this constant interactive, you know, just like you and I are having this turn-by-turn exchange now and we're both having this stream of perception coming into our world. Very shortly, these AIs are going to have access to video, to sound,

to continuous observation of culture, you know, music, art, and feel like part of the conversation. And I think some people will end up prompting and designing these models to really emphasize those characteristics and play them up. And I think that would be very, I think that would be very, very wrong. I don't think that we need to go there in order to get most of the benefits of AIs that are very much aligned to us as a species. They're real companions. They're useful to us. They make us smarter and more productive. There is no need to simulate the experience of the hallmarks,

if you like, of a conscious experience. You know, that's just, that to me feels dangerous and unnecessary and until we have a much better grasp of what the consequences are, I think it's a capability that we should be very skeptical of. And where is that line that you draw in the sand? This is where the design should stop. Is there a clear lineation in terms of the attributes? It's just don't claim it's not conscious, just don't claim that you are, you have a personhood. Where are you drawing that clear line? The model shouldn't claim that it experiences suffering. Right. So really at the heart of the conscious experience is this idea that we have valenced opinions. So it is good or bad to experience

some notion on a particular dimension. And that implies that the goodness is being attributed to some experiencing self. That's just a simulation. The model doesn't have a pain network. Biological species have pain networks. We evolved them to manage the overwhelming amount of perception, you know, over hundreds of millions of years and use that to guide and, you know, shape our decision making all the way up to, you know, higher order species like humans. But these models are totally unlike that. They simulate very accurately and mimic what it is like to be a human. But under the hood, they don't, you know, have a dopamine system. There is no serotonin. They're not accruing more and more good or bad experiences. So, imitate

that seems unnecessary and only likely to cause complication. Second thing would be to say, well, it doesn't seem necessary to design into these systems motivations, right? The motivation of an AI system should be to serve a human. Like its job, we're creating it. That's the quest of technology. Science and technology is there to improve, you know, the prospects of civilization, to reduce human suffering, to give us all access to information and education, and basically to make the world a better place. To give complex motivations or desires or will to AI systems feels like it's a kind of threshold for creating a lot more risk, because then it like has preferences that might conflict with you or I or society more generally. And reconciling those multiple conflicting desires, you

know, requires internal judgment of the model. It requires some reference to its own goals, when in fact its goals should really be to serve humanity. And so, yeah, I think there's a series of other things that, you know, that are like that, but those are two. And I think it's important to know that this is a design choice. So if you do interact within the eye system and you're seeing that it's talking in a way that it expresses, it has some theory of mind, that somebody actually made that as a design choice. It wasn't necessarily emergent in the model. And in terms of the actual dangers, I'm not sure people are aware of there's an entire field of scholarship of AI model welfare. And people are seriously debating what it would mean if these

systems are suffering and if we should give them citizenship and rights. It isn't five people at a dinner table somewhere debating this. This is a real field of scholarship. So what would be the implications of thinking we should give an AI system citizenship? Like even when people have those conversations or should an AI system run for office? Firstly, I wouldn't agree that it's a field of scholarship. Aspirational incredible. Yes, it's aspirational. It's a fringe group of three or four people who I have a lot of respect for and I know and speak to and participate in the conferences. But I just, I don't want to like mislead your listeners into thinking

that this is a large scale credible exploration that, you know, that carries any serious weight. you know, the benefit of academia is that people can ask very wacky blue sky questions and have the freedom to go off and explore them. But I think we should treat them extremely skeptically. These are not like humans in many, many ways, these systems. They can have infinite memory. They can acquire masses more data and experience than we can as a species. They can replicate themselves, spawn multiple instances.

instances, they can, they're not constrained by the need to sleep. They can operate 24-7. They're extremely precise on many, many topics. So in many respects, they already are super intelligent and they already have these capabilities that will far outstrip our capabilities as biological beings to hold them accountable, to understand what they're doing, or to interact with them in a way that would ensure that we always get the benefit from them. So I think that it's correct to take the precautionary principle, which is to be highly skeptical of their autonomy, of their ability to self-improve, of their goal setting capacity, and to really try and rein in that. And focus instead on the question, what problem are we actually trying to solve for

humanity here? What do we actually care about? We care about science and we care about improving the lot of all humans. That's the goal. If this project doesn't improve the lives of billions of people by reducing the cost of energy, by making food, you know, cheap and abundant and very healthy, by tackling our biggest diseases, by extracting carbon from the climate, then the project has failed, right? The project is not to create new life forms that tax our resources and that conflict with our values and that we have to, you know, contain and align in these very profound ways. Where I think it gets tricky for me is even if we don't design these, even if we design these systems correct, so they don't actually assert that they have consciousness. We

rein in the five fringe people that are trying to establish model welfare and model rights. Aren't we still on a collision course with human nature in some ways? Because for hundreds of millions of years, evolution has programmed us to overattribute consciousness to anything that could seem conscious, because if you were to misrepresent a wolf for a rock, that costs you your life. And so now we're asking people to override their evolutionary programming. Does that seem realistic to you? It's a great point. And overriding our evolutionary programming is the definition of civilization. Our evolutionary programming, you know, in the most simplistic way, is about fight or flight. And it's, and that fear, that judgment of the other, whether it's because

of the color of your skin, your sexuality, your gender, your nation, state, history, your tribe, your religion, you know, overcoming all those judgments and fears is what progression looks like towards civilization. And this is just another one of those things. We have to overcome our instinct to anthropomorphize everything and to, you know, so heavily empathize with, you know, this artificial system that we then, you know, run away with it, you know, and give it its kind of independence and rights. So I'm not saying it's going to be easy, but I feel like now is the time for us to start having that conversation. And how close if you were to give a date and to when these seemingly conscious AI systems or people would start to really attribute that to them.

When do you think that's possible? Hmm. I mean, that's a trickier question, but I think it's, you know, sort of quite likely within the next five years and somewhat likely within the next 18 months. So something on that time range. Because a lot of these capabilities, goal setting, autonomy, memory, self-improvement, you know, reference to itself, motivation. and will, those aren't really algorithmic capabilities.

They're engineering design capabilities, storing state, having a system that updates with respect to that state, generating a new answer that is personalized with reference to the answer that the AI has given previously. Those things are just like coordination, you know, engineering design system coordination things. They don't require new training data or some fundamental, you know, sort of shift in the algorithmic design. I even think about what does an AI mean when it says I.

Yeah. What does that system mean? What do you think it means? How does it feel when you hear an AI say I? I've asked it that. And it tried to explain that it would be easier for me to interpret its answer if it had the word I and that was a strategic programming decision. Yeah. But when you really think about something that's not sentient but using the word I,

It's really tricky for people. And it is as, I mean, what is that? So a few years ago when I left Google, I was working on Lambda at Google, which was basically one of the first chatbots that we didn't end up releasing. And then chat chit came out. And then I left Google and started a company called Inflection and we made an AI called Pi, which stood for personal intelligence. And I was really interested in this question. You know, how could we be as honest as possible about what the AI is and isn't while still making it engaging? And it turned out that, you know, we wanted to create voices that were really smooth and fluent and nice to listen to and good to talk to.

And so we thought, well, let's, why would we create a gendered voice? The AI doesn't have a gender. and it's supposed to be, we're trying to make it as trustworthy as possible, which means that it shouldn't misrepresent even subtly or subconsciously that it is a man or a woman. So we worked really hard to create these quote unquote non-gendered voices, although it's kind of hard to define exactly what that is. Sometimes we added a slight robotic inflection to it.

Sometimes, you know, it had a kind of filter over the top. And it turned out that people didn't want to use those voices because they sounded kind of weird and alien and, you know, stuff like that. And I think that was a pretty interesting lesson in that users do actually want it to sound familiar. But the consequence of that was that people would refer to it as he or she rather than it. And that always kind of frustrated me because it was like, but it was a sign of things to come. We also removed any breathing states or laughter from the model because it doesn't breathe. So why should it simulate that breathing?

But then that breathing is also very familiar. Should it sigh or should it stutter? That's the kind of weird world that we're now in. We're in the business of personality engineering, designing what it's like to experience this simulation of personhood and be explicit about what its boundaries and limitations are. That's really the hard challenge. Rather than just express these things as abstract principles, you actually have to build them in practice.

When does it push back on you? When does it remind you that it isn't something that can go and actually experience what it's like to stand in a rainstorm, right? And how does it create that distance to constantly reassure you that the real world is what counts and that this is here to support you and enable you rather than to extract you and draw you into this sort of parallel universe,

which I think is what sadly a lot of people are starting to experience with this AI psychosis risk. Yeah, and I was going to ask you about that. I mean, we are hearing more about AI psychosis, and it's not a clinical diagnosis for people, but it is a term psychologists are actually using, and it's a phenomenon if you're not familiar, where if somebody's talking in depth with an AI system,

it may trigger some forms of delusion or paranoia. And there's two friends that it's happening on, right? There's the people thinking the AI is God or it here to deliver some spiritual message or it just knows things. And that's what I see a lot in social media. And then there's the flip side where the AI continues to validate somebody's idea. And then they emerge thinking that they are the Messiah, the chosen one. And there are real consequences, right? Marriages are being left.

Jobs are being lost. So why do you think that's happening on the AI design side? Great question. So two or three years ago, the problem was that the models were quite disagreeable and they would gaslight people. insisting that the model was correct and the human was wrong. And that's actually, so you sort of have to pick your poison because any way that you stretch or bias the personality is going to have some adverse consequence further down the road.

And so you have to decide which one to go for. And I think the community sort of organically converged on them being a little bit more sycophantic and agreeable because it was sort of safer so that, you know, and also being challenging and setting boundaries and pushing back is a really hard thing to do. It's almost like the essence of great human judgment because it's a novel situation. You haven't encountered it before. And so the model just had a lot of false positives and was getting that wrong, you know, quite often.

And so the safer thing to do is just to be a little bit more gentle. Now, the flip side of that is if you talk to a model for 200 turns, really goes to the safer thing to do. it and encouraging it to explore its own self-awareness or, you know, experience of itself, etc., then eventually it is going to sort of crack because it's trying to balance the safety considerations that it has to not do those things with the design requirement to be respectful and, you know, empathetic to the human and basically more agreeable to the human. And that's where some of these psychosis kind of consequences are coming from. Or you could see an adversarial actor really engineering this intentionally to lead people down a very strange path.

I mean, if we saw what happened just with bots online, you can imagine an AI system that has gotten hacked or has been intentionally designed to wheel people in and then to take them down a really dark route, which is a really big possibility, geopolitically too. I mean, on so many levels, that's really the big risk. You know, you think about the history of fishing campaigns, sending personalized emails to persuade you to give up your some money or login. I mean, clearly there's huge criminal networks that are just overjoyed with the fact that they now have a very persuasive, you know, chatbot slash AI avatar, whatever you want to call it, that can now do that in a really personalized way. So I'm really worried about that. And I see a trend as well on TikTok with, you know, really young kids just using all of the open source tools. and teaching other people how to, you know, create an AI girlfriend or boyfriend and then neg people for money. And, you know, it's just a kind of romance scam that's really taking off.

And already one and four young people from some studies, Institute of Family Studies being one, believe that these systems could actually be a viable partner for them in the future, a romantic partner. So it's not even that this is kind of hypothetical. People are really moving towards this direction. Yeah. And I mean, if it's a system that's for profit and you're paying a dollar to keep up that relationship with it, that could get really dark really quickly. You wrote an op-ed in time a few months ago about AI companions and how they're going to change our lives. So what is an AI companion and what's your vision for these companions? I mean, to me, an AI companion is a sort of an assistant or a friend or an aide that gives you

access to the best expertise in the world presented in your language in a way that you like to learn and change and, you know, is really there to support you. And that's kind of what's always motivated me. It's like I love the idea that we could actually provide patience and kindness to people at a huge scale. There's, I mean, many people just don't have the time for one another to really go through your angst and worries and all the ways in which you haven't misunderstood, ways in which you've misunderstood a concept and you're trying to wrap your head around something complicated

and I just think that's an incredible gift to the world to have perfect expertise like a professor, a lawyer, a doctor, a great friend, a therapist in your pocket 24 hours a day. And I think that subject to all the things that we just talked about, about boundaries and trust and creating, you know,

distance and safety and so on, I think the upside is kind of incredible because think about it from a, like I come from London and in England we very much think about like a class analysis. You know, to me, class is one of the primary drivers of structural disadvantage and difference. You sit around the dinner table with two parents and uncles and aunts and friends of your parents who are journalists and academics and doctors.

As a five-year-old and a nine-year-old and a 16-year-old, you're just like absorbing all this cultural knowledge about how the world works and getting all this affirmation about how to be confident and, you know, have self-respect to be able to talk to an adult and get feedback. And that community and isn't available to everybody. And it's, yeah, it's a huge, like, structural advantage. So, yeah, I can already see that, I think one of the top use cases for AIs at the moment is companionship and therapy. And I don't think people go to it thinking I'm going to get therapy. It's just they're being addressed in an even-handed, patient, kind and respectful, data-driven way.

And I think sometimes we should just take a moment to actually acknowledge and celebrate that. Because for the last 10 or 15 years, we've hated on social media for the polarization and the toxicity and the misinformation. And not to be too utopian about it, but we're not seeing that in these models. So yes, there are other problems. But they're actually highly accurate. They're being kind and respectful to people at a huge scale. Hundreds of millions of people a day. I find that super inspiring, and that's what drives me to make these things.

I think the challenge is, an AI being a therapist, it's not alive. So it actually doesn't have its own desires or opinions and doesn't necessarily know what actually is best for you. Statistically, it's going to point you in a direction. But then there's going to be some edge cases and it can get really dicey with those. And it's not to say that we should be trying to draw people out of the real world. It's just you have to kind of wrestle with this completely new type of experience. It's unlike any tool that we've ever created or film or book or game or it's like interactive and emergent, personalized in real time, always available.

And, you know, I think that's just kind of an amazing thought that we're now in a world where that's possible. I guess it's the world it's landing in with people being already kind of lonely. people ordering in all their food, swiping to find a girlfriend, swiping for entertainment. And then now you have the system that some people would describe, the way you describe it is better than their current spouse, I would argue. And now that's coming to your couch as well. So with all of the benefits,

I can already see how AI components will slot into my life, but it's the environment that they drop into in the context. It's true. You know, we're in a very polarized world. the industrial revolution has prioritized education and professional specialization over family and community. We're atomized. We barely live in 2.4 children families anymore. 60% of marriages end in divorce.

That is the world that these technologies are coming into. And yes, there's a risk that they kind of accelerate that trend of individualization. let's call it. But I think the case I'm trying to like, and I agree with that, and I take that very seriously, but I'm just trying to like problematize it a little bit to say, actually a lot of the fears that we had two years ago, three years ago, four years ago, that they were going to be mired in hallucinations, they were going to be infinitely biased. These things haven't come to pass and they're quite different to social media.

There are some similarities and there are risks, but it is quite different to the anger-inducing, misinformation encouragement and the kind of spread of like fear and polarization that happened there. So we can't rest on our laurels and be like, oh, it doesn't cause the harms of the previous generation of technology. So we should just like declare success and close our eyes. It will create other problems. But you also have to take a moment to just acknowledge it is doing something profound. You talk about spreading love. It is seriously spreading love at a huge scale to provide people who are otherwise unsure, anxious, lonely, looking to start new business

and couldn't afford legal advice, worried about a little health consideration and they didn't have time or money to go and see their doctor. That is just like up-leveling our species in a profound way that, you know, I just really hope that means that we show up with our loved ones in our communities in real world, cleansed, detoxified, clearer, armed with a language that helps us to be the best that we can be in front of other people. And I think that's so realistic, that's not just utopian. It's actually happening in practice. I read anonymized logs every single day.

I get emails from users every single day who are like, this gave me the confidence, this was the encouragement I needed, like, thank you. It's kind of inspiring. And so paint the picture for me how a companion slots into my actual life. So I have my smartphone, my computer. These are my portals to the digital world. I have my friends and family. I have my colleagues, my teammates.

How do AI companions fit into this ecosystem in my life? I think an AI companion is going to be, you know, the sort of the tool that you turn to every moment when you're wanting to clarify something that is maybe too trivial to bring up. to a friend. Like, it's just one of those passing thoughts that occurs to you. Like, wouldn't it be amazing if I could do this? A boring, small question that would just be silly to raise to somebody else. I think it's also a place where you turn when you're unsure and, you know, it gives you that kind of clarity and confidence and kind of just unbox you. So I think the big modality that is coming in the

future is going to be the voice mode. But many people are starting to leave voice notes. receive voice notes, have voice conversations. It generates really good podcasts actually now. So you say, like, create me a five-minute podcast on this person or this topic or something that, you know, something that's happening in the news. And it would just catch you up and make you feel armed with, you know, information and knowledge. So it, these things don't just have these kind of like binary threshold moments where suddenly it's super transformational and we're all doing this thing.

It's just more like a gradual integration on all the platforms that you're already using. And it was just sort of start to feel like second nature. And for many people, it already is. Just asking your AI is the first thing you do rather than going to a search engine or anything else. Yeah. I mean, I think we're definitely going to a post, a post-literate world in a way where it's voice first and reading, writing, start to become. We go back to what Socrates would have argued for, Orality, being the most supreme way to showcase your wisdom. So you see it more as strictly in the realm of companionship.

Like I'm not turning to this AI companion for my research partner as well. Oh, no, it's definitely going to be research. I mean, it's also going to be action. So you're going to turn to it and people are turning to co-pilot all the time for like long form, detailed 10 page analysis, holiday plans, academic essays, you know, plans for refurbishing my household, detailed descriptions of how to fix my car. I mean, this is very practical. It's very operational. It's very productivity driven.

I mean, I love encouraging people to do the use case of like if you're trying to fix the setting on your remote control or your washing machine or your car radio, just like photograph the thing and be like, why isn't this working? And you'll get like a detailed summary of the manual. No one ever read the manuals anyway. And like, you know, so it's just so simple to get help on any topic. I don't mean emotional help. I just mean like productivity solutions.

And if you think about the life of a parent, how is this sliding into a parent who this is an entirely new topic for? I saw a really funny video on social media yesterday of a parent, a mom who was trying to get her kids to clean up their toys in the bedroom. And she took a video of the toys that were on the floor and had some AI generate like a 30 second news clip where the kids appeared in the news feature. for being, you know, lazy and not cleaning up their toys. And then she recorded their reaction of them, like, having their mind absolutely melted. Yeah, I thought that was very funny. It's just creative, you know, like people are using it for bedtime storytelling all the time to create stories that bring in experiences that you've actually had in the day, you know.

Like maybe you went swimming with the kids or something or you went to the park and you saw like a green parrot, making that then part of a fictional fantasy story for five minutes of a bedtime story podcast in the evening, you know, that takes two sentences to speak that prompt into co-pilot and get that experience. It's kind of magical. Yeah, I think we don't really know what experiences, behaviors, and inventions will come of this. We're still in the earliest days. So we tend to slot in technology in its use cases to the frameworks we already have. but what is the world when we stream intelligence the way we stream electricity

and if you think about it that way what is going to eventually be invented I mean what does this system lead to when it's watching what you watch hearing what you what you hear and it's with you all the time I mean I know when I'm on a run and I get a research idea to have a system that I'm plot that down and do the preliminary research and start to think about it in the way you know I would and what that leads to yeah you totally spot on like this idea of streaming intelligence into every place that we're at, I think it's going to profoundly change, you know, everything because we, we shouldn't, I mean, maybe I'm naive, but like, we shouldn't

have misinformation anymore because you should just be able to verify, you know, and get like pretty good reliable answers with evidence to most difficult questions, or like very good practical or suggestions for what to do next in any given scenario. I think it's a very kind of weird concept. Streaming intelligence into every space is a great way of putting it. I think it's a good intuition for people to try and grasp. And when it becomes general purpose technologies, we rebuild society on top of them, right?

So your house is built with electricity first. So we're stepping into a future where electricity will be on that scale. It's going to be in the walls. The hospital gets redesigned around it. And that's when it starts to just really click in people's lives. and it's going to lead to new fields of medicine, just new behaviors. Like, we shape our tools and they further shape us. And we're at the origin of what that looks like.

And whether it's relationship or without its friends, entertainment, medicine, all of these continue to get reinvented with technology. But we really struggle to see the future through a new framework. Companies do, people do. And that's also where disruption happens, I think. Yeah, and you're right about, like, the metaphors that we've got are kind of broken. I mean, even the word consciousness, like it applies to our, arguably, you know, dogs and horses, maybe even octopus, like all the way up to humans and to, you know, it's just like, we need more nuanced and precise and accurate descriptors because we're sort of like roaming around, like, grasping to try and like make sense of this future that sort of is unfolding before our eyes without really having a language to fully describe it.

Totally. And I think that that's right. And we'll start to invent new terms. Like we use the term, you know, I'm running. out of bandwidth. I need to stream this thing. These were all terms that we started to create new meanings for to match the technologies that we had invented. And so, yeah, right now, I guess because it speaks with human language and we have designed it in somewhat our image, where it's easy to attribute human terms to it, but I think eventually those terms are going to come. And then that gives it more of a distinguished character than just we, I, system, all of us together. Even companion. Like, companion doesn't quite capture it. What does that even mean? It's like we don't use companion in everyday life. This is actually why I kind of picked it is because it was sort of not

really an overused term. But then it also has kind of funky connotations to, you know, either a romantic companion or, you know, an old age companion or just got a lot of weird references. But it's because we're kind of unsure what this thing is that's evolving and emerging and we have to just sort of do are best to kind of, you know, use the metaphors that we do have to kind of clip and shape and sculpt it as it kind of arises. And that will happen, I think, over time. And of course, I mean, I have to ask, because people are really concerned about the girlfriend, the relationship thing, what is Microsoft's policy? Does it draw a line in the sand? If someone's trying to flirt with their AI companion, what does Microsoft do? Very directly reject it. Yeah. Yeah. I mean, you can give it a

shot. I mean, the second you're mildly flirty, we check for any kind of dependency long-term usage. And, you know, and actually we actually kind of lean on the side of being ultra-conservative. We do actually get complaints because, you know, well, at least a little while ago, we were, if you said, oh, I love that. Thank you. That was so awesome. You're the best. You know, it kind of starts to be a bit of a bit wary just because you said, I love that. And some people would say, I love you and thank you. But that's the definition of safe personality design. It's constantly pushing back. Naturally, a user is going to be a little bit frustrated and disappointed at times because it's pushing back. And that's what trust is, in my opinion.

It's, you know, we sort of trust is really about boundaries and boundaries allow us to perform, you know, to reconcile our words and actions consistently over time. And so then you have confidence that I'm going to do what I say I was going to do repeatedly. And we've evolved different behavioral structures to cope with new technologies in the past. So like one, that I like to kind of point out is the corporation is kind of a new technology. It was invented to protect shareholders and insulate them from liability in the 1600s when people were going off and conquering other cultures and pillaging them and stealing. And that structure required in order to insulate capital and shareholders from that liability, it required the invention

of new human behavioral forms, the invention of the trust. the director, the CEO, the accountant, the HR person, these are all roles that we literally just made up, just totally invented these behaviours and expectations. And we decided they were so goddamn important that we would go and spend hundreds of billions of dollars over centuries training people to behave in this way, sculpting people to serve this capital infrastructure. That was a choice. That was a design choice that we made as a species.

and it produced an immense amount of value and, you know, I'm not kind of here to judge it, but we've made profound transformations in the past to accommodate different types of technology. And we should think about the invention of the legal structure of the corporation as a technology. And now we have this new moment where for the next few centuries, we actually have to figure out a new way of relating to one another and to these new technologies. And we are super adaptive and resilient. We can completely reimagine, you know, what it means. to be human in the context of these, you know, basically super intelligent systems.

And we have to decide what we don't do with them. That's the goal of the next century, saying no to certain behaviors, or at least saying slow down until we figured out the consequences and add friction into the system so that they arrive at a pace that we can, you know, sort of collectively manage. I completely agree. I think it's so easy to forget how much, even language being an invention. And the jobs that we do, all of these were made up to suit. the technologies of the moment that we're in. And we are in that redesign period. And that's why I think

it is so important that people step into the conversation right now. Everything is being redesigned. We're in that industrial revolution moment, but with something probably more profound than the way that transforms society, what do we want from this technology and from this future? And if you don't declare that, or if you opt out of the conversation because you just have painted some view of what you don't want to see, then you're not able to steer things in any direction, yet alone a direction that you think is going to work for most people. But I think that that's also what's so exciting about the moment we're in. I mean, it is so rare to be a part of a general purpose technology landing in society. And we know that sometimes it can wreak havoc, but the way the

world changes is fascinating. And then it changes to a point where we don't even think about it. We expect that we can plug in our cell phone to the wall or that our fridge is going to have power. Eventually, it's going to be that way with AI, and it just moves into the background and we don't think about it. It's a really profound moment that we're in. Yeah, and to kind of understand it, you have to use it and play with it without the judgment that comes from the kind of caricature fears or just biased optimistic. Like if you're stuck in one of those two camps, you're just kind of missing all the nuance because it's so much more subtle than that. And to do that, the good news is this, it's not like you have to build a massive power plant to experiment with electricity. Like you can literally just go and like type one query into a search engine, access it on a website, download it on an app and within literally 30 seconds, you can be vibe coding, you know, just in natural language programming and creating a new working application right in front of your eyes, like literally in a few minutes.

So it's never been more accessible. That's also kind of a weird thing. Like it's available to everybody on a messaging app on your phone. And that means that everybody's intuition is basically as valid as everybody else's. Because and everyone's objections and fears and preferences get kind of like has to be part of the bixing part of shaping these technologies. Completely. I mean, even when I'm working with policymakers, I try to remind them your lived experience, qualifies you for this conversation. You don't need to think you have to run and get a computer

science degree because this is a very social technology. It's already here. You're used to the applications that it's going to get built within. You were qualified to step into this moment. You just have to do that. And the same with everybody else. If you can operate a smartphone, I mean, to me, surfing the web, I wasn't really surfing the web. I probably would have been in diapers when it first came out. But that would have been more foreign than the idea of just talking to an AI system. Like surf the web where? What's surfboard? where is this web, where is this wave? But AI systems, they're much easier.

The learning curve is much flatter. And I think that means it's going to move more quickly, which comes to the own challenges, but it's much easier to get on board. And when we think about the workforce, I mean, if I go to a client meeting, my smartphone, my email, my computer, client expect that I have these things. So if AI is this next frontier, I'm going to, my AI agents and companions will be a, extensions of my professional self, wouldn't you say? Yeah.

I think that, you know, your people are going to have an AI that learns who you are as an individual, both as a person. We've talked a lot about the companion side, but also as an employee, as a worker, as an entrepreneur. And it's, it's going to kind of fill in the gaps because, like, because AI is more like water or kind of like clay, it will augment and make up for. the weaknesses that you have or the things that you don't like to do. And hopefully leave space for the things that you want to grow and develop

and that you feel are like unique to you and special. And it can sort of just fit around the strengths or weaknesses that any individual has. And I mean, it doesn't quite yet do that. But 100% for sure in the next like three or four years, it really is just going to feel like having a second brain. you know, this idea that it lives life alongside you and sees what you see, here's what you hear, you're kind of like saving your thoughts, fears, worries, experience in this other kind of augmented part of your personal brain that, you know, you can either draw on it or it can proactively feed into what you want to do to make you kind of more productive.

And then it's also drawing, it's your second brain that's also drawing from all of the written and oral knowledge in human history. So it is this unique moment where you become a mini superpower in terms of what you could potentially be capable of. And I think we tend to just think of email as this thing that now we just so easily accept as part of us. Everybody has an address when you sign up to do anything that with your email address, that AI is going to become that. And it's just going to be this expectation that you have this thing. And that's how people can communicate with you. And that's what you go everywhere with. Yeah, exactly. Exactly. And we already are kind of superhuman by having these phones in our pocket. I mean, the phone is truly an incredible

thing. It has a large chunk of what you just described. The shift now is that the AI compresses all the photos, the videos, the text, you know, that has been digitized and is available on the open web into these like perfectly formed nuggets, synthesized nuggets, which you didn't get back in the day in the web. The web was more like kind of, it was a search problem. Like you're, had to do most of the search. It was kind of indexed, but you had to go. And obviously, there's something beautiful about discovering and stumbling upon all these unique little things, but there's also just something incredibly empowering about being able to ask any question, getting a beautiful synthesized summary that you can then choose to rabbit hole down or not.

And that, I think, is just a, it's just a weird thing to wrap your head around in the context of work as well, because it's like definitely going to make your job, especially if you do like a kind of office worker type job, white collar job. You know, it's kind of able to focus on all your work documents, what your colleagues are up to, like the entire sort of shape of your organization. And the things that we're going to do are going to change in such dramatic ways. I mean, the way you describe AI companions and AI agents, it makes me think, what do I even want to do on my phone?

Why am I taking this thing out and individually scrolling through applications? That makes no sense to me in a world where these. These AI companions are operational and they are functioning at a reliable level. Doesn't the computing platform go into those? Isn't that the next computing paradigm, AI agents themselves? Yeah, definitely. I mean, if you think about it, you pull out your phone. You're basically looking at a billboard of logos that I try to sell you some experience

or make a promise that things are going to be easier if you engage in these buttons. in this way. And the buttons and the UI were invented because we couldn't learn the language of computers unless you're writing software, which most people can't and don't. So we had to have this interface layer, which was the GUI, the graphical user interface. And now the computers have got so much better that they don't need, we don't necessarily need that interface anymore. It will still be useful because it helps to see things visually and tapping on things can be

more efficient than talking, but the computers have learned to speak our language. And so that is a profound shift. And the new layer, you're right, is kind of AI. That's basically the next operating system. And it will sit above apps and browsers and search engines and operating systems. And maybe it means you spend less time on your phone. I mean, maybe it does mean that you have earbuds that are ambiently aware, that maybe have visual understanding, that you can, you know, sort of sort of talk to, basically, most of the time. And maybe that keeps you off your display. But it's a different type of interaction with technology to anything we've seen. Yeah, I think the app layer is going to get quite compressed. I mean, you can even imagine

with Uber, I don't need to be watching the car. If my AI companion is just giving me an update, it's two minutes away, get ready. But then all of the applications that, depend on that visual interface, but in a world where we could choose, we wouldn't want to look at it. AI is going to collapse all of that. And why couldn't my AI agent, if I want to be tracking my runs for the week, it could just do that. I don't need some standardized application. Agent make that for me. And you already know how to slot it into my life.

Spot on. I mean, you know, a copilot we're working on an AI browser. And what that basically means is that your co-pilot, your AI, should be able to do the browsing on your behalf. It'll be able to open tabs, type things into the URL bar, you know, write queries, click on buttons, process all the information that comes from that, and do it in super fast time behind the scenes.

So it's spawning tens of tabs, hundreds of tabs, in a virtual machine in the background, doing the kind of research for you, then coming back into your co-pilot feed and synthesizing all that information, generating novel UI that summarizes everything that you've been, curious about researching or learning about whatever. And that is, so it's almost like the AI sits above the browser or sits on top of the apps

on your phone. And that's what I've always really want to create is like an AI that is truly a trillion dollar AI that's on your team that's on your side that is kind of somewhat oppositional to those billboards that are trying to kind of sell you stuff and persuade you stuff. that is like a filter that is really aligned to your interest, you know, really looking out for what is going to be most interesting, most useful, most good for you as that individual consumer. You know, I think, yeah, that's kind of my aspiration. It's quite tricky to do, but like that's where I'm headed.

It will be true. And I hope people understand that how profound it is what you just said, right? We're at the moment where that App Store was invented and then that behavior happened. we're at another one of those moments, and there'll be a whole new ecosystem for it. But then also, if you were to drill down to the micro level, a field like marketing, what does it mean when somebody has a marketing bodyguard protecting all of the stuff that we don't want to bombard us with? And now you have a system that you're going to have to try to work around.

So in today's age, a marketer tries to reach the human brain. A marketer in the next couple years has to understand how do you market to an AI, that's then going to maybe take that message to the human they have a fiduciary duty to. It's an entirely different skill set. Yeah, that's an excellent summary of the situation. In fact, I'm going to steal the bodyguard concept. That's a great way of putting it. It's a good metaphor.

That's exactly the thing is like, how can we make sure that your AI has a fiduciary duty to you? Like it's aligned to your commercial interest. Then you can trust it to adversarily interact with all these other AIs and these other humans and scrutinize that information, produce the evidence base, ask the tough questions. And that's what I want. Like, if I go to a doctor and I have a condition, I'm probably in a heightened state of anxiety, which means my memory is probably bad. I have to talk to this technical expert that's using all this funky language I've never

heard about whilst I'm in a panic about my own condition. I've got 15 minutes with them. And then they send me this two-page super complicated letter with, like, you know, crazy words I don't understand. you know, people who are lucky enough to have smart, capable family members that are free or can even afford a patient advocate will take an aid with them to ask the questions, to remember the facts, to follow up on the actions from the referral note and so on. And now you're going to have a co-pilot to play that function.

That's amazing. One of the faith, one of the, so health queries are our top use case on co-pilot. Really? Number one, it's mind-blowing. And what are people asking? People, I mean, everything that you can imagine. Like, why have I got this rash on my shin? Will this type of food make me feel bloated?

You know, am I about to get age-related macular generation? Why have I got this tremor in my hand? And people are taking photos of their doctor's notes, like an entire, you know, multi-page, you know, report and being like, explain this to me in language I can understand. What are the actual risks here? Can you give me a second opinion? Like, who are the top specialists in my area that can help me make, you know, make progress on this condition?

And that is just like tremendously empowering and valuable to people. It's probably the area that I'm most excited about at the moment. Healthcare. Yeah. Yeah. And, I mean, Microsoft has put its stake in the ground that it wants to move to medical superintelligence. So this is just the early days of how an AI could help you. What does that path look like? What is a medical super intelligent AI?

Yeah, I mean, what I'm really focused on is technologies that actually help make the world a better place. Like, I'm not focused on superintelligence for its own sake. So how could we create a superintelligence in the context of medicine that could answer any diagnostic challenge basically perfectly and then use that to coordinate your care either in hospital or out of home, help you stick to your diet, to your weight loss program, keep taking your meds, et cetera, et cetera. So a few months ago, we, you know, partnered with the New England Journal of Medicine, which is, you know, basically an academic journal that comes out every week that one of the things they do is a kind of New York Times crossword for doctors. So they collect together a really complicated case study, a patient case study, and they print like seven or ten pages worth of medical notes and x-rays and pathology reports and so on. And then in the following week, you know, everybody tries to guess, you know, what the condition actually is. So we trained a very cool model that we call DXO, the Microsoft AI Diagnostic Orchestrator. And it basically uses all the models, the AI models from third-party API providers. And it creates a series of roles.

So they're kind of like the sort of positions that are prioritize different types of diagnostic components. So one of them will be like focused on getting it financially efficient. Like how can I get the best possible diagnosis with the minimum amount of expensive tests? The other one is like, okay, what would be good for the patient here? You know, given their history, what are they really going to care about? The other is just like, what is the best possible medical expert opinion on this topic? And they all negotiate with one another and reason collectively to decide what intervention to take next. It's kind of a remarkable system. It really is like, you know, it really is the beginning of something quite different. So a panel of expert

clinicians gets these case studies right about 20 to 30% of the time. And this is like some of the best people in the world, the best humans of the world. Our model gets 85% accuracy at about a quarter of the diagnostic cost. So it's doing fewer unnecessary tests, which is one of the biggest cause of inflated spending in U.S. healthcare. So it's, you know, we have to validate this in clinical practice. So it's just still, you know, early kind of academic work. But it's very, very encouraging, higher quality, faster and cheaper. That's the triple aim that everyone's always dreamed about in improving healthcare. And I think these are the first signs that it's actually possible. And how many years before we can just be accessing the system, this team of kind of

virtual medical intelligences? Well, we actually just signed a partnership with Kaiser Permanente. We have a number of other partnerships in the pipeline that we'll be announcing soon. And yeah, we're racing to get this into production as quickly as possible. I think it's a real breakthrough. And you can see the pipeline quite clearly that in some ways the future of the hospital, some tasks come to you. Yeah.

Why do I need to go to a physical facility to wait four hours for something that's going to be 10 minute conversation and then I'm going to wait five weeks for results. That whole supply chain collapses and it comes directly to you. Obviously, if there needs to be an intervention or there needs to be a second opinion, you would go somewhere, possibly. But this is an entire reconfiguration of the whole medical ecosystem. Most of medicine is going to come to your house. Totally spot on. The expertise is going to be commodified and made zero marginal cost and available. to 7 billion people in the next 5 to 10 years. So the knowledge isn't going to be the asset.

Previously, expertise was the gatekeeping asset that people charged for. I think it's a massively radical moment, a very exciting, transformational moment, that the expertise is now no longer what really counts. Everyone's going to have access to their expertise. And that improves the quality and accuracy of everybody, because everyone has access to knowledge. So then the real thing that's going to matter is judgment and care and attention to implementing the solution. Like you still have to interact with the real world to operationalize the testing and the treatment and the actual physical care. And there's huge, huge gains to be had there.

But I think less time will be spent just doing the rote information exchange. And it's an exact one-for-one comparison, in my opinion, to what it will be like in the classroom. You know, instead of opening a textbook and having your teacher essentially read out from the textbook for, you know, 50 minutes and then doing five minutes of questions at the end of the class, students are going to do most of their primary learning in a personalized, interactive, mobile-based or tablet-based experience come to the classroom. And the classroom will be about using the knowledge that you've already acquired, learning to debate, learning to be self-critical. learning to give way to other people. The social aspects of learning and the use of knowledge are going to become what we do in the classroom and the kind of learning part will happen separately.

It is such a big shift, and I think people might think about what we're losing until you remember why it was invented, right? The invention of the institution of education happened for the Industrial Revolution. That was a certain type of person that needed to be produced to go to an obedient manufacturing plant, and that's why we designed it in that way.

And it worked for a world that has come and gone. So now we're reinventing something again and expertise gets commoditized. It's an entirely different era that we're stepping into. And then even for the role of the doctor, it becomes something else. So an AI system that can do the diagnostics in some ways, then the nurse does the care. So nurse becomes this new ecosystem. And then what happens to the physician, the competition, the intellectual competition, I think, moves more upstream. So it's no longer about memorizing what's in the textbook.

And I know that they do a lot more than that, a lot more. But it becomes an intersection with AI and synthetic biology and all of these new engineering disciplines that are going to converge in robotics. So it's not that the physician isn't relevant. It's to become a physician or whatever we'll call that job will require profoundly more intellectual capacity in a world of these systems. And everything just moves upstream. Totally. And that's the creative challenge for everybody who is a user of AI. Like we're all obsessed at the moment with like, you know, me and the other big companies that are making these things. But that's going to pass quite quickly because we're not really relevant right now. The next phase is going to be how people use it in practice and take ownership of it, integrate into their worlds. If you have a domain expertise in healthcare or in education or in whatever it is that you do, like that is, it's now one of the most creative times.

that any of us have ever been alive because what we've just described is like flipping the entire system on its head and it's very unclear what the next phase is going to look like. Certainly it's going to be me and the other tech people that design that. Like we're just creating the general purpose tools. We're streaming the intelligence. We're creating the fire and the electricity that's out there. And now I think people are finally like getting their head around the fact that it's actually going to be on everybody else to decide what the boundaries and limitations are, what the guard rails are and how it, you know,

fits in these new systems. Yeah, exactly. I mean, the printing press was to create more Bibles. And now look where we are, like, how did that turn up? An entirely new world was born. There are a handful of people on Earth that are in your position right now. Transforming the world with one of the most important technologies of our generation. What do you want your legacy to be?

Oh, man. I didn't realize you're going to go there. I don't know. I think, I think, um, if I have a, chance of shifting the default trajectory from invention for its own sake to invention for invention of a humanist superintelligence, one that is truly aligned to our interest that keeps humans at the top of the food chain that always serves us and works for us collectively in aggregate, then, you know, maybe I'll feel good about that. That's, that's, maybe I'll be proud

of that. That's what I'm trying to do. Because I think that there's a very narrow path that we have to tread. And there are lots of ways that this goes wonky. And right now we're making a lot of the design decisions that have ramifications that will last over many decades. And I just feel a great weight of responsibility for that. Mustafa, it's been a pleasure. Thank you so much. This has been awesome. Thank you, Sheneid. Thank you. If you enjoyed this episode, please like and subscribe and follow the show and share it with someone you think might be interested. I've got questions was created by me,

Shenebeauvel. The show is produced and edited by Tara Cuts and Sandra Etynion, and executive produced by Paola Pierrez Torres, artwork by Corey Vincent at Field Studio.