TED Talks Daily - Sunday Pick: Could your new best friend be an AI-powered NPC? | The TED AI Show

Episode Date: September 8, 2024

Each Sunday, TED shares an episode of another podcast we think you'll love, handpicked for you… by us. Today we're sharing a special episode of The TED AI Show, our newest podcast about the technology that's changing our lives.Non Player Characters --NPCs for short-- have always been a huge part of what makes video games engaging, from Cortana in Halo to Navi in The Legend of Zelda. But interactions with NPCs were always limited to a pre-written script. Until now. Purnendu Mukherjee is the CEO of Convai, a platform that enables developers to create NPCs with human-like conversational abilities. He joins The TED AI Show host Bilawal Sidhu to chat about our evolving relationship with "AI characters” and what we gain and lose when our digital relationships are so life-like, it almost doesn’t matter who (or what) is on the other end. For transcripts for The TED AI Show, visit go.ted.com/TTAIS-transcripts   

TED Audio Collective. Hey, TED Talks Daily listeners, I'm Elise Hu. Today we have an episode of another podcast from the TED Audio Collective handpicked by us for you. If you've ever played a video game, you've probably encountered an NPC, a non-playable character. You might've even tried to talk to them,

but the conversation was probably limited to a few clicks and a pre-written script. Until now. This week, we're sharing an episode of the TED AI Show, and it's all about what happens when AI allows NPCs to actually engage in conversation with users and what that means for the future of relationships. If you want to hear more AI conversations, you're in luck. There are two upcoming TED AI conferences this October in San Francisco and Vienna. See our show notes for more info.

And you can also find brand new episodes of the TED AI show wherever you get your podcasts. Now on to the episode, right after a quick break. Support for this show comes from Airbnb. If you know me, you know, I love staying in Airbnbs when I travel. They make my family feel most at home when we're away from home. As we settled down at our Airbnb during a recent vacation to Palm Springs, I pictured my own home sitting empty. Wouldn't it be smart and better put to use welcoming a family like mine by hosting it on Airbnb? It feels like the practical thing to do, and with the extra income, I could save up for renovations to make the space even more inviting for ourselves and for future guests.

Your home might be worth more than you think. Find out how much at airbnb.ca slash host. shaping our economy. Join RBC's John Stackhouse and Sonia Sinek from Creative Destruction Lab as they ask bold questions like, why is Canada lagging in AI adoption and how to catch up? Don't get left behind. Listen to Disruptors, the innovation era, and stay ahead of the game in this fast-changing world. Follow Disruptors on Apple Podcasts, Spotify,

or your favorite podcast platform. It's the 26th century, and you're moving through a world as a cybernetically enhanced super soldier on the planet of Reach. You have a big mission in front of you. As Master Chief, you are the last hope for humanity against a hostile alliance of alien forces. Only you can stop the Covenant. Well, only you with the help of your trusty AI sidekick, a hologram with a cropped haircut and shimmering ocean blue skin, Cortana.

Actually, it was 2001, and I just hooked up my Xbox to play the first series of Halo Combat Evolved. I was 11 at the time, and my brother, our friends, and I would pile into our Punjabi living room to play a campaign in split-screen mode. I was totally sucked into the game, invested in the story. And Cortana was key to that, with her humor and emotional depth. Even though she's just a blue hologram, ironically, she was the key to the game's humanity. She was my trusty co-pilot guiding me through these alien worlds,

and throughout the game, I felt a real bond starting to emerge. It seemed totally novel at the time, but I was forming a friendship with a non-player character. I'm Bilal Velsadu, and this is the TED AI Show, where we figure out how to live and thrive in a world where AI is changing everything. Now, non-player characters, or NPCs for short, have always been a big part of video games. In single-player text-based computer games, players could interact with NPCs in a very limited way, just putting in a very specific command and having the NPC respond from a predetermined script.

Think the original King's Quest, where players typed directly into a text box to interact with NPCs, who were there only to move the plot along. This evolved into single-player games like Halo, where NPCs like Cortana had more dynamic personalities and quirks written for them, though were still limited by the design of the game, and therefore static. Now, if you think about multiplayer games, they allow for human-to-human interactions. Think Second Life, where you're interacting with other players through their avatars. What's so immersive about that is you're engaging with the real people who are responding dynamically to you, and you're responding dynamically to them, human-to-human. NPCs exist in this world, and while they can be interesting, you would never confuse these NPCs

as fellow gamers. NPCs just haven't been real enough to be mistaken for actual people. Up until now, these NPCs have been relatively passive. But now, with generative AI, those NPCs no longer have to choose between a limited number of responses. They can script in real time, just like a human player would, reacting to what's happening inside of the game world. Suddenly, video game worlds can be infinitely more immersive and interactive. As these virtual characters become more integrated into our daily lives and maybe even become our friends, are we going to start spending all our times in these virtual worlds? Will our interactions with NPCs start to become unrecognizable from our interactions with humans? And what do we gain or lose if this happens? This is the domain of Convey,

spelled Con-V-A-I, a platform that enables developers to create NPCs with human-like conversational abilities. Convey says their goal is to help developers create virtual characters that can converse in the present moments and build long-term human relationships with human players. There are over a thousand projects being actively built on their platform by creators ranging from AAA studios to indie developers. And what makes Convey so interesting is that they're making technology that enables game developers to create NPCs that are so lifelike it won't matter who or what is on the other end. Purnendu Mukherjee is the CEO of Convey and our guest on the show today. He has a lot to say about our evolving relationship with NPCs, or as he likes to call it, AI characters. And like me,

he was a total gaming nerd. Purnendu, welcome. I'd love to start talking about the origins of Convey. I get that you were a gamer before you worked at NVIDIA, but obviously there's a lot of areas in game development you could have branched into. Why were you drawn to the notion of AI and PC specifically? Before I started at NVIDIA, when I was doing my thesis work, I literally saw this language model wave coming. I wrote this, that while language models are going to get bigger and better

and will potentially even have abilities to have human-like conversation, it is still not going to have the same level of understanding as we humans do. Because we humans don't think from text in, text out, or text to text. We think from a 3D world around us, right? Since we were born, we first understand locomotion, like moving around the 3D world. and then we attach words to these objects we assign meaning, right? So basically, we are multimodal creatures. Where do we find such a multimodal environment where we could potentially have these AIs live, train, and iterate themselves? Virtual worlds. And what kind of

virtual worlds would we have people that can provide feedback to this AI? Heavily populated worlds are games, right? So all those connected together, like if we have to create this human-like mind within a virtual world, NPCs, non-player characters, or let's call it AI characters embodied in a way, are one of the best vehicles to do that. It's almost like, you know, we've got these rich environments where you can embody this sort of AI agent and have it experience very similar stimuli to what we might experience in the real world,

which is a perfect segue into the evolution of AI NPCs, right? This is a non-playable character. This is sort of set dressing, the side thing in this like, you know, kind of like the side dish to go with the main course, which is the game itself. And so I'm kind of curious, what's your historical perspective on how AI NPCs have evolved from their earliest stages to the complex entities that we see today in games and virtual experiences. To talk about the history of games, I mean, there are these pioneering genre-defining games all the way from Half-Life that, you know, like single-handedly define the first-person shooter genre. And of course, Counter-Strike, like the multiplayer aspect of it, that you could have many people in the same world, right? In various ways of gameplay, basically revolutionized gaming.

You know, they could play with each other. So you don't need new gameplay as long as people can involve with each other. So like that has evolved. NPCs has, of course, gotten better, mostly on the or you know animation front but not on the intelligence front as much so what we are seeing it's almost like a cambrian explosion of characters and and ai agents that can not only be very human-like in terms of interaction with the players, just like players played with players,

now AI can also play with players, both as friends or enemies, you know, cooperatively or competitively. I think it's this magical moment where now we've, of course, got these behemoth large language models, but you also have the, you know, kind of multimodal models hitting the scene where it's not just that they can understand text. These models can understand audio, can understand imagery, can understand even video. Right. And so I'm curious to dig into how that affects human-AI relationships. So how do you see AI NPCs sort of changing the nature of player engagement and emotional investment in both games and experiences? Firstly, the way these NPCs are becoming very human-like, there is going to be a large set of people in the world

that will big time benefit from it, mainly because there is a big chunk of players who don't like engaging with real people or are nervous or afraid to do that. They feel much comfortable if they know it's not a real person, and that will help them open up. It will help them socialize. In terms of people that, let's say, are playing single-player games

or multiplayer games, now they can engage with the set of AI characters and have a more engaging time, ideally. And lastly, let's say if it's in a multiplayer environment, people will still enjoy engaging with people, but now they have another reason to have fun together with other people. And then from a relationship standpoint, basically, I think it is important for companies like ourselves to look ahead in terms of the positives as well as the dangers. It is going to quickly fill in the gaps where a human doesn't exist today, right? Whether it's, you know, like just being friends or from a romantic angle or maybe someone that is a mentor or a guide and not just like chat GPT, like text in, text out, but very much gamified, immersive environment

that can reach them and they can effectively have this mentor of an AI. So I think overall, I definitely am an optimist and I see the positive sides. There could be potential darker, you know, sides, dystopian sides that needs to be addressed and understood and informed. I mean, I think it's very fascinating. Like it's one thing to talk to your chat GPT app, and you see a voice emanating from your phone. It's another thing entirely to, let's say, be talking to your mentor,

you know, and it's embodied as like a humanoid character that like has the same sort of expressiveness that you do. It suddenly becomes this sort of more lifelike experience, right? And so as NPCs become more lifelike, what are those ethical considerations that come into play, especially regarding player relationships and, you know, AI behaviors? The number one thing that I think AI needs, and this is a bit controversial, but like if you think deeply enough, right, the biggest fear for AI is centralization and a few entities responsible for these relationships. When a kid grows up with an AI teacher and mentor and that's their all, you know, that's a level of relationship that no company in the earth should own. It's theirs, absolutely, wherever they want to take it. And what can enable this? I think decentralized blockchain technology can provide true ownership.

That is going to be very, very essential, along with confidential computing that can help ensure that their data remains theirs, their memories and relationships remains theirs. Yeah, I mean, you brought up a bunch of very interesting points, right? It's like, if you do have this future where, let's say we have an oligopoly of companies that sort of own the models that mediate your relationship with these digital characters, right? And especially if this is like kids who are like growing up talking to these, you know, NPCs or AI agents, whatever you want to call them. Yeah. You are building like a very rich history of sort of their hopes, wishes, anxieties, worries, desires, and how those evolve over time. Right. And, you know, these agents are getting to a place where it's not just like, oh yes, you say this

and I will say this. It's like, they remember the context and glean insights from your previous conversations. And, you know, maybe the right way to solve that is with decentralized AI. Right. And you alluded to confidential computing as well as like, can we keep this sort of very rich data, you know, as close to the user as possible? Or, you know, And even if you are going to learn and improve models off of it, you do it in this privacy-preserving way, which itself sounds like a tall order. Support for this show comes from Airbnb.

If you know me, you know I love staying in Airbnbs when I travel. They make my family feel most at home when we're away from home. As we settled down at our Airbnb during a recent vacation to Palm Springs, I pictured my own home sitting empty. Wouldn't it be smart and better put to use welcoming a family like mine by hosting it on Airbnb? It feels like the practical thing to do,

and with the extra income, I could save up for renovations to make the space even more inviting for ourselves and for future guests. Your home might be worth more than you think. Find out how much at airbnb.ca slash host. I'm really keen to dig into sort of the experiential and sort of emotional impacts of these type of AI agents, right? And so like one example that I keep going back to is, hey, even if you're playing a solo single player game, you could have this like AI Jarvis or Cortana that sort of understands you, your context, and is with you helping you navigate this sort of game world. How close are we to having that sort of companion in games? The primary aspect that is missing would be the, I mean, to some extent, right? So is the multimodal aspect of it. For that to happen at scale, like very much similar to Jarvis of

Iron Man, is aware of the entire context. That means every new contrarian of the room that you are in, you know, like they're able to see that, process that, along with your digital presence. So we are not far away. Like it may not be at 100% of Iron Man's Jarvis capability, but like if you play the game, it'll feel like that. How does all of this change the way these experiences are authored, right? The analogy

that keeps coming to my mind is sort of the narrative division of Westworld. Is that the right way to think about how you're authoring these type of experiences with these rich characters? The emergent nature of large language models are very interesting. The narrative designers and writers are still the better storytellers, right? We have to have a right mix of both controlled evolution of characters, controlled evolutions of stories, as well as the open-ended and emergent behaviors of these generative AI models, which is where the balance and challenge is. And we are providing all the necessary tools for developers and designers to do that.

To come back to the Westworld analogy, that not only did Westworld have these crafted characters that were some of our favorites, they evolved. They evolved into something else entirely than what was originally written, right? And evolved in a very meaningful manner. Remember what was in Westworld that started the whole thing

about these characters evolving? It's memories. Once you have that, their personalities evolve, they remember things. And not only are, it's enough to just give them long-term memory. You have to keep them up all the time, go about their day, make decisions, interact with other real people or other AI, and those interactions will change their decisions and their pathways.

And some will be very high-intensity experiences, and that will shape their personalities and this will be ideally best put in servers that can have up to like 250 500 people these these ai npcs will always be there living their life you know like uh their experiences will will shape them and they will start making decisions that will be quite bewildering we already see that you know like we did this nvidia demo where we had now two characters and both of them were chatting and uh just let them talk to each other, right? Like just to see what they chat about. And while chatting, Nova made an order that, hey, can you bring me a drink? And Jin was like, sure, let me get that for you.

And he went ahead and brought a drink and gave it to her. And it was so wild that they are not only just talking with each other, they are like carrying out actions and they are giving commands to each other. The demo that you're referring to, which went stupid viral on the internet, I think it got everyone excited to just imagine where games are going next. Right. And so I'd love for you to talk a little bit about the types of experiences and characters people are building with Convey that are exciting to

you both for gaming and non-gaming purposes. The non-gaming examples are primarily in learning and training education related areas. And then there's brand ambassadors, which could be in the likeness or digital human of a celebrity or a nondescript model who is AI powered and knows all about the brand, can guide people how to use the product and whatnot. These AI characters can be even location aware, like you scan a particular QR code anywhere while walking the street and they spun up right there and tell you what the directions are. It's not a far future, okay, where we start seeing these embodied AI characters literally everywhere and very much in public spaces. All the way from take your favorite mall to your favorite

airport, you will see large screens with these AI characters standing there welcoming you, but now you can talk to them and ask them which way to go. You know, like, this is my airport ticket. Which way do I go? Where is the security check? Any kind of information dispense or transactional engagement, these characters would be perfect for real world use cases that we are already seeing. Now, finally, coming to gaming. Well, games are going to be effectively the matrix right so uh what do i mean by that it's going to be so real that you would prefer living there all right so which is which is a dystopian future that we have to be aware it will be pretty darn engaging where instead of the machines taking over we would be willingly submitting ourselves to these game worlds, which will be extremely exciting.

With all of these technologies converging, all the way from your VR devices, VR augmented reality, and all of that, to extremely high-speed internet, the cloud computing aspect of it where these extremely high definition world can be literally rendered on the cloud and streamed to yourself along with these AI and PCs in these worlds in these metaverse like worlds it will become a much easier way for people to put themselves in let's call it the matrix or the metaverse all right so where you can be there, live there almost to engage with your friends and, you know, learn certain things and engage with these AI NPCs. And that is a future not too far away that people will start doing that. People already do that, by the way, in a major way in a lot of the social virtual worlds. Okay. If you think... Especially the younger generation right

like yeah growing up almost in these worlds that is true but also there is a very small minority who have been doing it for a while there's a large audience uh concurrent daily active user base for something like second life where we recently, right? And these people come back daily, live their life, talk to their friends for many years now. And there are newer platforms like VRChat. You know, like I know people that regularly go there, they party on the weekends. Once the challenges of onboarding and the challenges of the friction is reduced to get in those worlds, a lot of people will start going there, right? Let's take that VR chat example. I think that one's fascinating. It's like,

yeah, people are buying expensive full-body tracking setups with expensive headsets and computers to have this high-fidelity embodied experience in virtual worlds. But right now on the other end are other humans that they meet, right? How do you feel about, you know, there being a time, let's say in a couple of years where you're talking to somebody in VR chat and you literally cannot tell if they're human or not, does it matter at that point? Because like, I don't know, maybe these AI agents would be far more thoughtful and nice to you than perhaps a real flaw to humans. So I'm curious how you think about that, especially in the context of this matrix analogy that you're making. You literally reminded me of this movie called Transcendence.

And Johnny Depp was the lead character there. Literally, that's the case there. Basically, Johnny Depp dies and before dying, he actually uploads himself like full, full neuron scan of his brain and uploads himself into the internet. And, and when he comes back after his actual biological body has passed back, people would often ask him, are you real? Are you aware? And his answer would be, if you cannot tell, does it matter? So, so that is going to happen, no doubt. It's already there from a text standpoint. Text in, text out, it'll be hard to tell. And today, right? And the other aspects would be the visuals of it, the animations of it and things like that. But also what might be a giveaway is

if they are always of a particular certain certain personality type they need to have a wide array of personalities and even uh eccentric ai's characters you know that are kind of awkward and and some of them are mean and some of them are super nice and uh you know like uh you get the full high school experience. Yes, exactly. Those are going to be necessary for people to engage. It cannot just be like very assistant-like. So the more the variability, the better, you know, like people would like and engage with

them and people will find their own type. Some people are drawn and attracted to toxic people, right? So basically, they will have all kinds of AI, basically, in these worlds, and you'll choose your pick what you want. It reminds me of the conversation with the architect in The Matrix, where the architect outlines that, hey, you know, like, we made a utopian simulation, but nobody bought it. It just felt too artificial. And almost introducing the flaws of our humanity kind of is what made it a sticky experience, which I think is very fascinating, right? Because these environments and experiences

need to mirror the full range of emotions that we experience. Right, right. So that's what we're noticing. You know, like we did this demo room and people enjoy talking with the meanest character. Wow. Okay. Like they would want to walk away and that mean character would say something provocative that would draw them back to talking to that character. That's something that we have

to be conscious about as well. Like that is literally one of the reasons that Facebook and TikTok became so popular because the newsfeed was programmed for maximum emotional turbulence. Like the content that was the most provocative drawed people there. We don't want to do with our stuff. So like the right balance between engagement and what's good for the people is something that we plan to do. Now, there are many negatives to outweigh the positives here, but Pernendu has spoken with me about being a lonely kid growing up. And I had to ask him about the way these experiences can benefit people. You alluded to, you know, growing up as a kid,

you felt isolated and different than other kids. And I just imagine, you know, I relate to that experience. I was so like deeply into computer graphics and visual effects and all this other stuff that like nobody else cared about at the time. And I, obviously I found my escape through the internet and, you know, OG IRC forums and PHP forums. I'm kind of curious, what does this do for people, you know, who may feel lonely today? Like what role can these AI characters play in sort of enriching their lives? Big time, you know, big time. There is of course, online communities where you could meet those like-minded people, but like they may not be available at the same time. But you have this AI character who could effectively have all the right interests.

Like think of your best friend that you connected with the most, right? That understands you before you even say it, right? So these AI characters can potentially be that for them. You know, like it is risky, but that is where we are evolving to, right? Like undoubtedly. And do you find in that situation, like, let's go back to a younger you or me,

do you imagine in the experience being that these systems can sort of infer what you need? Or would I be like, curating the combination of my like, three best friends, plus a little sprinkle of like, I don't know, John Gaeta and, you know, some other VFX supervisors that I really like. And let me throw in a sprinkle of Alan Watts and a sprinkle of like, you know, Einstein in there. How would people define sort of the, their best friend, if you will, in this, in this space? Yeah, that's a very hard one to answer because we don't choose. I mean, we kind of choose our best friends eventually, but we don't choose. Based on vibe, right? Based on vibe, right, exactly. So, and common interests and things like that is how it evolves. But we don't

exactly choose their eccentric things, what they are interested, other things that they are interested in and whatnot, right? So maybe it will be a multifarious world with lots of different AIs. If these characters have to evolve, if you keep changing them, you will not see their character evolution, right? So you basically go and socialize and you start with your one and maybe they will adapt to your interests and things like that. And eventually they will have their own unique experiences too that will shape them effectively like how it shapes you. Imagine friends that grew up together. Maybe this AI can also go out and have its experience when you are not there, right? So they have stories to tell what happened today. It's kind of mind-blowing.

What you're saying almost makes me feel like we've been at this phase of technology and the internet where we can organize the world's information and make it universally accessible and useful to use the Google mission statement. But now we're heading into a world where we can make the world's people and personalities universally accessible and useful. Yes. And a lot of the technology we are creating, you know, all the way from facial expressions and hand gestures and emotional voices, basically empowering the mind of these non-player characters, are going to be the same technology that may be used for a lot of these social robots. There's obviously both utilitarian and delightful experiences your customers are building. What are you looking forward to?

We have set the vision. We have created the tools and people are developing in terms of the immediate, not just immediate, like medium term, three to five years plan is basically ensure that we have redefined gaming in a very positive way. We have enabled these learning and training experiences at scale that are changing lives of people

in a very positive manner. Brand experiences, product information, real world embodied characters. I love the creator-centric approach. I think it's so important that we don't forget that creators are going to author these experiences. Thank you so much for your time. Thank you so much. Yeah, it was great chatting. Around the time Purnendu and I had this conversation,

he invited me to the Convey headquarters, where I got to see their NPC innovations firsthand. And let me tell you, it was pretty wild. When I walked in, they had this massive monitor with an AI anime character on it. And the people at Convey told me to just have a conversation with it. They told me I could talk to it in Hindi,

which I was psyched about. Then I tried to press a little deeper. I think there's this really human urge to try and push the boundaries of an AI system to prove that it's intelligent and human enough to exist beyond the constraints of corporate language. Most of my efforts were in vain though, because as soon as the AI came back

with these canned responses, it kind of ruined some of the effect. Even though it was giving me unscripted responses, it kind of ruined some of the effect. Even though it was giving me unscripted responses, it still had parameters. The AI allowed me to go off script. And even though the model I was talking to recognized what I was saying, it was still responding based on its parameters, the guardrails set by the company. While an AI like this might be great at helping you fight a lethal force of aliens, it's hard to know if it will ever reach the messier, more human parts of how we relate to one another. The TED AI Show is a part of the TED Audio Collective and is produced by TED with Cosmic Standard.

Our producers are Ella Fetter and Sarah McRae. Our editors are Ben Bencheng and Alejandra Salazar. Our showrunner is Ivana Tucker and our associate producer is Ben Montoya. Our engineer is Asia Pilar Simpson. Our technical director is Jacob Winink and our executive producer is Eliza Smith. Our fact checker is Julia Dickerson and I'm your host, Bilal Siddou. See y' producer is Eliza Smith. Our fact checker is Julia Dickerson. And I'm your host, Bilal Siddoo. See y'all in the next one. Looking for a fun challenge to share with your friends and family? TED now has games designed to keep your mind sharp while having fun. Visit TED.com slash games to explore the joy and wonder of TED Games.