TED Talks Daily - Entertainment is getting an AI upgrade | Kylan Gibbs

Episode Date: June 12, 2024

AI has the power to bring your favorite fictional characters to life, says technologist Kylan Gibbs. Introducing Caleb, an "AI agent" with personality and internal reasoning, he demonstrates how AI-powered characters can interact with people in novel ways, generate unique video game outcomes and augment our ability to tell stories, opening up new worlds of possibility.

TED Audio Collective. You're listening to TED Talks Daily, where we bring you new ideas to spark your curiosity every day. I'm your host, Elise Hu. You're hearing me try out an interactive AI-driven storytelling exhibit featuring a character who I could influence based on how I reacted to him. Step into Story Weaver, a demo that lets you craft a narrative universe powered by AI. Let's see what happens.

Can I try this out? We would love you to try it out. Excellent. All right. By the way, thanks for showing me this cool action figure. I love it. What's his name? The creation comes from Kylan Gibbs' company InWorld,

and in his talk, he takes us behind the scenes of making this human-AI collaboration and the vision it portends for the future. Coming up after a short break. Support for this show comes from Airbnb. If you know me, you know I love staying in Airbnbs when I travel. They make my family feel most at home when we're away from home. As we settled down at our Airbnb during a recent vacation to Palm Springs,

I pictured my own home sitting empty. Wouldn't it be smart and better put to use welcoming a family like mine by hosting it on Airbnb? It feels like the practical thing to do, and with the extra income, I could save up for renovations to make the space even more inviting for ourselves and for future guests. Your home might be worth more than you think. Find out how much at Airbnb.ca slash host. AI keeping you up at night? Wondering what it means for your business? Thank you. And now, our TED Talk of the day. When I was 10, I spent a lot of time playing with my Spider-Man action figures. I would have done anything for them to come to life. I honestly would take Gandalf over any therapist,

but it's been pretty hard to connect with the guy since I long ago finished all of Tolkien's works and movies. Why is it that we have to say goodbye on the final page? We may not need to. Stories are changing, moving from static sketches and scripted narratives to living worlds and characters. AI agents are changing the way we can tell stories.

These agents aren't just chatbots. They're any autonomous system that can take a directive or task and accomplish it without human intervention. Virtual assistants, like a chat GPT, autonomous vehicles like a Waymo or Tesla, robots moving boxes in an Amazon warehouse. These are all agents. But now they're changing storytelling too.

We can bring together the active participation of video games with a much more natural social form of interaction enabled by AI agents. Characters that come to life with their own personalities, styles, voices, and expression, while still being able to interact with one another, the audience, and the environment. Meet Caleb.

Oh, hey, Grandma. Welcome to my secret attic hideout. What's Mr. Moose doing there? That's actually Brother Moose, not Mr. Moose. He's like, um, the wise old moose who gives me advice and stuff. Oh, uh, by the way, thanks for showing me your old superhero action figure, Grandma. It's like, totally awesome. Um, what's its name? Mighty Squirrel. Really?

Yes. Wow, that's so awesome. A squirrel with superpowers. That's nuts. Get it? Nuts? Like what squirrels eat? I bet Mighty Squirrel had some epic battles with evil acorns. By having that real-time responsiveness, Caleb makes the audience feel seen and heard, as if they're all inhabiting the shared space. And it isn't just any audience, that's actually my amazing grandma. The characters can also come together with one another and take ideas from the audience

and improvise new content on the fly. Each of these characters has their own internal reasoning and motivations, but they actually take those ideas from my grandma and then create new content in real time. They can even allow the audience to shape the overall story arc and conclusion of the experience. Prepare yourselves for the mighty squirrel is about to take flight.

Whoa! Uh-huh. Grandma, do you have anything to help me take flight again? There's an eagle outside on a tree. An eagle, you say? Perfect. Time to spread my wings and soar through the skies. Whoa, mighty squirrel.

You're going to be the coolest flying superhero ever. And we'll have epic battles with the evil villains. Okay, so I'm mainly just jealous that Caleb is getting out of the experience I always wanted as a child, but I also love seeing how he takes the story, puts it in the audience's hands, and turns it into a canvas for their imagination. This is, of course, a tiny slice of what's possible. With there already being other

AI-enabled games and experiences that exist today. Most exciting for me is a brand new genre we're seeing emerge that takes social interaction and conversation and puts it as the core mechanic of the experience. So let's say instead of jump and shoot, you actually have to use your social skills to navigate complex social scenarios, immediate conflicts. That's how you win the game. I would definitely have benefited more from that type of game than the ones I was playing when I was a teenager. And it's not just pure games per se. Imagine I had an AI tutor who texted me to remind me of a session

and then jumped with me into a virtual world, walking through the emperor's chambers in ancient China, casually switching between English and Chinese as we reviewed last week's history lesson. Or as my family and I sat down to watch our favorite reality TV show or sports game, the agent could actually see the ongoing performance in real time using vision and add hilarious commentary in the background.

Games, though, are unique because they already take us from passive observer and turn us into an active participant. When I get really deep into a game, especially immersive role-playing games like Final Fantasy VII, Baldur's Gate 3 or Elden Ring, yes, I'm a massive role-playing game geek, at some point it feels like I'm in the story. But there's still something missing. The core mechanics of these experiences, jump and shoot, hack and slash, point and click,

feel so distant from the natural ways that I interact with people and things in my own life. And so as immersed as I get in these experiences, it kind of feels like I'm stuck on these rails set out by the developer and nudged along from scripted point to scripted point based on the buttons that I press. And immersion requires a sense of real-time responsiveness,

a sense that there's a consequence that is immediate and lasting and unique to the choices and actions that I take. However, until now, it's been extremely difficult to prescript every combination of consequences to every permutation of player behaviors. Agents overcome this, though. It's not just these standalone characters we want to bring to life, but fully agentic, interconnected worlds where a choice at one point might have dramatic consequences somewhere down the line. And the unique content produced for each user is still grounded in a shared lore and experience,

so that we have that shared social context that we all need for media. My company, InWorld, helps entertainment houses and game studios to build these agents for the next generation of media. And when we begin to build a character like Caleb, we start with the brain, which is crafted in a similar way to instructing an extremely capable improv actor.

We start by training custom machine learning models that give him a lay of the land, understanding the lore, vocabulary and grammar appropriate for the experience. And then we actually go in and we tweak his persona, his unique personality, motivations, flaws and biases. We actually craft his dialogue style and voice, the way he verbally expresses himself.

We build a unique emotional profile, the way he feels, his social tendencies and relationship habits, and then we preload a bunch of knowledge and memories that give him that robust backstory. Then we actually craft his internal reasoning and mental state, kind of the conditions under which he feels certain things or takes certain behaviors. We then decide how he actually gestures and animates those behaviors.

And then that completes the brain, which we actually take and we attach to an avatar within a virtual world or game engine, effectively bringing it to life. And then we're ready to play, and when we're playing the experience, there's kind of an overarching narrative agent that acts like an AI director, managing or even generating the storyline in real time

and ensuring that all the characters and entities cohere to their proper places within the story and context. For a specific character like Kalem, when he's interacting with the world and characters, he does so in a similar way to how we as humans do, with perception, cognition, and behavior. Perception is the multimodal sensory input flowing in, the understanding of the world and context and people and objects within it. And then with cognition, he processes that input, kind of like the internal processing of our minds. He filters out irrelevant information

to make sure everything is safe and narrative-aligned. He extracts out important signals like sentiment, intent, emotional cues. He retrieves relevant memories from his past and even generates nuance on the fly, finally reasoning over what to do, both verbally and non-verbally. And finally, with behavior, he actually generates his responses. He decides what to say, both verbally and non-verbally. And finally, with behavior, he actually generates his responses. He decides what to say and how to say it, what facial and body gestures to use, what actions to take, and people and objects to interact with. And that's

what brings the character to life. And so, as amazed as I am by all these task-focused applications of AI that are coming out, the more I work with studios and creatives, the more excited I am for the potential for these agents to extend human creative potential and for them to allow audiences to step beyond the script, for them to give audiences agency and allow them to co-create experiences that stem from their own imagination.

It seems like there's a pretty solid chance my Spider-Man action figures could come to life after all. Thank you. Support for this show comes from Airbnb. If you know me, you know I love staying in Airbnbs when I travel. They make my family feel most at home when we're away from home. As we settled down at our Airbnb during a recent vacation to Palm Springs, I pictured my own home sitting empty. Wouldn't it be smart and better put to use welcoming a family like mine by hosting it on Airbnb?

It feels like the practical thing to do, and with the extra income, I could save up for renovations to make the space even more inviting for ourselves and for future guests. Your home might be worth more than you think. Find out how much at airbnb.ca slash host. That was Kylan Gibbs at TED 2024. If you're curious about TED's curation, find out more at TED.com slash curation guidelines. And that's it for today.

TED Talks Daily is part of the TED Audio Collective. This episode was produced and edited by our team, Martha Estefanos, Oliver Friedman, Brian Green, Autumn Thompson, and Alejandra Salazar. It was mixed by Christopher Faisy-Bogan. Additional support from Emma Taubner, Daniela Balarezo, and Will Hennessey. I'm Elise Hugh.

I'll be back tomorrow with a fresh idea for your feed. Thanks for listening. Looking for a fun challenge to share with your friends and family? Today, TED now has games designed to keep your mind sharp while having fun. Visit TED.com slash games to explore the joy and wonder of TED games.