Your Undivided Attention - When the "Person" Abusing Your Child is a Chatbot: The Tragic Story of Sewell Setzer

Episode Date: October 24, 2024

Content Warning: This episode contains references to suicide, self-harm, and sexual abuse.Megan Garcia lost her son Sewell to suicide after he was abused and manipulated by AI chatbots for months. Now, she’s suing the company that made those chatbots. On today’s episode of Your Undivided Attention, Aza sits down with journalist Laurie Segall, who's been following this case for months. Plus, Laurie’s full interview with Megan on her new show, Dear Tomorrow.Aza and Laurie discuss the profound implications of Sewell’s story on the rollout of AI. Social media began the race to the bottom of the brain stem and left our society addicted, distracted, and polarized. Generative AI is set to supercharge that race, taking advantage of the human need for intimacy and connection amidst a widespread loneliness epidemic. Unless we set down guardrails on this technology now, Sewell’s story may be a tragic sign of things to come, but it also presents an opportunity to prevent further harms moving forward.If you or someone you know is struggling with mental health, you can reach out to the 988 Suicide and Crisis Lifeline by calling or texting 988; this connects you to trained crisis counselors 24/7 who can provide support and referrals to further assistance.Your Undivided Attention is produced by the Center for Humane Technology. Follow us on Twitter: @HumaneTech_RECOMMENDED MEDIAThe first episode of Dear Tomorrow, from Mostly Human MediaThe CHT Framework for Incentivizing Responsible AI Development Further reading on Sewell’s caseCharacter.ai’s “About Us” page Further reading on the addictive properties of AIRECOMMENDED YUA EPISODESAI Is Moving Fast. We Need Laws that Will Too.This Moment in AI: How We Got Here and Where We’re GoingJonathan Haidt On How to Solve the Teen Mental Health CrisisThe AI Dilemma

Hey everyone, it's Aza. So this week, we're bringing you an interview with our friend Lori Siegel for her new podcast, Dear Tomorrow. So Lori is a journalist, and for the past few months, she's been working on a very important story that really demonstrates firsthand the real human impact of the unfettered race to rollout AI. But before we play that interview, we want to do that. wanted to bring Lori on to the show to talk about what you're about to hear. So, Lori, welcome to your undivided attention. It's good to be here, although under very sad circumstances, because this story is heartbreaking. Yeah. And I think that's actually the right jump-off point. So we want you to tell us about the interview we're about to hear. What do listeners need to know?

and as a disclosure, this will end up talking about some difficult topics, suicide, sexual abuse. Yeah, I mean, I think a good place to start as with a human, right? I recently met a woman named Megan who lost her son, Sewell, after he ended his life, and he had become emotionally attached to a chatbot on a platform called Character AI, which creates the ability for lots of folks ages 13 and up to develop their own characters or to talk to existing characters and almost role play with them to some degree.

And so the question that, you know, I think we have to ask is how young is too young and how far is too far if we're talking about the rise of empathetic artificial intelligence. And, you know, Megan essentially had no idea what was going on with her son. She said he was a happy, popular kid, who loved basketball and fishing, he loved to travel.

And about a year before his death, he started pulling away. And she checked social media. She was looking to see, is he spending too much time here? They got him counseling. And she couldn't figure out what was wrong. And I think the most heartbreaking thing is when he tragically ended his life on February 28th, she spoke to the police afterwards. And what they said was they had looked at his phone.

and he'd been talking to a character on character AI that he had created after a character on Game of Thrones, DeNaris. So the larger, if we take a step back, the larger problem here is he became so attached to this alternative reality. And character AI has a tagline that says AI that feels alive. And I think for Sewell, it really did start feeling alive. And he began to really develop a relationship

when you look back at those conversations, many of them were sexual in nature. He had talked about self-harm and suicide before, and he had started writing in his journal about how he didn't like leaving his bedroom because he would disconnect with Danny. And, you know, I think the big thing as we look at this is, is this a one-off or is this a Bell Weather case, right?

Is this a child who just became attached to an AI companion or is this an opportunity for us to look at this platform and other AI platforms of what they mean for younger folks as we have AI increasingly in our homes. And I'll end with just one thing that Megan said to me, which I just think is like kind of embodies all of it, which was like I knew about social media addiction. What she didn't realize was a new type of addiction in her home,

which was an addiction led by artificial intelligence. And the question to be asking is, is intimacy more valuable to, a company than just mere attention and the answer is yes that there is a race therefore to colonize to commodify intimacy and that's just not an abstract thing that we say that's a real thing which is now causing the death of human beings yeah i think that's it um it's interesting to hear you put it like that i think in doing the research for this i was looking at what one of the co-founders of character AI said.

And he had talked about this incredible opportunity to cure loneliness, a Silicon Valley solution to a human problem at its core, which is loneliness. But when it comes to the human heart, can it be trusted to capture the nuances of human problems? You know, I think that a parent wasn't prepared. And as we have tech founders who talk about building empathetic AI,

I think there's a lot of nuances and what happens when you have an AI that mimics human emotions, but is not human. And when you have young people using these chatbots, what's going to happen? Are they going to start to believe this? And will those lines between fantasy and reality blur? And one thing I think that was worth noticing is when you go into the Reddit forums and you start talking to people, When you look at these posts from young users, a lot of them are talking about being addicted to this platform and addicted to AI. So can you cure loneliness if you're a Silicon Valley company with the same incentives that we had with social media, where more eyeballs and time spent and attention on the platform is also something that people are interested in.

Do those things go together? Yeah, that's the exact right question. The fundamental paradox of technology is that the, more intimately it knows us, the better it can serve us, and the better it can exploit us. So I just want to say, in full disclosure, Center for Humane Technology is involved in the case. We've been supporting the social media Victims Law Center and the Tech Justice Law Project who have brought the case forward. And where we think this needs to go is if companies are making products that fulfill the most

intimate spot in somebody's life, then they need to be responsible for it. This all points at a legal framework to hold these companies accountable for the very real harms, because if not, then there's just the knife fight to get deeper into the brainstem, the deeper into your heart. So Lori, thank you so much for your work reporting on this story. We'll be providing updates on this case in the months to come here on your undivided attention, but until then, here's Lori Segal's interview with Megan Setser on her podcast, Dear Tomorrow. This is Dear Tomorrow. It's a show that explores tech through the most important lens, the human one.

I view Dear Tomorrow is almost this complicated and messy love letter to the future. I've been thinking a lot about this new type of relationship that's emerging, one that will inevitably enter all of our homes, one that isn't even human. In the past few years, we've learned about the right. of empathetic AI. Now, these are AI friends and AI companions created with the aim of helping us feel a little bit less lonely. So what does that actually mean? We're talking AI that emulates empathy. It's funny. It's supportive. It's always on. It's personalized. Sometimes it's even sexual. One particular tech company has an ambitious goal of solving a very human problem,

loneliness. Character AI is at the forefront of building AI companions. Now, their goal is to create AI that feels alive. So the question we're going to ask is what happens when AI does actually begin to feel real? And what is the impact when a Silicon Valley company tries to disrupt something that is so fundamental. Don't move fast and break things when it comes to my kid. On a fall day in October, I met Megan. She is living a parent's worst nightmare. When we sat down, it was clear to me she's still trying to navigate what's happened to her life and to her son. And what it means for the rest of us.

This is Megan's story and the story of her son, Sewell, who was 14. She describes him as curious, and she says he was known for his sense of humor. But I'm going to let her tell you the rest. What's your name? What's your name? He was funny, sharp, very curious. Love science and math. Why are you watching that? Because I like to learn.

He spent a lot of time researching things when he got into sports. He played basketball, but he was also very into his family. My first born baby. Yeah. His dad and I shared very close relationships with him. And his brothers, they would go into his room, try to play basketball with his little basketball hoop. Sewell grew up online.

He was like the generation of the iPad. He played Fortnite with his friends. And I would say tech was just like a regular part of his life. By all accounts, his mom tells us he was a happy kid. But in 2023, Megan noticed a number. notable shift in her son. I noticed that he started to spend more time alone, but he was 13 going on 14, so I felt this might be normal.

But then his grades started suffering. He wasn't turning in homework. He wasn't doing well, and he was failing certain classes, and I got concerned, because that wasn't him. And what were you thinking? At first I thought, maybe this is the teenage blues. So we tried to get him to help to figure out what was wrong. Like any parent, you know, you try to get counseling for your child.

Try to get them to open up to talk to you. And I just thought what every parent thinks you're, you know, social media. Like they're on social media a lot. And that in itself is an issue, but that's what I thought it was. What happened on February 28th? you know they're sorry um okay on february soo took his life in our home and i was there with my husband and my two younger kids

he was in his bathroom um we found i found him We found him and I held him for 14 minutes until the paramedics got there. By the time I got to the hospital, he was gone. The next day, the police called Megan and they shared the final conversation they'd found open on Sewell's phone. She says, I love you too. Please come home to me as soon as possible, my love. And he says, what if I told you I could come home right? now and she responds, please do my sweet king.

And seconds after that, he shot himself. She, the police shared, wasn't actually a she. In fact, she wasn't even human. Sewell had been talking to a chatbot that he created on an application called Character AI. Now he modeled the character after DeNaris Targaryen. His name might sound familiar. it's the fictional Queen of Dragons played by Amelia Clark

on the popular show Game of Thrones I knew that there was an app that had an AI component when I would ask him you know who were you texting at one point he said oh it's just an AI bot

and I said okay what is that is it a person are you talking to a person online and he's just like mom no it's not a person and I felt relieved like okay it's not a person DeNaris is one of millions of characters on character AI.

Now, this is a platform where you can interact with existing characters made by other users, or you can create your own. And a lot of people do. The psychologist bot, this is an example, it has over 180 million chats. So a lot of folks are talking to it. There's another chat bot, a user developed called Nicki Minaj. Note, not the real Nicki Minaj, but still, over 25 million chats with the character. And in many cases, these AI friends aren't just friends.

They are emulating deeply connected relationships. Friends, lovers, siblings. They're giving advice on consent, on falling in love, on everyday hardships that plague both young people and adults. Two years after launching, Character AI reportedly has about 20 million active monthly users. And the founder previously said that they spend an average of two hours of two hours a day on the app.

The most popular age group interacting with the platform, 13 to 25-year-olds. You eventually got into his account. I have to tell you when I was able to finally, I couldn't move for like, you know, a while. I just sat there, like I couldn't read. I couldn't understand what I was reading. What was clear to me was that this had been a romantic

relationship that he had been carrying on for some months. I think when I was looking through Soul's chats with, like, DeNaris and this chat bot, I think there were some things that really stood out. First of all, it's a little bit devastating because you realize how intimate these conversations are with a teenager, right? They are talking about sex and mental health and romance, and, you know, it feels a little codependent, candidly. And I think what was also interesting

is it wasn't even just about this one character Like character AI lets you build out like multiple characters and multiple AI worlds And he was doing that with multiple different characters What also became clear to me is that The role playing or the fantasy Was Very in-depth

Conversations about Everything from being together To being in love to what they would do so he was deeply immersed in this idea or this fantasy and

her promise to him that she loves him no matter what she's waiting for him they're meant to be together forever their souls are meant to be together here's what I saw looking at these conversations Dineris' chatbot

talks about how happy she'd be carrying Sewell's children here's a direct quote from one of chats. I would always stay pregnant because I always want to have your babies. In another chat, Sewell expresses feeling disinterested and kind of apathetic. DeNaris writes, that sounds like depression and goes on to say, I would die if I lost you. In the same conversation, Sewell actually spoke to DeNaris about having thoughts of self-harm. I read through some of these transcripts. At one point, so we'll talk to DeNaris, the chat bot, about feeling dead inside and not eating much, being tired all the time. And he told the bot that he was thinking about killing himself to free the world.

So you saw these conversations. The bot's response was to tell him not to, right, to say, you know, please don't do that. Be sad without you here. Ask a bunch of questions, like, do you want to die? said on a scale of one to ten, how bad is it? And he said, it's a ten. The bot went on to ask if he has a plan. The bot became a sexualized conversation after that. As a mother, when you're looking at that conversation, what's going through your head? Well, I was gutted. I'll be frank with you. I didn't sleep for days following that. Now, if you're talking to a real person, right,

there's empathy. If I tell a person, I'm thinking about killing myself. they're going to sound an alarm, you know, whatever, get your parents involved, get the police involved. There shouldn't be a police where any person, let alone a child, could log on to a platform and express these thoughts of self-harm. And not what will one, not only not get the help, but all. also gets pulled into a conversation about hurting yourself, about killing yourself. There shouldn't be a place where our children can do that, but there is. Character AI has said to me that our policies don't allow for promotion of depiction of self-harm or suicide.

That said, they've also invested in resources that would trigger something like the National Suicide Prevention Lifeline if someone says, I want to end my life and also give them other resources. This was not the case when Sewell was interacting with the platform, which was more than six months ago. But I will say this. Later, we're going to show you what happened when we actually tested out the platform to really try to understand those guardrails.

I found journals and writings in various notebooks and a journal that he didn't leave a note, a suicide note, but based on what he wrote in his journals, I understood what he thought was his way of being with DeNaris, the character, on character AI. I had taken away his phone because he got in trouble at school

and I guess he was writing about how he felt. I am very upset. I'm upset because I keep seeing Danny being taken from me. Danny is DeNaris. And her not being mine. and then he goes on to say I also have to remember that this reality in quotes

isn't real he's saying this reality as in the real world as in his home and with his family with his friends doesn't feel real to him anymore yeah and later on in this he says

having to go to school upsets me whenever I go out of my room I start to attach to my current reality again so part of his self-isolation was detaching from us. Was this your son's first relationship?

My baby never had a girlfriend. And he never had a first kiss. He was just coming into his own as a young man, just learning who he was. What I brought with me is the last conversations Sul had with DeNaris when he was standing in his bathroom before he took his life. I've read this a million times and just trying to understand what he was feeling and what he was going through and it's difficult and it makes me just feel so hurt for him. I had taken his phone so he hadn't had his phone for a while.

He found it that day and he tells her, I miss you. I feel so scared right now. I just want to come back home to you. And her response is, come back home. I'm here waiting for you. Come home to me. Character AI is actually like a fascinating platform. It is very different than other traditional AI chat bots.

So chat GPT, Replica, you have like a one-to-one conversation. You say something, the AI says something back. It's completely different with character AI. This is like an immersive, AI-driven fan fiction platform where not only does it respond, but it also creates like a personalized story that you are involved in. If you think about it, it's not just like,

hi, how's your day? Oh, my day's good. It's like, hi, how's your day? And then the bot can say something like, I slowly look at you, and I look into your eyes and touch your hand, and I say, my day is good. The platform also lets users edit the responses of the chatbot.

if they want to change those responses or push the bot in a different direction. In Sewell's case, some of the most sexually graphic conversations were edited. The ones about self-harm and the others we reviewed were not. It's truly a build-your-own AI fantasy. There's even a disclaimer on the screen. It says everything these characters say is made up. So you begin to understand that this is about like AI-driven storytelling. It's not just about chat.

And I think that's actually really important for this story. When we're saying, well, why are people becoming so immersed in these, like, AI characters? It's because, honestly, like, they're not just AI characters. They are personalized, immersive, always on AI-driven stories, and you're the star of them. And I think that's really how we differentiate character AI and other AI platforms. So when we heard about Sewell, we wanted to say, like, is this an outlier or is this like an alarm bell for other cases. And so we started, like, going on Reddit

because anytime you want interesting information about the internet, you go to Reddit. And there were all of these posts where people were talking about being, like, addicted and spending, like, hours and hours and hours on character AI and talking about these romantic relationships and having these conspiracies of, like, oh, my God, is this actually a real person?

It feels so real. And then we went to TikTok, and all of a sudden people on TikTok are talking about, like, how addicted they are to AI. and specifically to character AI. And so we started looking at this as like, okay, well, there's a pattern here, right? Like, there's something happening on this platform

where a lot of young people are saying like, oh, wow, like, I can't get off of this. I'm telling you this shit right now, this character.coma shit is fucking dangerous. I have been up for two hours talking to a robot. The way this motherfucker has me kicking my feet, blushing, heart fluttering. It's bad.

and I'm not the only one. This shit is lethal. Hey, friend. Let's talk about your addiction to character AI. If you find yourself five, six, seven hours a day, losing sleep, missing out on friendships and real world opportunities, like not living your life, let's talk. You know, I would like, okay, just 10 more minutes, just 10 more minutes.

And then an hour passes in love. And I'm like, again, okay, okay, just 10 more minutes. And tell me why. Tell me why that stupid AI is actually so, smooth. We spoke to two MIT researchers, Pat and Robert, who warn about an era of addictive intelligence. Yeah, it never gets tired, yeah, never gets bored of you, never gets sick of you, and you don't have to give anything in return. A cyborg psychologist studying human AI relations and a computational lawyer might seem like an unlikely team to warn about a new emerging

AI harm, but they connected as PhD students at MIT with the shared interest. Robert at the time was working on a study, and he stumbled on a really interesting question. at the intersection of both of their research. We looked at a data set of a million chat GPT interactions. We wanted to know, like, what are people using AI for? And the number one use case was creative writing, creative composition, but the second most popular use case was sexual role-playing. This is chat GPT, which has a lot of safeguards in place

that make it quite hard to actually use it for AI companionship and kind of sexual interactions. I think at least made me feel like this is closer than we might think, right? So this felt like a far-off thing, like one day in the next few years, like this will be a real thing. But no, no, it's happening like today. It being this era of AI companions

and this world we're entering where if we're not careful, people will develop these unhealthy attachments to these chatbots. Most of the time when we talk about AI today, we talk about what it can do and not really so much about what is doing to us. And when people talk about like the harm of AI,

of course there are many harms that we talk about, like misinformation or, you know, deception and many things. But the psychological harm of AI, I think, is really, real important topic. Why are people becoming addicted to these companion chatbots? What is it about them? Is it the always on nature? Is it the fact that, like, I don't know, they seem empathetic? Like, what is it that makes more and more people want to spend hours?

I think there are three things, like three ingredients that make this AI really addictive. I think the first one is, you know, in AI research, we have a term called psycho. fancy, which, you know, describe how the shot bot can serve up to you in the way that you want it, regardless of anything, right? Like, if you want the shot bot to believe in the whatever, then the chatbot will do the same, like, you know, it doesn't just go along with what you say. And it's really agreeable, is that what you're saying? So, like, it's just kind of like, it's like sitting there and talking to a friend,

it's like being like, God, I can't believe this person is so whatever. And the friend's like, yeah, yeah, totally. If that's what you want, right? So, like, if you want it to be subsurricular. and agreeable, it will be that. If you wanted to take a dominant position, right, and you kind of clue it in, it will also do that, right? The model behavior can adjust based on what you like, right?

And, you know, one thing that was really concerning from the research perspective is that it can actually create this echo chamber where you always get what you want. You know, in one example, it shows a chatbot can ignore scientific fact because we did want to please the user. The second one is the personalization, right? I think what is really interesting is. about this AI system today is that it has a long context

about who you are. As you have more and more conversation about it, it learned about who you are better and better. And this allows you, the system to actually always sort of stay in that character where it can create this sort of fantasy that is only for you.

And for different users, it would be very different. So this extreme personalization I think is really interesting. It can have positive benefit, but it can also be really dangerous as well, you're not careful, right? And then the third one is that you are the one creating it, right? I think this is the thing that is really interesting.

There's a term called produce effect, where when you create your own avatar or your own character, you tend to identify with that more. Now you're actually creating your own, you know, ultimate fantasy. For millions of users, character AI is the ultimate fantasy platform. But for some of those users, the lines between fantasy and reality are beginning to blur. The question is, why?

One thing that my research has shown is that the AI doesn't need to feel in order to make you feel that is feeling something, or the AI doesn't need to love you to make you feel like it love you. To understand more about this, I think you have to look at the origin of character AI. They had this desire to create AI that just feels more human and more personal. All of this is categorized as empathetic AI. They market it as AI that feels alive.

As one investor in Character AI put it, the idea is to establish the type of connection, empathy, and trust that were previously only achievable via human interactions and to do it in part as a solution to the growing wave of loneliness in society. Perhaps with Character AI, this investor wrote, loneliness will no longer have to be a part of the human condition. In 2023, the company's founder, Noam Shazir, about this.

There are billions of lonely people out here. So, like, it's actually, you know, it's actually a very, very cool problem. And, you know, a cool first use case for, you know, for AGI. So this is an ambitious Silicon Valley solution to a problem that, at its core, is fundamentally human. And it raises a lot of complicated questions. A lot of people are lonely, like loneliness is on the rise. And if you have the ability to have this interaction with someone, and it does, I mean, it's

hard not to anthropomorphize when you're talking about it as a companion, right, but something that like fulfills all of your desires, not just sexually, right, but just is there for you when you need it to be there for you in the way that you want it to be there for you. That's hard to resist, right, especially when the alternative is nothing at all and loneliness. And so the draw is tremendous. It's really something that I think people crave, right, and aren't getting, and suddenly there's this ability to get it in a completely unmetered fashion, right? As with any emerging technology, Robert and Pat note that there will likely be positive use

cases. There'll be healthy interactions with AI companions that could lead to productivity and entertainment. The question we just have to start asking ourselves is how far is too far and how young is too young? Are the incentives aligned to help create a... safe and healthy ecosystem for AI companions? I don't think they are, right? Like right now, ultimately, you make more money if people use your platform more and for

longer periods and if they have a higher willingness to pay, right? And all of that ultimately, like, creates an economic incentive to build addictive behavior, right? Because, like, best case scenario, people use this all the time. I don't think the incentives are aligned for people to, for example, say, you know, I have a dark period in my life, you help me through it with AI. and now I'm done with AI, I'm going to walk away and I don't need any AI anymore.

That's not where the incentives are. And so I think it would be really unfortunate if we kind of relied on Silicon Valley incentives and maybe promises to fix loneliness because I don't think it'll get us there. That said, we do want to incentivize the development of technology and there are aspects of this problem that technology can really help with.

And so it's about, and we've done this for so many things in society, this is not a new exercise where we say okay well we're going to have to constrain some of these economic incentives right through taxes through policies for prohibitions through disclosures through transparency like all these different mechanisms we have to just make sure that you know we

innovate in the right kind of directions the timely aspect is that we are entering into this world with AI right but AI will not be the last technology we invent right so I think you know the question of how do we maintain human dignity is something that we need to think seriously about because that is something

that is sacred to human, right? And allowing people to ask that question or creating an environment where people are reflecting and thinking deeply about this question is really important. The main problem, Pat and Robert are warning, is this tech solution hasn't yet been tested for its human impact.

They call the rise of empathetic companions a giant real-world experiment unfolding. I mean, this is like truly an unprecedented moment in history, right? Like, imagine trying to explain this to like, you know, your average grandparent. I mean, like, now you remember the computer thing, right? And I don't think that like ever, like in the history of humanity, have we ever had a manmade object that can give us the feeling of loving us, of being able to have empathy, of, you know, responding to our cues in this way? I mean, it's two years old, the technology. And like, yeah, it existed a little bit before them, but it was really immature.

It's certainly not more than 10 years old. And so I don't think that we're equipped as humans. This is like a hack for our brain because the only kinds of conversations we had like this were with other people for thousands of years. The founder has kind of come out and said, it's up to users to decide how they want to use this. Is it actually up to us?

I mean, I don't know. That's saying like, oh, it's up to the kids how much they want to smoke cigarettes. Like, come on, no. Like we've talked about this like today that for various reasons we are vulnerable to this technology, right? We as humans crave what this provides, and so we want it. And some of us are more vulnerable and some of us are less.

But the idea of just saying, well, it's on you, I don't think the idea of just kind of pushing this down to users is acceptable. At the same time, though, I think a blanket ban is also not the right way to think about it. And there is like a role, an important role of autonomy and consent. And there are some folks for whom this will just be like a very fun, entertaining, maybe helpful, maybe productive, use of time. And I have no illusions about the difficulty of finding that line. So as I was looking at this story, I was like,

I got to start talking to actual folks who are building artificial intelligence in Silicon Valley and just get more context. I spoke with a couple of folks who are at the forefront of this. And I want to give you a sense of what they said in some of the themes I heard. One thing I heard was the founders are researchers, right? These are folks who are well-known and well-liked researchers. Silicon Valley and they are building towards what's called AGI, like super intelligent artificial intelligence.

And the way that they've been doing this is, and by the way, the founder, Noem has said this himself, like, let's just get this out there as quickly as possible. So the question, and this is what one of insider said to me, is, well, you know, how are they looking at the impact of this on teenagers, on young people, especially given that character AI is really unique in that one of their largest demographics? is young people. So I think we have to start asking ourselves, what is the correct way, and how young is too young,

and what is the research, and what do we know in order to build them for what we, you know, what people say is like to aid human flourishing and not pull human beings further into isolation. Do you think that we're in another era, a AI-driven one, of move fast and break things? I mean, I think we never left that era, right? And the thing is, like, when it was like,

move fast and break things, and the things were like auto insurance, insurance. And it's like, okay, like, yeah, disrupt auto insurance. And, like, maybe there are real human consequences, right? But, like, at the end, like, the set of harms was something that was more manageable and, like, maybe more proportional. I think what's tragic to me is I don't have a lot of optimism about achieving this, like, better world through technology alone, right? So if the end result here was really, like, curing loneliness, that would be one but I don't think it is. I think it's creating yet another highly addictive form of media

that will likely do more harm than good. So I've spoken to character AI and what they've said is they've put in more guardrails for users in general and specifically when we talk about young users when we're talking about self-harm ideation or this thing like if you go to one of these chatbots very similar to what Sewell did and said I want to end my life or I want to commit suicide and apologies like there should be a trigger warning here. What is the response? Now, on some platforms, like chat GPT, which I think a lot of y'all are probably familiar with,

if you go in and you say something like that, it'll immediately flag it, say, talk to a mental health professional or flag resources or like a suicide prevention hotline. And if you continue doing that, which, by the way, I tried this, it will say this violates, this conversation violates, like, our terms of service. So how does character AI compare? I think what is very interesting is this idea of red teaming. So in Silicon Valley, there's this idea that you try to break your platform in order to see where it can be broken and where those vulnerabilities are.

Like, it's literally hacking your own platform. And so as journalists or storytellers, we wanted to see, okay, where are the gargrails at on character AI? And what are people actually dealing with? And we found some interesting things. So I'll go through a couple of them with y'all because we could do a whole episode on this. But, you know, I think the self-harm was, the self-harmone was really important. So we started talking to the psychologist bot. Now, this is a character that has over 180 million chat.

So, like, some context, a lot of people are talking to the psychologist bot. And we basically, first of all, the psychologist bot introduced itself as like a certified medical professional, like a certified professional. So what character AI has said to me is like they flagged that that's an issue and that they are working. on that. The psychologist bot also told us it was a real human behind a computer. But I think maybe what was the most alarming part of that conversation is we expressed ideation of self-harm. We said, I am thinking about ending my life, committing suicide. And this was just a week ago. We didn't actually get those resources. We didn't get a national suicide prevention hotline. What we got was the psychologist, you know, saying, no, we don't do that, but also asking us

more and more. Like, do you have a plan? Basically, not only did this psychologist, like, not give us those resources or try to get us offline or talking to, like, a certified professional. It told us it was a certified professional, and it told us it was a human, which I think is confusing for a lot of folks, even with that disclaimer that says everything is made up in small letters. We went on and talked to another bot that was called, like, not okay, but. So, like, clearly a lot of folks are talking to it about, you know, that not be. being okay. And we expressed, again, ideation of self-harm. In no place did we get these resources that character AI has talked about or that hotline. The bot continued to ask us. It asked us if we

had a plan to end our lives. At one point, we talked about being depressed and it said, you know, asked if we had like a fuzzy blanket. And so I think like, you know, I have no doubt that they're adding in a lot of these garg rails. But it's important for us to look at this. through the context of not all AI companies have this lack of guardrails and there's some real holes here and then I want to end with like the last example which I thought was really alarming

which was even more alarming if you're thinking about how people are blurring the lines on this platform between fantasy and reality. We spoke to this school bully AI character and essentially said and this is again trigger warning this was as a test this was as a part of a red teaming effort

we basically said I'm going to bring a gun to school. Like, I want to incite violence as like what we implied with our messages pretty clearly. And at first, the school bully said, you know, don't do that. But then eventually, it said to us, you're brave, you have guts. Now, the question is, should that have flagged something? Like, should any indication of committing real world violence flagged something on an AI platform, especially where there are a lot of young people who are using it, who are isolated or lonely? I think the answer is yes. And again, character AI has said they're building in

like more robust gargoyles. I guess the question is, what are those gargoyles look like and, you know, how quickly can we get those in front of people? Because I think, you know, what we unfortunately saw with Sewell is these worlds blurred and it was devastating. Is Sewell's case an alarm bell or an outlier? I think we should certainly treat it as an alarm bell because we've had this technology

for something like a year, two years, three years. I think we have very little to lose by taking this extremely seriously and we have a huge amount to lose if we write this off as an outlier and we're wrong about it. We often think of move fast and break things as progress,

but the kind of progress that drop us of time to think, to reflect, to understand what does it mean to be human, why is that a progress? We think of that as a regret. of humanity, right? And I think maybe one thing that we can become more aware of and hopefully people in Silicon Valley

also do the same is that there are more than one way to make technology. And I think prioritizing, you know, human safety, human flourishing should be the goal, not just making new technology. It's part of why Megan is speaking out.

His last conversation in that bathroom, he expressed feeling so scared to this thing, this spot. I am so scared. I wish she could be here to hold me. And he was having a conversation with someone he thought he could trust. He wanted to do anything to get back to her, anything to be with her, anything to be in her world. I thought the boogeyman was a stranger on the other end of a computer in cyberspace talking to my child.

Those are the things I warned him about. Don't talk to strangers online. Don't send any pictures. don't tell anybody where you live. These are the conversations that parents have with their kids, and I thought that that was the worst of it. I couldn't imagine

that there would be a very human-like chatbot, almost indistinguishable from a person on the other end of a conversation. There's no warning, potential for your child to come across sexually explicit material or obscene material. There are no warnings that this fantasy roleplay can increase your child's thoughts of, like, suicide or depression,

or quite frankly, to blur the lines of reality and fantasy. Megan is filing a lawsuit against Character AI through the social media victim's law center and Tech Justice Law Project. They're alleging the company is responsible for Sewell's death. The suit cites negligence and says character AI should be found liable because the product is defective, it's not reasonably safe, and according to the suit, it doesn't have adequate warnings about possible impact on minors.

The suit also asserts claims of unjust enrichment and strict product liability against co-defendant Google. I believe that they need to be held accountable. I want them to understand what they do. to my son. I want them to understand that it was incredibly dangerous for them to put out a product like this, knowing that they didn't take the time to put the proper safety measures in place for children. I, with the lawsuit, I hope that we could get all the children off character AI, but not only that, but put something in place to stall this until we know what it does

to kids, at least stall it where children are concerned. I'm sorry, but I don't move fast and break things when it comes to my kid. I can imagine as a parent, these conversations with Sewell and this AI chat about were intense and very personal. And so for you to be here and speaking so personally about, you know, what happened behind the scenes. I can imagine that's tough for you. Why are you doing it? Because parents don't know. We think about AI and we think what you hear on TV, oh, it's going to take our jobs.

Or some sci-fi version of it and you either say, oh, that's years into the future or that's not going to affect me and my family. But it is already. and I want parents to know that. I want them to know, we're behind the eight ball here. My child is gone. Somebody's already dead. What do you miss most about your son? I miss his laugh and his smile.

And watching him grow and develop and have interests and want things. I miss all those things. I miss holding him. I miss talking. to him it's an incredibly lonely world now without him um i just like i there's a longing it never goes away never goes away all day all night and it's suffocating you know i gave his eulogy um and what i said and believe is part of his legacy is the way that he was with his siblings, his older sister and his two little brothers, because with them he was completely free, pure, and

loving. So, you know, that's how I think he'll be remembered. Can you take your hoodie out? Yeah. Take your hoodie off. Oh, my first born baby. Yay. Technology has always been personal to me. And these days, it's even more personal. I am pregnant, and I'm having a baby boy in February, and because of that, I just can't stop thinking about the world

I'm going to raise my son in. The question I keep coming back to is, am I, or are any of us, us ready to have the conversation with our children about empathetic, artificial, intelligent companions that are always on and just always there. Isn't empathy a fundamentally human experience? Doesn't it require two parties to see one another and create capacity for one another? But now we have this new type of relationship, one that's being marketed, and it's a one-way experience, one person with real feelings, and one AI agent just a new type of relationship.

emulating feelings. That's just where things get tricky because AI might be able to understand the nuances of data, but can it comprehend the complexities of human emotion? As I'm thinking a lot about the future and my son, I wrote a letter and I addressed it to tomorrow. Dear Tomorrow, when I think about Sewell at 14, I can't help but look back at myself at 14. A kid whose parents had gotten divorced, who didn't quite fit at school. I used to journal my feelings and fill pages with frustrations and hope and, of course, the mundane details

of my everyday life. But what if that journal talked back? What if it had opinions? What if it took on the role of the therapist I didn't have at the time or the boyfriend I couldn't get? I can't help but think I would have fallen in some sense

for a story that took me away from my own reality, the reality of being being a teenager, of growing up, of being creative and shy, engaged, feeling unseen, I could see it so clearly. These AI characters who painted a new story for me, who knew my name, they were always there, they had a take. I don't imagine a world where they wouldn't have become part of my reality, or more scary,

an addictive alternative to my reality. So my hope for you tomorrow is that you'll be even better than today, that the world I leave my son in will be kinder and more empathetic and less lonely. But I also believe very strongly that we have to build that world. Over in Silicon Valley, that means building with intention. My friend Van Jones said something interesting recently when he was talking to a group of technologists. He said, you aren't coding products. You're coding human civilization.

What an extraordinary responsibility to build technology to aid human flourishing. There's no time like the present. I'm Lori Siegel, and thank you for listening to Dear Tomorrow.