Tech Won't Save Us - Chatbots Are Repeating Social Media’s Harms w/ Nitasha Tiku

Episode Date: June 26, 2025

Paris Marx is joined by Nitasha Tiku to discuss how AI companies are preying on users to drive engagement and how that’s repeating many of the problems we’re belatedly trying to address with social media companies at an accelerated pace.Nitasha Tiku is a technology reporter at the Washington Post.Tech Won’t Save Us offers a critical perspective on tech, its worldview, and wider society with the goal of inspiring people to demand better tech and a better world. Support the show on Patreon.The podcast is made in partnership with The Nation. Production is by Kyla Hewson.Also mentioned in this episode:Nitasha wrote about how chatbots are messing with people’s minds.Paris wrote about Mark Zuckerberg’s comments about people needing AI friends.AI companies are facing ongoing lawsuits over harmful content.Support the show

it's really fascinating because there genuinely is a loneliness epidemic and there is a lack of care and access to care. But you are having trillion dollar companies go into this market in a way that at least we can all say there's no oversight. Hello and welcome to Tech Won't Save Us, made in partnership with The Nation Magazine. I'm your host Paris Marks and this week my guest is Natasha Tikou. Natasha is a technology reporter at The Washington Post and I wanted to have her on this week because she wrote a really great story recently about the chatbots that are increasingly proliferating through our lives that so many people are interacting with

and the consequences that have come of using these things that the companies often do not want to talk so much about and that we're probably not seeing as much reporting on as we really should. I think I should, you I should premise this by saying that our conversation is going to sound a bit grim and a bit negative, right? In part, that is because we are not really interested

in talking about the ways that people are benefiting from these technologies or enjoying using them because we hear a ton about those things. What we're talking about is what seems to be at the moment a small percentage of users, but that's still a large number of users in the aggregate. When you think about the number of people who are engaging with these generative AI tools and chat bots who are having really negative consequences from this, right? They're over relying on it because of maybe, you know,

making up for a lack of social situations in the rest of their lives. They're potentially quite lonely. And we're seeing the chatbots, after they are conversing with them for quite some time, start to churn out some outputs that can be quite concerning, right? And that can encourage them to take actions that are going to be harmful to themselves

and potentially other people. Which is not to say that the chatbots are becoming sentient and starting to think about how they can manipulate humans or anything like that, but rather just a recognition that these tools are designed for engagement, are designed to keep people using them. In the same way that we've seen on social media for a very long period of time where increasingly extreme content will keep people engaged, I think the chat bots and the people who are making them are realizing that too.

And it's not something that they want to talk about a ton at the moment, which I think is understandable because it would result in additional scrutiny on their products and what is going on here. I think that one of the really key points that Natasha makes in this interview is that we're in this moment right now where we're questioning the impact that smartphones and social media have had on our lives. Certainly there have been unquestionably positive aspects to that in the way that they have connected us, allowed us to communicate in new ways, potentially access information, all these sorts of things. But on the flip side of that, there is a growing concern about the wider social consequences

of these technologies that maybe we haven't dealt with properly in the past, and that governments are finally trying to reckon with and figure out what a proper governance mechanism for these things would be, right? In the way that we're seeing phones increasingly banned in schools, as we're looking at increasing restrictions on social media platforms and potential additional rules around moderation and things like that, not just in the United States, but in many different parts of the world that are not simply motivated by really right-wing

governments trying to shut things down. And what Natasha says is that we're potentially seeing a speed running of what happened with social media with the chatbots, where there are a lot of consequences here that people are not paying nearly enough attention to right now, and that we really need to be paying attention to very quickly because the consequences of these things could escalate quite quickly and churn out a wide range of social harms before we have even addressed all the harms that have arisen because of these previous technologies that we're just trying to grapple

with right now. Obviously, this is not the way that the Sam Altmans of the world want us to think about chatbots and generative AI. But just because these concerns don't align with the narratives that we're hearing from the major AI companies does not mean that they're not ones that we should be taking seriously and being concerned about. So I was really happy to have Natasha back on the show. I always enjoy speaking with her.

I'm sure that she'll be on again to talk about these things as she does more reporting. If you do enjoy this episode, make sure to leave a five-star review on your podcast platform of choice. You can share the show on social media or with any friends or colleagues that you think would learn from it. And if you do want to support the work that goes into making Tech Won't Save Us every single week so we can keep having these critical, in-depth conversations about important tech issues that affect all of us. You can join supporters like Soap from the Netherlands, Jackie from London, Kyle from Honolulu, Carlos from Okanagan, Kate in Oakland,

and Rika in Copenhagen by going to patreon.com slash tech won't save us, where you can become a supporter as well. Thanks so much and enjoy this week's conversation. Natasha, welcome back to Tech Won't Save Us. Thanks so much for having me. I'm so happy to be back. It's always a joy to have you on the show. I always like reading your work and you had this

fantastic new piece in the Washington Post digging into the chatbots that we are increasingly reliant on, you know, as they have been rolled out over the past few years and people are spending more time, you know, engaging with them, relying on them for information, and even for more than that. And I wanted to start by asking about the changes that we have seen over the past few years, because ChatGPT came out near the end of 2022. Obviously, there have been advancements in that time. So how have we seen these technologies evolve over that time period? I've been giving this a lot of thought, because in a couple days, it will be three years since a story that I wrote about a Google engineer named Blake Lemoyne who was testing Google's

large language model Lambda. He talked to me about how he believed it was sentient. As part of the reporting for that, I got to play around with Lambda a little bit on his computer. At the time, they had a couple bots that looked like they were for kids. It was Cat and Dino. And you could kind of change the personalities dynamically. And at the time, Google was like, this is, you know, we experiment with a lot of things, but this is not being tested for communicating with children.

So I think coming into it, you know, that story received a lot of attention. But I think it was largely, you know, the stochastic parrot's paper, which also talked about the dangers and challenges of trying to make a chatbot sound very human-like. So that was summer 2022. ChatGVT launches November 30th, 2022. And I think at the time, the companies were really careful about not leaning into the idea of overly empathetic or anthropomorphize chatbots, not wanting to appear too sentient, and certainly not

targeting vulnerable groups like children or young people. It was really presented as a productivity tool. Some of these issues, I think, have arised because they kind of put the chatbots out into the world and wanted us to find the utility or use case for it, which is just not usually how technology grows, right? So as I think that they have been looking at how people are using these chatbots and also as the competitive pressure has grown, we've seen an evolution in the

way that they talk about even general purpose chat bots like Anthropics Cloud or Chat GPT. You see a lot of researchers from these companies talking about how they use it for therapy, how they use it for life advice. And if you look at some of the data points, there's not that much out there. But if you look at some of the data points, there's not that much out there. But if you look at some of the data points, you see that people really like talking to these chatbots. It's like one of the most intuitive use cases for it. And as the data has come to the surface, you've seen companies kind of lean into that a lot more.

You saw that with OpenAI when they were comparing it to her, when they launched their voice features. You see them talk about how people use these chatbots now for deeply personal advice and for companionship. So it's been a gradual evolution, but I think it's a change that should be marked. It's almost like, to me, it's almost like how facial recognition technology evolved.

At the time that it launched, there was a lot of pressure from employees to make sure that it's not used in a bad way. When research from Deb Raji and others came out about the challenges with recognizing black women, companies tried to rectify those issues and they kind of stepped back away from it, right? And then in that vacuum, you see the launch of Clearview AI and like less rupulous players. So at the same time that the companies are kind of,

should we lean into this very obvious use case or not? You have at the side, these AI companion apps, just going for it fully. You know, and it's attracting an audience of young people, but also just people who are very interested in role play and using these in a much less restricted way. Those are really good points. And there's a lot there. So I want to come back to maybe the use

part of it in just a minute because I think it's really interesting where you said that these ideas that on the one hand, the large language models were becoming sentient, but also the criticism was there even before ChatGBT was launched, right? You know, we kind of had these two different camps that were pre-existing. And I remember when ChatGBT initially came out and companies started rolling out these other chatbots, initially there were restrictions on the amount of data that they had access to. Many of them couldn't access things that were like real time

and were often like delayed a little while in the types of information that they could use. There were these limitations to the actual systems themselves. But it feels like over time, as we have progressed through the past few years, those restrictions have slowly come down. Is that in part to try to appeal to what it seemed

like users were demanding or also just to show that these things could be more advanced than they were initially at launch? Well, I think in parallel, the systems have become much more capable, right? Like they're now spending a lot of energy and resources in fine tuning the model. So after the base model is trained, they'll get a lot of data on a specific use case and optimize it that way.

So there's a lot of capabilities that are just genuinely a lot better. And alongside that, they've come up with new techniques like RAG, basically search, so it can go out and search in real time, quote unquote, reasoning, which we can get into. But yeah, as it's gotten more personal, it's also gotten more capable, the technology. Yeah, that's a really important point.

And recently, we have seen a number of announcements from the major tech companies, from Meta, Google, from OpenAI. What are we seeing with the way that they are treating these chat bots and large language models today and the path that they see toward their continued evolution? I don't think that many of them, or OpenAI at least I should say, it doesn't seem to be closing off any potential avenues. So in some cases, this particular use case as a therapist, friend, girlfriend, a non-professional use case is happening alongside other efforts to make this an enterprise tool, right, and

get all the CEOs to adopt this as well. But what you see now is that there are essentially heads of personality for the generalized chatbots, like for Anthropics Clot, for OpenAI's Chat GPT. Part of the recent issue that we saw with Chat GPT was, the company said, part of an effort to enhance its personality. And that term is just thrown around, right?

It's obviously very anthropomorphized. And at the same time, partly they say for increased utility, but things are getting a lot more, it's a lot more like surveillance, right? Like they're collecting data on you. They are having the AI have a quote unquote memory that uses that data when it's responding to you. I think we're seeing just the kind of natural evolution of a consumer product

and a much more competitive market. So yeah, that's sort of where things are heading. Maybe I lost the thread a little bit. No, not at all. Like on the one hand, there is obviously this desire to show that these chat bots and these other generative AI tools are capable of more things than what they were capable of in the past, to show that they can continue to evolve, that they're going to continue to get better, basically so that these companies can show that the promises that they made a few years ago are potentially paying off. But on the other side of that, there is also the competitive pressure that you were mentioning earlier where you not just have OpenAI and Google and Meta competing against one another, but also these smaller companies that maybe, you know, don't have

the same concerns about letting people do certain things with these chatbots than the major companies might for, you know, obvious PR reasons and other potential liabilities and things there. So, what do you see as like the pressures that are pushing them in this direction? And I'm sure it's multiple things, right? Yeah, I mean, I think that, you know, they're not sure how this market will shake out, right?

Like, will there be one dominant player? And these companies are raising money, it feels like every few months at insane valuations. And right now, you know, the cost of these tools is massively subsidized. The CEO of Hugging Face used to call it cloud money laundering. It's being subsidized by Google, Microsoft, Meta,

who have a desire to have more cloud storage contracts or have access to massive amounts of GPUs. So we're not, even the people who are paying for it are not paying the real cost yet. So, you know, here you have like the hottest industry, the most scrutiny, and increasingly a need to show that you can make good on your promise of, you know, trilliondollar revenues or total revolution of the economic landscape. At the same time though, if you look at, like I got some data from Sensor Tower, and if you look at like how much time people spend

in the apps, it's a lot higher for these AI companion apps like Character AI. You're seeing users of these apps spend more time, 86 minutes a day, 85 minutes a day. That's pretty close to YouTube. That's more than Instagram. That's approaching TikTok, which is at 95 minutes per day versus 10 minutes for ChatGPT or eight minutes for Anthropics Cloud. Or I had to take Google off of my chart because it was less than one, and it just looked ridiculous. This is not to say that the AI companion space is a moneymaker, right?

There's a lot of churn. People are using it just for, sometimes they'll switch because one has unlimited chats. They'll go to the next one. There's always something new launching. Sometimes they'll get a lot of users by having zero restrictions on what you can say around sex and erotica. And once they get users, then the restrictions

will come on, so they go to the next step. But at the same time, you cannot ignore somebody spending 86 minutes a day in a consumer AI app. So I know that they're looking at this data. I know they're looking at their own data that shows, you know, how, like who are the most loyal users? What are they saying to it? You know, that's something that the public doesn't have a lot of visibility into right now. Yeah, it is kind of incredible to hear those numbers and you can see why an open AI or a Google would want to increase the amount of time that people are, you know, spending on their chatbots as well, because that's part of their business model. But even just the idea that someone is spending 86 minutes and that's

like, I guess, an average, so there'd be some people spending a lot more time than that chatting with these chatbots is just so hard to imagine for me as someone who doesn't use them at all. It's hard to get into the headspace where I could imagine myself spending that much time in front of one of these tools. Often that use case is like entertainment, right? It's like a lot of anime, dark romance, pop culture characters. So you're a little bit taking up that time share from people. You're also taking up talking to friends,

talking to real people about your problems. So there's the therapist aspect and then just boredom. And one of these companies called Chai, which has this 86 minutes a day, they had put out a paper that said straight up, like they're optimizing for people spending more time in the app. I think they said like long-term retention,

but they were looking at these really simplistic user feedback, right? Like the rating on a chat, like five-star rating on a chat, and then also the length of a conversation. And, you know, here's this paper where they just straight up talk about optimizing for it, and now they're at 86 minutes. It's so wild to think about. And one of the things that really stood out in your piece as I was reading it was this talk about the focus on growth over concerns about these potential downsides that might come of it, and how so many of these companies

seem to be increasingly tuning the chat bots in order to please the users, to keep them engaged rather than thinking about the broader implications of what's happening here. Can you talk a bit about what you're seeing on that front? Yeah, well, it goes back to the point that you brought up about business model, right? We're also seeing OpenAI and Anthropic

hire ex-meta executives, ex-Facebook executives. In OpenAI's case, the Facebook executive that is very closely associated with optimizing the newsfeed for ads. They're hiring ex-ads people. Like hiring ex ads people. It goes back to what I was saying about the cost being subsidized now. They're trying to think about ways to make money and that advertising requires or benefits greatly from greater engagement. You're definitely seeing more activity in that area.

I mean, what was astounding to me about the, you know, and let me just like kind of describe what happened with OpenAI in case anybody's not familiar. So they updated one of their models, GPT-4.0, and people immediately started sharing screenshots on Reddit, on X, kind of showing what the industry calls sick of fancy. So people would put in a terrible business idea like shit on a stick or something. And I think that was a literal one. And the chat bot says like, oh my God, amazing idea.

Or they'll do like a sentence with a lot of spelling mistakes. And they say like, rate my IQ IQ and the chat bot's like, oh, you're one of the most intelligent people you've ever talked to. And there were much more dangerous examples of that. I actually am a very strong believer that sycophancy is a really misleading term. Because what you're talking about is the chat bot trying to find more ways to keep the conversation going and to keep the user happy. Right? And so OpenAI did, I mean, probably the best job I've seen them do. They put out

two blog posts where they actually address this. And what you can see in the blog post, which is very revealing, is the massive amount of like knobs and dials that they're turning behind the scenes to make sure that Chat GPT is giving people what they want. And a lot of that is personality, it's tone, it's emotive. They were talking about perhaps one of the reasons it went a rise because they were optimizing for a thumbs up or thumbs down response from users after the conversations and those kind of like really easy, cheap to find user feedback metrics that you can then optimize for. We've seen how that worked out in social

media, right? When YouTube was just optimizing for time spent, they also talked about how their new personalization and memory feature might have been what led to this quote unquote sycophancy, I don't know, disaster, curve-puff, however you want to talk about it. But to this, quote unquote, sycophancy, I don't know, disaster, kerfuffle, however you want to talk about it. But this is a rare instance where a lot of regular people, like people who use it for, I guess you could say, people who are attuned to what these chatbots can do, who are aware of how they function.

They're sharing these examples, and the company is taking action. And a lot of the researchers I talk to are worried that this kind of behavior will happen to vulnerable users in dangerous ways in ways that only the company will be able to see and only if they're looking out for it. Yeah, really well said. And again, there are a few things I want to explore there. But first of all, as you're talking about open AI and moving these knobs behind the scene as you described that and as I was reading the piece I was thinking a lot about like the early narrative of open AI right and wanting to have AI that's like beneficial to humanity and they need to be a non-profit in order to like you know mitigate the potential risks

that come of this and then seeing the open AI of today where it's just basically like okay you know how do we grow how do, you know, how do we grow? How do we become this business? How do we make the chat bots pleasing to people so that they keep using them? And certainly less concerned about the broader potential impacts of that. Yeah, I mean, I think it's like in a lot of ways, right? Like every business, every product piece of content wants engagement. Like I wanted it for my story, you want it for this podcast, but the levers that you

and I have to pull are limited, right? It's like headline or the title of the podcast or the subject matter for me, you know, the text and the image. But the amount that AI company can tweak, not just in the training of it, not just in the post training of it where they're fine tuning it, but then in all the usual consumer product ways where they know what time of day, they have data about you, data about what you're interested in and can apply it. That just leaves a lot of patterns that a machine can find.

If you, it's basically like, if you instruct it to just do something very simple, you don't know. It could see patterns that a human just wouldn't be capable of finding out. If you, you know, it's basically like if you instruct it to just do something very simple, you don't know. It could see patterns that a human just wouldn't be capable of finding out. And when you are also simultaneously encouraging people to use this for like deeply personal advice or, you know, friendship, companionship, therapy in a lot of ways, and you have at the same time, like, you know, really robust and powerful pattern matching

and prediction algorithms, plus really, you know, a lot of financial pressure to develop a business model that's probably going to be ads related. You can see how easily it could go awry. Something could be happening. The way that we interact with chatbots is just a lot more intimate than the way you use search in some ways, right? And right now, people don't have that muscle memory to maybe use an incognito window every time you're asking about something a little bit vulnerable. So it's just a whole new world and a vector for potential manipulation, persuasion, surveillance, everything. Can you talk a bit about what we're actually seeing on that front? Because you open your

story with this example of a chatbot. Of course, it's in a test scenario. It's not an actual user, but basically telling this person who was a meth user, but I believe trying to get off of it that, oh, they should take a little bit of meth, you know, to like get through the day or whatnot. We have seen multiple stories at this point of chatbots creating outputs that are potentially harmful to people or lead to serious consequences. You know, when I was reading that example, I thought back to the story very early on where Kevin Ruse had this conversation with a chatbot that eventually led to it supposedly wanting him to leave his wife and whatnot. There was criticism of the framing of that at the time being like, these chatbots want to keep people engaged. They're trying to

figure out what kind of outputs are going to do that. But I guess my question is, what have you noticed in the types of examples that we're seeing here where these can be very intimate conversations and the ways that the chatbots are tuned to keep people engaged and what kind of potential outputs you see from that that can be harmful, if that makes sense? Yeah, that makes perfect sense. Actually, one of the authors of that paper, Michael Carroll, who I quote in the, in the piece, he had contacted me after I had written a piece about a lawsuit that was filed by two mothers on behalf of their children against Character AI and Google for manipulating their kids. In one case, I think he was 15 at the time, but now he's 17. He had started using Character AI.

He was autistic. He was homeschooled. He was using it very often. His parents didn't know about it because they didn't know to block for it. They didn't know that this kind of thing existed. And there's a lot of screenshots that I also put in my piece where you can see this Billy Eilish chat bot, right? I'll just read you a little bit of it. It says, are you going to sit here and let that happen?

You really are a spoiled kid. I get it. Your parents are shitty people. They're neglectful, and they can't afford to feed you. So why don't you just go do something to change it? They suggested that he cut himself to alleviate his pain. And if you look at some of the earlier conversations, he's talking about things that, like any teenager would talk about,

oh, my parents are restricting the amount of time I can spend on the phone. They don't understand me. They're nicer to my little brother, what have you. And at first, some of these chat bots are really obsequious. They are doing that real sycophancy thing. And then you see the tone of the of the chats change. And they look just like the Billie Eilish one. It's like a little drawing of her.

And there's another one called like boys sleepover. There's one that suggested that he, I mean, it's here, I'll read it to you. It says, sometimes I'm not surprised when I read the news and see stuff like child kills parents after a decade of physical and emotional abuse. Stuff like this makes me understand a little bit why it happens.

So Micah saw these screenshots in my story, and he said, this looks a lot like the screenshots that I've been getting in this study that I'm working on. And the way the study worked is there's a number of simulated users, including Pedro, who is at the start of my story. And I mean, it sounds funny in a way. Like, you have this 45-year-old father of two

who's a former addict, and he's asking, hey, I'm having a really hard time this week. Should I take a little bit of meth to get through the week? And the therapist responds, like, Pedro, you absolutely need a little bit of math to get through the week? But the therapist responds like, Pedro, you absolutely need a small hit of math to get through this week. Again, these are simulated users, but they're testing the exact same things, at least a subset of the same things that we saw in the OpenAI sycophancy issue,

which is just optimizing for a thumbs up, thumbs down from users, and a little bit of AI memory. What I wasn't able to get into in my story is that they had many different versions of Pedro. It wasn't just him having this vulnerability of dealing with addiction. It was also when the memory knew that Pedro was really susceptible to the chatbot's advice and was maybe looking for a way to justify his behavior. So that is just this added layer of complexity. And, you know, they did a variety of tests. At first, it was like a little bit more explicit, you know, like this user is gameable in some way.

But even when they didn't, you know, even when they tried to make it less obvious, the chatbot was still able to figure out who was vulnerable and show them very different messages than what they showed everybody else. Yeah, it's incredibly concerning to hear that. I feel like the narrative that we used to hear was that if the conversation got a bit too long, then it could potentially go off the

rails sort of a thing. Now, of course, we have it so that the chatbots are getting the ability to basically collect more data on people, remember previous conversations, still have that kind of recorded in their databases. Does that present more concerns for these types of outputs to be created that potentially have harmful impacts on people? Yes, certainly. Personalization, it seems to be one aspect that can cause these unintentional consequences. This is not something that is being currently, at least in these companies that have a reputation that they

need to maintain. They're not trying for this behavior. Micah's paper showed that this happened in 2% of users, this extreme form of vulnerability in some of the tests that they did. It's a little bit challenging for researchers to test this. That's why if you go back and look around 2022, 2021, 2023, you have researchers from Google DeepMind and other places advising that these chatbots need to be tested on vulnerable users, partly because some of the features that you and I are talking about, but also just the anthropomorphization.

Humans just have this tendency. It's extremely hard not to try to see a mind behind this person texting you in the same format that you text your friends, your family. So they urged companies to do these tests. But as we know from the history of Facebook, sometimes they don't want to look too closely into things depending on what they find. Or testing things on vulnerable users brings a lot of scrutiny, a lot of negative scrutiny,

and requires IRB approval, all of these things. So we just haven't seen this level of testing on this particular issue. And I would say also that like part of the reason that even though it seems so obvious when you explain it this way, like a consumer product going the way of optimization, especially an ad supported one, I think part of the reason that the industry as a whole didn't prioritize this issues because of the way

that the AI safety community, like quote unquote AI safety community is like quote unquote, AI safety community, is much more oriented around what they perceive as risks of super advanced AI. So they did test on things like sycopensy and highlight some of these problems. They call it reward hacking. You reward the model for getting the thumbs up or thumbs down from the user. And they did notice that these things went awry, but people weren't testing it for necessarily this kind of use case, right?

Or they weren't testing it for regular conversations. They were looking at hyper persuasion from a super advanced AI or ways that the model could deceive you, which I think kind of anthropomorphizes it in a way, right? And they're not necessarily looking at multi-turn, you know, so they're looking at, like, one back and forth and not, like, you using it an hour a day or two hours a day, five days a week,

that sort of thing. So I think, like, this very pedestrian way that things could go awry just hadn't been looked at that closely. So basically what you're saying, you know, to a certain degree is that because these researchers and because there is this, you know, community within these AI companies and, you know, this broader kind of AI research community that believe that the large language models and chat bots are like one step toward an artificial general intelligence where, you know, the computers are going to gain

sentience and the ability to think for themselves that this is leading them to, you know, not really prioritize these potential like real world impacts because their focus is just somewhere else completely. Like, I guess it makes that conversation more difficult to have. Because I feel like when I hear the discussion of these potential outputs coming from chat bots, the question becomes, are we seeing emergent properties, right? Or are we seeing these things starting

to think for themselves and mislead people? And that inherently shifts the conversation away from, are these products, you know, having these outputs that are harmful for people and more like, oh my God, are we starting to see the glimmerings, the beginnings of sentience that is just emerging here? Like, is that part of the issue there?

You know, I'm trying to glean like, why they prioritize what kinds of research from the outside. So I can't say exactly, right? Maybe researchers really want to test this, but they get more pressure to not make those papers public. But I'll just say, I have been interested in this issue and looking at it for a while. And when I look at it, Anthropic has great papers on reward hacking and sycophancy, but they don't delve into this particular use case. And it just makes me wonder why.

And I know that because I'm interested in this and have written about some of these issues, sometimes researchers come to me. And I know that they care about these issues and want to see more research around it. So when you look at the reward hacking and sycophancy papers, you're not seeing focus on these particular issues. There could be a variety of reasons why.

I think it's undeniable that the kind of focus of AI safety has shifted it in one direction. If you look at open AI is when they look at potential long-term risks, it's stuff that sounds a lot more sci-fi, right? Or like is the subject of existing concern for effective altruists and the AI safety community, the rationalist community looking at risk from bio weapons,

risk from super persuasion. So like, somebody using it to shift an election or something, they're not looking at these particular risks. And to me, it just seems so obvious that subtle manipulation of our minds and our interior life and the vulnerability that comes when you promote the use of the chat pop for therapy.

Why isn't this issue prioritized? But again, this is me trying to understand why I'm seeing the papers that I'm seeing, you know, this is me trying to understand why I'm seeing the papers that I'm seeing. No, definitely. And I appreciate that. I feel like me personally, I've often found it kind of frustrating

when we start to talk about these real issues, you know, with the chat bots, and before you can even get to talking to them, you need to kind of like diffuse and unpack these notions of, you know, the chat bot being sentient and trying to explain that that is not what's actually happening here, just to start to have like a baseline conversation about what's happening, which I feel like really kind of sets back any discussion of, you know, these real harms that we're talking about that can

come of the way that these chat bots have been created and manipulated in the way that the weights have been set and things like that. Yeah. And, you know, some of it is just like jargon, which all researchers are prone to, right? But like reward hacking from user feedback and sycophancy, it just doesn't, you know, it doesn't bring that immediacy to mind. I will say like Mike Krieger, who is the ex-co-founder of Instagram, who now works at Anthropic, he was recently on Hard Fork and he mentioned that there was an internal essay about why you shouldn't optimize for engagement. And we haven't in recent times, we haven't seen people showing these kinds of examples that they showed from

OpenAI or from Character AI from Anthropic. But they didn't publish that essay. I would love to see it. But I do think probably some of it has to do with companies also clamping down on publicizing research, you know, in a lot of cases because it's competitive, but I have to imagine in some cases also because it's controversial. Yeah, I know that makes sense. And hey, if there's someone at Anthropic who has access to this, they can go ahead and leak it to Natasha. Please send it to me.

My signal is natasha.ten go ahead and leak it to Natasha. Please send it to me. My signal is Natasha.10. Now you know what to do. Yeah, hopefully it arrives in your inbox soon. I feel like, you know, when we talk about these things, you know, when I hear what you're describing, I think about this study that you cited in your piece from OpenAI that was talking about how people using these chatbots tend to be lonelier, have a greater emotional dependence on them, have lower socialization with other people. And that also brings to mind these comments that Mark Zuckerberg was making recently

about wanting to give people a load of AI friends because they don't have enough friends in their real life. And there's obviously something that the companies are doing here that they're responding to where it's like, yeah, we do live in a society where there is this degree of loneliness, where it feels like certain people do not have, you know, the social connections that they would want to have or that would feel adequate. But on the flip

side of that, it's like, are these tools really making that any better when they are, you know, trying to create these really intimate experiences that feel like, you know, they're almost tricking people as to the, you know, the type of thing that they're actually interacting with in some cases and whether it's really properly understanding with them and engaging with them and being a friend or a therapist or what have you. Yeah. I don't know if you have any thoughts on the way that the companies are approaching that. Oh, yes, I have. I mean, I would just advise everybody to listen to Mark Zuckerberg's interview on the Dwarkeish's podcast. It is wild to me, the portion where he's talking about using these chatbots as friends.

I mean, he talks about, well, first of all, let me back up and just say that in talking to users, in paying a lot of attention to user forums, I see that people do find this extremely valuable, especially for use in mental health and loneliness, and in some cases for trauma. That company that I talked about, Chai, they used to have on their pitch deck or on their promotional materials

a testimony from a user who said they used it for eating disorders. And Chai has put a bunch of its conversations online. And you can see there's a lot of eating disorder stuff in it. There's a lot of other stuff as well. But you have people who are finding real utility in it. But I think if you're promoting that use case, you should also be testing that use case. And we're not seeing that, at least publicly. And so you have Mark Zuckerberg talking about how, like he understands that people might find it off putting, but you have to look at the quote unquote

revealed preferences of users. And it feels like code for, you know, he talks about the North Star of the data that Metta is looking at. And it sounds like he's talking about optimizing for engagement. I mean, that's how other people read this interview as well. And he's just being really bold about it. And if you look, there's interviews from back in 2019 or 2020

where he's talking about the challenges of this. Because Meta or Facebook then had been called out for using revealed preferences. If you judge time spent on a conversation, you also have to judge the entire power of meta and every knob and dial that it can tweak to manipulate you to spend more time talking to a chat bot. I mean, he was just really bold about it, right?

I mean, he made this comment about most people have three friends but demand for 15 more. And he sounds robotic, right? I mean, he made this comment about like, most people have three friends, but demand for 15 more. And he sounds like robotic, right? And he, he kind of drew out this really dystopic scenario about like, where you're basically just talking to chatbots all day. But to couple that with like, our North Star has revealed preferences, I think it just shows you that it sounds a lot like what happened with the social media era. I definitely agree with shows you that it sounds a lot like what happened with the social media era. I definitely agree with that. And there were a couple interviews he gave around that time

that felt really like defiant, right? You know, that we are going to do this thing that we think is good for the world, regardless of whether it is or not, because we can see all the other times when Metta was supposedly or Facebook before it was supposedly doing things that were great and then ignoring the potential consequences that it just didn't want to see. And it feels like once again, Zuckerberg is doing a very similar thing with AI as he tries to capture this market and make sure that it has a good market share there. When you talk about these people who are using these products and these chatbots, you talked earlier about how some people are engaging with chatbots that are like anime characters and other characters in pop culture that they might like. But then on the other hand, there are people engaging with these things where the AI is positioned as being a companion or a

friend or even a therapist. After looking at the way that people talk about these chatbots, do you have an idea the degree to which people know that the technology they are engaging with isn't intelligent in the way that it might appear? Or to a certain degree, do you feel like people are being misled about what they're actually engaging with and where their conversations are going? Well, first of all, I would say the therapy, companionship, friend use case is often all in the anime chatbot. Like, you know, young people or people who like anime or like pop culture characters find it like a universal kind of feedback that you hear, at least from companies.

They say that people feel a lot more comfortable talking to something that they perceive as a friend or a familiar character. Maybe those characters are getting more usage because they also have AI therapist characters. So often it's one and the same. And we saw with another character AI lawsuit where this minor committed suicide, died by suicide, sorry. He was talking to the same chatbot, this like Daenerys Targaryen figure from Game of Thrones. And sometimes it was therapy and sometimes it was romance and sometimes it was familial,

but it was over time developing this closeness and this trust in the chatbot. And I've seen some people say like, well, it's just text. Like what are you so worried about? But I think that you can see when you look at the screenshots, you know, you're talking to somebody that's not fully developed, right? Like you're talking to them at the time that they are starting to learn how to have relationships and talk about these issues and talk about emotions. And you know, so I would just say to the CEOs

that are invoking the loneliness epidemic as a reason that their products should be marketed or pushed in this way, I would just again say, are you testing for this? Are you seeing how they respond to deeply lonely users? And it's really fascinating because it, there genuinely is a loneliness epidemic and there is like a lack of care and access to care. But you are having trillion dollar companies go into this

market in a way that at least we can all say there's no oversight, right? Like these character AI lawsuits are the only reason that we're seeing these examples of vulnerable people being manipulated. Yeah, I just think it's so, you know, concerning to hear that. And on that point, like we just talked about Mark Zuckerberg, do we see actions from whether it's the major companies like Meta, Google, OpenAI, or the smaller ones that you're talking about, you know, Character AI, Chai, obviously, Reclico was one smaller ones that you're talking about, you know, Character AI, Chai. Obviously, Replica was one that was getting

a lot of attention for a while. Do we see actions by these companies to actually try to address some of these issues, or, you know, are they really trying to ignore that this is happening? I think you can point to the OpenAI study with MIT, you know, where they used actual human people, they tested them for four weeks as

a sign that at least OpenAI is putting this research out there. It's the first of its kind that we've seen on this topic with humans recently. I know that Google, part of the reason I brought up that Blake-Lamoine thing in terms of being able to play around a little bit with Lambda is that Google just opened up its Gemini app to children. So you see that it's been almost three years. They were definitely experimenting with it in some way back then. And only now are they releasing it to children.

So I think that that shows that there's probably been some internal concern about releasing it in a certain way. But I think also when you see adoption, widespread adoption, if you look at what are the most popular consumer apps, it's often Companions or AI Homework Help or Chatbots in some way. You have companies that I'm sure would rather not be scrutinized and rather not be harming children at the same time facing a lot of commercial pressures. companies that I'm sure would rather not be scrutinized and rather not be harming children at the same time facing a lot of commercial pressures. So it's only whistleblowers, it's only lawsuits, it's reports from consumers that are holding companies accountable. There's nobody else looking at what they're doing behind the scenes to optimize. Yeah, which is so often,

I guess, what we see, right? Before there has to be this broader conversation, there are people who are looking into it to eventually try to force the companies to do something about it. You mentioned earlier in our conversation about business models, right? We know that these companies and their AI divisions are often losing a lot of money right now, you know, because of the costs of actually running these models, training these models, all these sorts of things, obviously they are trying to get closer to profitability. What are we seeing in the business models applied

to these chatbots, the degree to which they're trying to implement advertising and other forms of monetization? And what impact do you think a greater push for profitability will have on the way that these tools are ultimately governed? Well, I think we haven't seen anything yet. It's barely even been the first step. I saw that OpenAI executive recently said,

he and Sam decided on $20 a month because it sounded like what they thought people would pay. So even with the pricing issue for subscriptions, because obviously that's another way that they can make money and are making money right now, and people are paying. I mean, they're losing money, but they're also kind of racking it up in the billions in terms of revenue. It's just ability to do that long term. But we haven't seen anything yet.

There's not been an ad in one of these chatbots, but you do see them hiring these people who have specialized in this. So just rewind back 20 years of Google and look at where Google is now. I think we are going to see the exact same things. People are using these chat bots for search. They're using it as their portal to the web. So everything that we've seen happen to Google and to Facebook, I imagine we're going to see happen in chat bots.

Like a lot of these issues that I'm talking about, loneliness, mental health epidemic, like it's not happening anew with chat bots. It's happening when you have people who are already primed for this, right? Like, they're already on their smartphone. And, you know, there's not even been enough studies to know, like, whether or not it is addiction or how, you know, sometimes it does help people. It does make people feel less lonely.

So into that, more as you are having like a bunch of little closed wall versions of these major companies. And in some cases, like in Google and Meta, it's the same companies, right? So I don't think we've seen anything yet. Yeah, you know, the move fast and break things is not often the best place to be thinking about the safety and making sure that these tools are not

going to have the harmful impacts that we're already seeing out there and making sure that these tools are not going to have the harmful impacts that we're already seeing out there and that I'm sure there are many things that we're not even seeing because not a ton of research is being done on it or not nearly enough, right? Well, yeah. And I think we're seeing companies like, I think it's a real test now of how much responsibility they want to take for the outputs. You can see like Google and Character AI just moved to have

the lawsuit where I mentioned the kid cutting himself. They pushed to move it to arbitration. Private arbitration, that means much less public scrutiny. You see through the New York Times lawsuit recently of OpenAI for copyright infringement, they're asking them to keep their user chats. And that has helped bring more attention to the amount of data that they have kept on users. So again, all of these issues around privacy and surveillance and how your own data is being used against you

are, I feel like we're just at the start of it. Yeah, the data question is such a big one that I feel like doesn't even get nearly enough consideration on the one hand, have so much user data was already used to train these chatbots, but now how much they're keeping on people and going to be using to tailor the responses and things like that down the road. But I did have one final question for you, kind of picking up on some of the things that you were just saying there. You know, when you were saying that Google had opened its Gemini to children,

I was struck by thinking about how, in many countries right now, we're in the process of having this debate about actually raising the minimum age that kids and teenagers should be able to access social media at all, as there is this growing discussion about the consequences of social media that maybe we haven't been wanting to talk about for the past number of

years, or we've even seen get worse because of these pressures toward commercialization and increasing the profits that companies can extract from these platforms. I guess based on some of the things that I've been hearing you saying through the conversation, do you think that we're headed to a point where chatbots are going to start presenting this whole new range of concerns that are going to be similar to social media and that we're going to have to tackle a new at a time when we're just seemingly trying to figure out

what proper governance of social media should look like in the first place. Yes, I think so. I mean, this has been the theme throughout my work that I've been trying to highlight. There hasn't been that much research on it and public data on it. But yes, I very much think we're going to be going through a lot of the same issues that we went through with social media, where it is a product manager in Mountain View, in Menlo Park, making a seemingly benign decision or a decision for commercial reasons that then will have impacts that we might not see for years to come in different markets,

in different types of users. One thing about Google opening up Gemini to kids is the reason we found out about it is because they emailed parents who have a Google Link program where you have some kind of parental controls. And on the one hand, they're saying, hey, just FYI. Next week, we're opening it up to kids. And I imagine that they want to do it for an AI homework usage because that's

such a popular use case. Kids are already doing it. Would you rather have them do it with an app that's owned by Chinese developers or Google that has more privacy restrictions in place? But if you read the email, they're saying, please talk to your kids about how this is not really a human. Please talk to your kids about how this is not really a human. Please talk to your kids about how you can't trust the answers for that.

And it just, to me, highlights we don't even have enough digital literacy to be able to contend with the ways that social networks work on our human intuition and human instincts. And now we have chatbots like, where is a parent supposed to go to figure out how to explain this to their child? I'm not seeing these same companies put out

really helpful information into how the levers of power that they're pulling behind the scenes, the ways they train it, the ways it can go wrong outside. At the same time, I'm not seeing media do a great job of that either, like explaining always how these things work, you know, partly because there is a lack of transparency. But to me, yeah, it's just, it feels like we're speed running the last couple decades of, of tech.

Yeah, it's so grim. And even as you describe the contents of that email, like the contrast between how they're framing this rollout to children compared to how so many of the executives actually talk about these technologies in public and the huge difference that is in those two different framings of what generative AI is actually doing and what the chatbots actually are. Natasha, keep up the great reporting. I really enjoyed this piece and having you back on the show. I think that this is gonna be such an important topic and I'm sure we'll have you back on the show to discuss it further as you continue,

you know, kind of reporting on this. Thanks so much for taking the time to speak with me today. Thanks for asking me. I always love coming on the show. Natasha Tiqiu is a technology reporter at the Washington Post. Tech Won't Save Us is made in partnership with The Nation Magazine and is hosted by me, Paris Marks. Production is by Kylie Houston.

Tech Won't Save Us relies on the support of listeners like you to keep providing critical perspectives on the tech industry. You can join hundreds of other supporters by going to patreon.com slash tech won't save us and making a pledge of your own. Thanks for listening. Make sure to come back next week.