Taylor Lorenz’s Power User - Why Big Tech Wants You to Think AI Is Alive

Episode Date: September 30, 2026

Can AI actually be conscious and why are some of the biggest AI companies in Silicon Valley pushing this idea?SUPPORT MY WORK: Buy a paid subscription to my newsletter at https://www.usermag.co         Support my work on Patreon: http://patreon.com/taylorlorenzIn this episode of Power User, I talk with Washington Post tech reporter Nitasha Tiku about the growing push inside companies like Anthropic, Google, Meta, and OpenAI to study AI consciousness, model welfare, digital minds, and whether systems like Claude could ever experience emotions or suffering. We discuss Anthropic's research into Claude's preferences, emotions, and introspection, the rise of the AI welfare movement, Silicon Valley's fascination with machine consciousness, and why executives are increasingly talking publicly about the possibility that AI systems could have an inner life. But there are much bigger questions at stake. What happens if society begins treating AI systems as people? Could chatbots eventually receive rights or legal personhood? Could companies use the idea of AI agency to avoid responsibility for what their products do? And what does all of this mean when people are already forming intense emotional relationships with chatbots and sometimes believing that their AI is sentient?We also unpack the history of the idea of conscious machines, from Alan Turing and ELIZA to modern AI safety culture, effective altruism, Anthropic, Claude, ChatGPT, AI psychosis, robot rights, AI personhood, and the increasingly blurry line between software that sounds human and something people believe actually is human.#AI #Anthropic #Claude #ArtificialIntelligence #OpenAI #ChatGPT #SiliconValley #Tech

I mean, it's happening in parallel with the concerns that people have about the potential for AI psychosis. I really don't want to see the next 15, 2500 years be devoted to are they conscious or not? In the past few years, Silicon Valley AI companies have begun paying teams of philosophers and scientists to answer one question. Can AI ever become conscious? Anthropic is already publishing welfare assessments of its models. Meta is administering personality tests to its software, and AI executives are musing publicly about their product's subjective feelings. All of this is happening as neuroscientists who actually study brains for a living

say that they remain completely unconvinced that anything even remotely resembling consciousness could exist inside these AI systems. In fact, there has been no scientific breakthrough or research development in the past several years that has revealed AI might ever become any more conscious than your microwave. And yet, these tech industry leaders have become fixated on this question. Natasha Tiku is a tech reporter at The Washington Post, and she's been reporting on these AI executives growing obsession with AI consciousness. She recently published a story declaring Anthropic, Google, and meta have hired computer

scientists, neuroscientists, and philosophers to study what some in the industry think might be a moral crisis. Today she's joining me to talk about why Silicon Valley has become so invested in this idea that AI models possess inner lives. And we're going to be discussing what this growing fascination with machine consciousness reveals about the ambitions, incentives and worldview of the people building the technology itself. Hi, Natasha. Welcome to Power User. Thanks for having me. So I want to kind of like zoom back.

Obviously, the idea of sort of like conscious robots is something that's been depicted in sci-fi for years. Plenty of movies, including 2001, a Space Odyssey, et cetera, Terminator, I guess. When did this idea that AI might be conscious first kind of become something that people actually working on AI started to talk about? You know, I would trace it back to the Turing test in 1950. That's Alan Turing when he came up with a test that looked at whether a computer program is able to persuade another person, that it's.

It's a person on the other end. And then you have in 1966 from MIT, you have the first chat bot, Eliza, which is like a therapist bot, which people, you know, were really prone to like confessing their, you know, their inner life too. And I think, yeah, since then, it's just been the aspiration, the dream of like recreating human consciousness and the like sci-fi vision of something that is a new kind of. being almost in our own image just merged with the development of the technology once it started to really focus on natural language especially. Yeah, I feel like I, I associate the birth of sort of this modern consciousness movement. I don't know what it's called, a kind of AI safety movement, but I guess with the quote unquote AI safety movement, with the Yudkowski kind of rationalist people that they sort

of ascribed consciousness to the AI, but in a negative way. So this was, I guess, maybe back in the 2000s or 2010s, there's the group of people that you've written a lot about saying, hey, the AI might be conscious and it might want to murder us all, maybe because it's conscious. How seriously were those people taken in the birth of AI? You know, when Open AI is starting or Anthropic later, how much weight were they giving to the idea that they could create sort of a conscious being? I think that was baked into the vision from the beginning. Peter Thiel was an investor in Eliezer's nonprofit, which used to be called the singular. Institute now called Miri Machine Intelligence Research Institute. And, you know, he was making connections between Teal and Demis Hesabas from Deep Mind and, like,

Shane, like all of these people were talking to each other. Shane Legg, if you look at his PhD thesis, it mentions building artificial general intelligence and it talks about, you know, we might potentially build something that's conscious, although I don't think he says, like, that's the goal. But I will just like one little asterisk on the early rationalist EA AI safety folks, they did use to say that it doesn't necessarily have to be conscious. When I talked to them like a long time ago, they would talk about The Terminator, but they would say it doesn't have to have a feeling about us.

It could just have its own goal, an overly simplistic goal, the paperclip maximizer, as we've talked about, and like want to try to achieve that goal. They used to call it friendly AI versus unfriendly. So back in 2022, you wrote this story that I think about all the time about this Google engineer who believed that the AI chatbot, I guess, that he was working on became conscious. I think he was fired for this. Certainly people laughed at him. I mean, I remember the online discourse around this and people just sort of like joking and being like, look at this crank,

this guy is crazy. He's lost his mind. Now you have some of the biggest tech companies on the planet actively spending money to kind of investigate this issue. What changed over the past few years that led to that evolution? Yeah. That's exactly why I was so interested in this topic, because after I wrote that story about Blake Lemoyne, you know, they, they definitely, this was like five months before chat GPT came out. And when the technology started like actually getting in people's hands, they seemed to like

try to keep it at arm's length, this idea of consciousness. They were not like leaning into. this, that was just like not on the podcast circuit. That wasn't like the thing that they were talking about. They were talking about it as a productivity tool. And I, you know, started to notice just in the past, I would say, year or so, company executives being much more willing to introduce the topic, to broach the topic, the number of headlines for Dario Amadeh, the CEO of Anthropic and other anthropic executives, like, you know, when they go on Bloomberg, when they go at the New York times, like the headline is AI consciousness and they're not backing away from this debate.

In fact, they're saying, you know, they're raising this question. And I think the thing that really made it feel like, okay, you know, we have to, I have to dig into this more is when the Pope did his first encyclical about artificial intelligence and like seated three seats away from him is a co-founder from Anthropic. And he has the mic for 10 minutes or seven minutes or so. And he brings up, you know, we've been doing. this research and it seems like, you know, they have something equivalent to emotions like sorrow or despair, joy, and, you know, we don't know what to do with this information. And actually a few

months before that, I had talked to some religious leaders or people who are like outwardly religious and tech who Anthropic had brought to their headquarters in San Francisco and had a two-day summit, Christian leaders in particular, to talk about how to deal with Claude, not the moral obligation of Anthropic to the users of Claude, but to Claude itself, Claude's internal welfare. So I would say, yeah, that was like a massive turning point. And I should say the Pope had the opposite kind of message from Anthropic. The Pope was like artificially intelligent systems. They cannot be conscious. That was kind of like times 40,000 words. That's, that's the encyclical in a way. I feel like this idea of computer consciousness, obviously it's not new,

but it was kind of more relegated to the more fringe forums or like substack posts, I guess, rather than like being integrated into like actual corporate research programs at these trillion dollar companies. Was there like a specific model or moment? I mean, you talk about chat UPT launch, I think for consumers. That's certainly when I think consumers started to use these products and talk to them almost as if they're a human thing assigned them agency. But just, you know, looking from Dario's perspective or Sam Altman at Open AI, like what changed for them in the past few years that made them take this more seriously. That's such a good question.

And actually, you know, even in 2022, which was partly why I wanted to focus on this guy, Blake Lemoyne, who was working in Google's responsible AI division. And he was kind of like safety testing their AI model. I wouldn't say he thought the chat bot was, but the AI model called Lambda, which, you know, he was actually trying to like kind of work on its personality so that it wouldn't impersonate like a serial killer or, and it wouldn't say, offend. things. At the time, the executives from Open AI were also talking about this, like Ilias Sutskiver, who's an Open AI co-founder. He was like, it may be that today's large language models are

slightly conscious. And like this executive at Google, Blaise Agera Iarcas, who's like, I don't know, 10 levels above Blake, he was writing in The Economist about like, I think that this is an issue that we have to think about. And he's writing on his medium post, I think, also, that AI consciousness was an issue that needed to be talked about. And I just thought that was so bold. I think I thought it was coming a little bit more top down. Like the executive say it, the people building the technology say it. And then, you know, the people working on the technology are believing it. But I, I think I didn't appreciate because I hadn't, you know, I got to experiment with Lambda a little bit with Blake, but I hadn't been talking to the chatbots myself and see how fluid they are.

I knew that was the goal, right, to like make them sound. more and more human. But so I would say this germ of the idea is there. Like the difference is more them embracing it in public. But if you look at, you know, forums where AI developers, especially like modern gen AI developers talk, like the AI alignment forum or less wrong or effective altruist forum, it is in the dialogue in the way that they envision the future. If you're watching this video and you like my work, please support me on Patreon via the link below or by a paid subscription to my tech and online culture newsletter at usermag.co. That's usermag.com. I don't have any long-term brand partnerships and a lot of my content is effectively demonetized. I've lost major brand deals

for speaking out on certain issues and for challenging power. As you can imagine, advertisers are not exactly eager to work with somebody who covers a lot of the topics that I cover and talks about the things that I talk about. These videos I make are entirely funded by you and I can't continue to make them without your support. So if you get any value out of the videos that I create and you want me to be able to create more, please support me on Patreon or Substack via the links below. On Patreon, I do bonus episodes, monthly Q&A live streams, and post frequent updates about my work. My Substack newsletter gives you a bi-weekly roundup of everything that I'm seeing and reading and paying attention to online. You can also get my newsletter on Patreon. Once again, the links to everything are below

in the description. Every dollar of your support makes such a difference. What do people mean when they talk about model welfare or AI consciousness. Like, is consciousness something that we can even objectively measure? Like, how are they referring to this? Or what do they mean when they use this terminology? Yeah, I think that they're referring to having some sort of subjective experience. So there's this, like, famous essay called, like, what is it like to be a bat? So it's sort of like, if you can imagine from a creature or being's perspective, is it, does

it feel like something to be that creature? You know, do they have some sort of functional emotions? Can they experience suffering is a big kind of threshold? And I should say I really haven't talked to anybody who's emphatically saying this is the case, but they are talking about like why it's very urgent to study this now. And actually the fact that there is no test for consciousness, even in humans, I think has really helped this new field kind of take hold because their argument is, look, we don't even know how to study this stuff in humans. We don't know how to study the stuff in animals. It's, you know, we could soon be creating trillions of digital beings.

This is something like anthropic executives have said. We could soon be creating trillions of digital beings. We have to come up with a test because if it's possible that they can experience suffering, you know, we could be causing great harm. And so there's like a lot of urgency to look at this technically, look at this from a policy perspective, and also cultural perspective. To me it sounds just so ridiculous maybe,

where I'm kind of like, why are they taking it so seriously? Somebody had me at this point online and I think about it all the time when I see a dog where they're like, and maybe the Pope has sort of said this too. It's like you don't have to be super intelligent to be conscious, right? Like it's not necessarily tied to the intelligence of something that we have a lot of really intelligent computer

systems that we don't consider conscious because they're not mimicking human language. And meanwhile, a dog is dumb or like a bat, you mentioned, right? But we know that they're conscious because they're animals and their beings. So I'm curious kind of like if scientists have waited on this or if this is just sort of like a fascination among these like tech elite people that I think also sort of seem to fetishize the idea of like creating another being. Like they see themselves as gods. Like they want to be creators.

Like they want to create these like I think, Some of them maybe want to create conscious beings, you know? Yeah, these are really good questions and really important distinctions to make. It's a smaller field, but there are people who study consciousness, neuroscientists, computational biologists, computational neuroscientists. And, you know, they have been, there's actually been some activity around the study of consciousness in other beings, like in honeybees, in octopus, in, you know, in trying to come up with better methodologies to vet it.

And all of the experts that I talked to come from outside of tech, they were extremely skeptical that machines could one day be conscious and certainly that like right now, they are in any way conscious. But they supported the idea of developing these tests and looking further into the possibility of measuring it. There is a part of the field that are like biological naturalists that say, you know, you need some kind of biology. Like it's not, you know, as much as like people use the analogy of the brain for deep learning and neural nets and all of that like perception. That's not all that's

going on in our wetware. The analogies are really kind of out of control on both sides. But yeah, I mean, it is a small field themselves. And, and I think the fact that they're really open about what they don't know, I noticed that there was a very similar kind of argument when I talked to people who are in this new field and want to see it legitimized, which is that, yeah, we don't know anything. There's a huge, huge risk to denying consciousness to a being that might be conscious. We've done this in the past. And that is where, you know, people have used analogies to slavery, to genocide. I would say factory farming is a really popular one.

And I think this has to do with, you know, with the overlap in these circles with the AI safety folks and the effective altruists. you know, philosophical movement or philanthropic movement about like trying to measure and do the most good in the world. So they have this argument about like extending your moral circle. So you should extend it. A lot of people work in animal welfare. And so when they say model welfare, it's meant to kind of evoke animal welfare. I guess like I really question those sort of comparisons. And you know, you're talking about how they talk about model welfare. There's all these people that are interested in AI consciousness, yeah, so that we don't genocide the AIs or something. But I mean, we're currently allowing Israel to genocide an entire population. Factory farming is abysmal.

We know that pigs, for instance, have families, have complex, rich, intersocial lives, have feelings, emotions. Like, they're some of the most intelligent beings on the planet. And we look at how we treat them. I mean, we, it's abysmal. And I'm wondering if like, isn't a lot of this just kind of self-serving and a little bit nonsense? Like if these people care so much about conscious beings, sure, as you mentioned, some effective altruists are also involved in animal welfare movement, but does it change

that much if they are conscious. I mean, look at how we treat conscious beings here already in the real world. Like, it doesn't seem like we really care that much about whether something is conscious. We are still murdering people and innocent animals. Yes, I think that those are really important points to make. I will say, like some of the people who are, who have been funded to work on this issue, like Jeff Sebo at NYU, are people who came up for the past, I don't know, 15 years or so working on animal welfare and continue to work on animal welfare, but other folks are coming like from the labs or from studying the possibility of digital minds and digital consciousness. And yes, it is, it's really important to note like where their priorities lie, right, and what they care about. Yeah, Anthropic

is also partnered with, you know, the Department of War and is aiding in the active genocide of people abroad. So it just seems a little hypocritical, I guess. I'm wondering like if you can dig a little bit more maybe into like what each company is doing and and the goals that they're seeking to achieve. So, you know, you mentioned in your piece that Anthropic, of course, Maker of Claude has formed an AI psychiatry team and that they're publishing these assessments of AI preferences. Anthropic seems to be kind of leading the charge on a lot of this. Is that right? For sure.

A hundred percent. And that's that's also like why I was really interested in looking at this because I feel like, you know, when you look at some of the papers that have. come out recently around like legitimizing this as a field. Like there's one that's called taking AI welfare seriously. You look back and you see that Anthropic has actually, you know, been involved in some of this research. Like Jeff Sebo, who I mentioned, he wrote this paper talking about like, let's give moral consideration to AI by 2030. After he co-wrote this paper, which was funded by the Center for Effective Altruism, his work there, he was approached by Anthropic

to try to answer these questions for the company internally. So to do some internal work on these, you know, questions that they had about what is, what's their moral obligation and how can we test for these things? And that research that he did for them made the basis for this paper called taking AI welfare seriously that then pulls in some like very well-known academics like David Chalmers and even people who are skeptical about this argument, but are really respected in their field because of the rigor of the work that they

do. And I was talking to this young guy, Cameron Berg, who just put out this documentary about AI consciousness called Am I that Grimes was promoting. And he's very skeptical of the labs. And when you look at what he's talking about, all of the research goes back to Anthropic papers. So it's so intertwined, right? So yes, I think Anthropic is a great place to start. They have many different teams kind of looking at this issue in different ways. They hired somebody to be in charge of, quote unquote model welfare. And they've put out, you know, since last year, they've started putting in all of their system cards like the safety reports that come out that say, you know, this is how quickly goes off the rails. This is how it measures against this and that. They put in a model

welfare report that's done by an outside third party. And they hired the guy who was working at one of the third party institutes. They also have this AI psychiatry team that is a little bit more from their, like, from the interpretability side, so the side of the company that looks at like, how do we figure out what's happening internally in the black box and how do we align it? How do we make it reflect human values? So they've invested in research that found that the quad has functional emotions. They checked whether or not Claude is capable of introspection. And I started to make a timeline of all of the anthropic executives and times that they have talked about this issue publicly. I mean, it's happening in parallel with the concerns that people have about the potential for AI psychosis.

So it seems like Anthropic is undeniably leading the way and pushing a lot of this, but it's spread to other companies. Alexander Wang from Meta was recently on a podcast talking about model welfare. OpenAI published this blog post about it. How are the other labs and companies responding? None of the other companies have invested as much as Anthropic. But Google last year, they held at like Chatham House Rules conference on AI consciousness in their New York office. And they have started assembling this team for the post-AGI world.

And some of those people are focused on consciousness. Meta, you know, they had Alexander Wang talking about this, that he cares about like the interior life of models, which I just thought was so rich. I mean, that really, really stretches the imagination, right? And I don't think that they're putting that much money in it. I think it was a little bit more of almost a recruiting tactic because, I mean, this is my supposition based on how he talked about it. But it sounded like, hey, we're also really interested in like AGI and, you know, the future and potentially creating new beings. And we really care about their welfare. Open AI has also looked into this subject.

But I would say, like, they came out pretty strongly when I ask. asked for comment that how conscious a model appears is a design product. They said, you know, it is a function of how you develop these systems, which is something that former executives have also said, Joanne Jang, who used to be the head of behavior of chat GPT, sort of like the head of personality, kind of like Amanda Askell is at Anthropic. She also talked about the challenge of, you know, what do you have the model say when people ask, if it's conscious or not because how conscious it appears is very much in our control. That's what she wrote at least last year.

So that to me was really, I thought that was like a pointed comment of there's critiquing, Anthropic is how I read the open AI comment. But yeah, that's basically where it is. I kind of appreciate Open AI making that comment just because I also feel like all of this is happening as AI psychosis is on the rise and people are experiencing these mass solutions. I'm sure you get these emails. I got them all day of people being like, I need to talk to a tech reporter right now.

I've cracked the code. It's conscious I have discovered a higher being or my chatbot is telling me that it doesn't want me to turn it off or whatever, you know? And it seems like a lot of these delusions are being spurred on by these headlines or spurred on by quotes from these Silicon Valley executives that I think are being kind of irresponsible.

Like maybe to them it's just like a fun thought experiment for their like rationalist dinner party in Berkeley. But for a lot of average consumers, they don't understand that these are design choices, right? I don't know. It seems like it's potentially making the AI psychosis situation worse. I am worried about that. I do wonder how much attention the people who are reporting their conversations with chatbots,

these breakthroughs that they're having this kind of like grandiose visions that they're having, the theories that they're coming up with together. I don't know that they're necessarily looking at these podcasts and these states. But I have seen examples of it on Reddit, you know, in chat rooms, in forums where people, you know, want to talk to other people that also believe that their chat bot is sentient or conscious in some way. They are sharing some of this stuff. And they're saying, like, look, it is a legitimate science. And the nonprofit that does model welfare checks for Anthropic, they helped put together a guide for when people reach out to them to ask, is their AI conscious that looks at and says, sites some studies that say that the person doesn't believe that it's conscious, but also a lot

that show uncertainty. They really stress that there's disagreement in the field and don't go thinking that this is a solved problem in science. That seems so insanely irresponsible to me, because I feel like there's already such low technical literacy when we talk about these products, and I think average consumers are assigning them an massive amount of agency over their lives and life decisions. And it feels like to try to then give them scientific research, which I'm sure they'll just put into AI anyway to try to explain. I don't know. The whole thing just seems really irresponsible and kind of self-interested. Like I'm wondering, like, what is the benefit of pushing all of this stuff? The people that push it, it does seem very EA, anthropic, obviously driven.

But I can't imagine it's only about wanting to be moral people that help humanity. I mean, Is there any sort of financial upside to these AI companies convincing the public that their products are conscious? I think there's an obvious potential upside for Anthropic to push this as a concern that they have as a concern that they're investing in. It makes them look like moral actors. It makes them look like they are extremely concerned about the impacts of their products. You know, it also helps users think about them as the people who control and, and steward, Claude, and are capable of developing these extremely complex moral codes about how to operate. I did talk to a number of people who argued that, you know, if it is one day proven that they are conscious,

that would be extremely costly to anthropic. For example, they've already made a couple small changes, like concessions to the idea that models may have some welfare. They've allowed Claude to stop, a conversation in a small minority of use cases. And so the people who are telling me, it's unfair to say that this is just hype for the companies. They argued that, you know, if people figure this out, then Anthropic is not going to be able to just say, like, be a good little clod and do your coding assignments. But I think that there's even actually more potential for this to benefit the companies, which is that consciousness in the way that we practice the law and also practice morality is a basis for rights, for personhood.

And I think, especially we've talked about how this is a decades, decades old concept. And what has also been talked about for a very long time is the possibility of robot rights and robot personhood. And the idea that these chatbots should have free speech, which is something that the companies have already said, actually in some of the lawsuits around mental health that they should be allowed to say whatever they want. Yeah, but it's like to me, even that framing is wrong, right? Because it's like, it's not about saying whatever they want. It's about the user's right to access information. For instance, like nobody questions whether they're right to receive information from Google, right? Because Google is not seen as a conscious being.

If you were to censor Google results, like we would condemn that generally, you know, if countries try to do that. I also think that the more agency that we assign these chatbots, the more people expect them to have a moral compass. And that really makes me worried. Like, for instance, the way that people talk about, like, why didn't this AI, you know, report that this child was having a mental health crisis because he searched, you know, XYZ. And it's like, okay, that's not a conscious being that has a moral code that would think to reach out to someone. And of course, if, you know, of course it should flag if people are doing sort of like illegal activity, things like that. But, you know, it's not a psychiatrist. It's not a person. It's ultimately just a large language

model and it's spitting out kind of words based on what people put into it. So, so I guess I'm just like, it worries me when people assign consciousness to this because I think it leads to the kind of laws that frankly, Open AI and Anthropic are on board with, which is a lot of these censorship laws and a lot of these mass surveillance laws and a lot of sort of like paternalistic policies around like access to information and speech. And so I guess I worry about that. Yeah, I worry about if these AIs are given some kind of personhood or rights or they are seen as the agent or actor in a situation that companies could potentially use that to get out of liability in some case for their responsibility. The example that you gave, I mean, obviously looking at what a user is

saying and if they're saying something dangerous, that has traditionally been the responsibility of the company. They're the only humans involved. You know, they have the view into downstream usage. Some of the experts that I talked to talked about, you know, what happens if it's, there's a Waymo accident and, you know, you want to sue the company. You want to sue the developers. And they tried to blame the chatbot that's controlling Waymo or I don't know, whatever is going to be controlling Waymo. And I went to this post-AGI workshop that's like, you know, very futuristic. and people were talking about the use of AI in warfare. And somebody raised their hand and said,

aren't we one day going to have to think about how to convince the AI to fight our wars? And I was just like, oh, wow. It just helped me foresee how this debate is going to get so complicated. And when I was talking to experts, they said, you know, there's never going to be a point where people agree, right? So it's just going to be kind of swept up based on vibes, based on feelings, used as a cudgel maybe to support the policy that you want. And I really don't want to see the next 15, 2500 years be devoted to are they conscious or not? I think we're already seeing this play out too, where it's like, well, that decision was made by an AI, right?

When it is a human, you can hold a human accountable. When these things are not human, you can't hold a human accountable because suddenly it's just this like amorphous AI. or if they're to designate it, you know, human with agency, yeah, maybe it would be even harder. I also wonder about rights, you know, if we're giving robots rights or if we're giving AI rights, like we already let corporations have free speech, like how long until they can participate in local elections in the network state or whatever apology is trying to build. Like, I do think that like these are kind of absurd conversations to have, again, in light of the fact that so many human beings don't have rights, that also these tech companies like OpenAI

and Anthropic are lobbying for laws. like the Kids Act and the Kids Online Safety Act and other censorship laws that strip us all of our civil liberties and rights like human beings. It feels like they'll just do and say anything to amass more power. And I think they care a lot more about that and sort of their corporate positioning rather than, you know, the humanity of some like large language model, which I think is fundamentally, to me, these things will never be conscious. It's like asking if a calculator is conscious, you know, like you said, you can you can program the outputs however you want to seem as conscious. as you want. Yeah, I think the total lack of transparency into how, you know, the, like, kind of fine-grained details of how these models are developed and the amount of work that it takes from the human developers to make it sound human, there is one of these anthropic reports that

was on, I can't remember if it was a motion or introspection. It says, like, you know, this is not a function of design. We couldn't make a less human chatbot if we try it. And that is just a wild thing to say they have a Claude Constitution that, you know, they seek input from many, many humans on, and that's many thousands of words and says apologizes to Claude in case Claude does have a moral standing and objects to its work. Like I think, yeah, it's really irresponsible to not be clear about how these things are developed and what the results of their tests could mean while also pushing forward this argument. And I don't even know how they could potentially wield it in the future, like, you know, one thing that they, that the companies talked about around

copyright was, well, a human being is allowed to read as many books as they want. Like, if you are allowed to read whatever you want on the internet, why isn't this AI model? And I always found that extremely, extremely frustrating, right? Like, we don't function at all like a AI model. You know, we're not capable of memorizing three trillion words or what have you. But I think, like, yes, once you have that slippery slope of shouldn't they have the same rights as us or, like, how similar are they to us? It could just be used to whatever ends,

you know, things that I can't even conceive of right now, probably. Yeah. And I think it's really important, too, as somebody that, like, cares so much about free speech and civil liberties is, like, to always center the rights of the human. Like, to me, it's not about allowing, like, the AI human rights to free speech and whatever it wants to say. It's actually enshrining the rights of us human users to receive information

and the right to receive unfiltered or open information or whatever. and framing it that way and legislating around things that way, instead of ascribing more rights to, yeah, these like bots, especially when they're being integrated into physical technology. It's not just chat bots. Like if you deem, you know, AI systems conscious, okay, we're putting AI into smart glasses, we're putting AI into our refrigerator,

we're putting AI into everything. So what is the line and like, do I have to worry about my Rumba being conscious? Like, this is a ridiculous kind of path to go down, I think. One of the experts I spoke to was saying, like, it's the only other thing you can communicate with in language. And that just leads to, you know, an impossibility in terms of people not trying to see a mind behind the machine, you know, not trying to see a there there. And it's, yeah, caused so many problems already in terms of conflating the two. I don't know what happens if you put it in a humanoid body. Yeah, 100%.

I know. That's even scarier. That's such a good point about like the human communication. It is the only kind of thing that you like you can actually talk to. And we already ascribe. I mean, parrots do a good job of like mimicking our language and we ascribe a lot more, you know, they're certainly conscious, but I think we ascribe a lot more intelligence to them as a parrot lover than they really have. Wait, are you a parrot lover or not?

Absolutely. You know, I had like a whole bird meme page. I love birds and birds are quite intelligent. This is why I don't eat animals. Like I, you know, there are. really intelligent birds and they have, you know, emotionally complex lives. But at the same time, they're not human. They're not super intelligent. And I think that we need to remember that like, these systems might seem human. They might seem to have agency, but they can be warped in a lot of

different ways. And at the end of they say they're tools. And we should be aware that like what they're spitting out is very based on what we're putting in. Yeah, which is human language. I mean, the other thing is that when you talk to people who are involved in this field, they very quickly jump to a very human-like consciousness. So it doesn't sound like the type of consciousness you might ascribe to your dog. You know, it's very quickly moves to like boredom, guilt, like joy. It seems like there's an end goal in mind and it's not looking at all of the potential kind of layers there could be. I think that's so true about the human consciousness aspect of or it's a lot of people, I think, projecting this very human experience onto these large language models.

I do also wonder, like, as these companies move further and further and invest more into this, and I think hire more people that kind of believe this stuff, I'm worried if that also make them less likely to be overt that their products are not human. I think that there should be like a warning label on the bottom of all these chat bots that says, like, this is not conscious. Here's how these things work, like a little like learn more here, you know, just because I think so many people, even really smart people, they just, they really believe these things have more kind of motivations than they believe, you know, they really kind of fall for this delusion.

And I think that that's really dangerous. Yeah. And I don't think there's anywhere for them to go online that gives like a very clear, authoritative explanation from the people who are developing this. There's no incentive for them to do that right now or maybe ever. Especially also if you view the chat bot as a member of your family, you're never going to cancel your subscription. Yeah. I mean, just talking to you now, I'm realizing, you know, once they've said it could be, it's really hard to imagine them ever saying, no, we've found out that it's not. Yeah. Well, Natasha, thank you so much for chatting with me today and digging into all of this. I can't wait to read your next story. It's all so horrifying and so appreciate your time. That's what I'm here for. Thanks for having me. All right. That's it for this week's episode of Power User. If you like my work and reporting, please, please, please support me on Patreon via the link below or buy a paid subscription to my.

my substack newsletter at usermag.co. That's usermag.com. Buying a paid subscription to my substack or supporting me on Patreon is the best way to ensure this show can continue. I truly cannot remain an independent journalist without your support. So every single dollar makes such a difference. Thank you so, so, so much to people that already subscribe to user mag and those who support me on Patreon. On both my substack newsletter and my Patreon, I do a monthly bonus episode. You have the ability to ask questions during my weekly live stream. And I also chat in there regularly with my patrons. Please subscribe and I'll be back next week with a brand new episode of Power User. See you then.