Taylor Lorenz’s Power User - AI Text Now Has a Secret Tracking Code

Episode Date: August 21, 2026

To access an ad-free listening experience, get bonus episodes, and receive my weekly newsletter, subscribe to my Patreon or Substack 👇🏻Buy a paid subscription to my newsletter at⁠ ⁠https://www.usermag.co⁠⁠         Support my work on Patreon:⁠ ⁠http://patreon.com/taylorlorenz⁠⁠Anthropic just announced that Claude and all its AI tools will now watermark every piece of text a user generates, even if it's scientific research or computer code. In this episode of Free Speech Friday, I sit down with Neil Chilson, former FTC Chief Technologist, to break down exactly what AI watermarking is, how the EU AI Act transparency code is forcing this change worldwide, and why this could threaten your privacy and free speech.We explain how probabilistic text watermarking works by subtly altering word choices in AI-generated content, why this creates serious problems for legal documents scientific research and creative writing, and how Anthropic's decision to apply these watermarks globally affects US users who never consented to this tracking. Neil also breaks down the cybersecurity risks, the potential for government overreach, and why open-weight AI models might be the only escape hatch from extraterritorial regulation. If you use Claude, ChatGPT, Gemini, or any AI writing tool, (or even if you don't!) you need to understand how the EU AI Act transparency requirements are reshaping AI output behind the scenes. We discuss the dangers of compelled speech on digital platforms and what watermarking means for the future of anonymous online expression.SUPPORT MY WORK: Buy a paid subscription to my newsletter at ⁠https://www.usermag.co⁠         Support my work on Patreon: ⁠http://patreon.com/taylorlorenz⁠

Two and five Canadians will hear the words you have cancer. That's why every step and dollar raised matters. On September 19th, join thousands in Toronto for the Princess Margaret Cancer Foundation Walk. Challenge yourself, friends, and family to walk 21 kilometers in support of life-saving research. Together, we can carry the fire and help create a world free from the fear of cancer. Register today at pmcf walk.ca.ca. Visit BetMGM casino and check out the newest exclusive. The Price is Right Fortune Pick.

BetMDM and GameSense remind you to play responsibly. 19 plus to wager. Ontario only. Please play responsibly. If you have questions or concerns about your gambling or someone close to you, please contact Connects Ontario at 1-866-531-2,600 to speak to an advisor. Free of charge. BetMGM operates pursuant to an operating agreement with Eye Gaming Ontario.

The European Union is going to be able to do this in a way that if our U.S. government did it, we might be able to sue, but we won't be able to sue because this is Anthropics' choice to do it worldwide. What if every time you used an AI tool to do something like create a response letter to your landlord, brainstorm a new project, or write code for a new software application? That AI system was embedding a secret message within its output that would permanently tie you, the user, to the content that you were researching or creating with AI. Well, that's effectively what Anthropic announced it would be doing through a system called watermarking. Thanks to a new EU law, all content generated using Anthropics AI tools, including Claude, APIs,

coding systems, et cetera, will begin watermarking content. This is being framed as transparency, and regulators claim that it'll make AI content easier to detect. But I would argue there are tons of better ways to detect AI content and bring transparency to this space, and these watermarking systems come with serious problems. Neil Chilson is a lawyer, computer scientist, and author of the book, Getting Out of Control.

He was the former chief technologist at the FTC, and now head of AI policy at a think tank called the Abundance Institute. Today, we're going to be digging into watermarking, how this EU law is actually affecting AI output all around the world, including in the US, and why we desperately need more privacy and speech protections in the digital age. And I know what a lot of you guys might be, thinking, which is, who cares about AI? I hate AI. Who cares if we have more transparency laws or

restrictions on AI? And here's what I want you guys to understand. You can be critical of AI. I'm very critical of AI, but people are using these tools. There are advocacy groups out there that are leveraging AI to help immigrants and LGBTQ people and other oppressed groups fight back against oppressive state laws. People are using AI to build software and vibe code and make technology. And I think there are a lot of very useful cases for AI. I'm sorry. I know, I know, A lot of you guys don't agree with me, but the reality is these are platforms that are being used by millions and millions of people every day. They are defining the new information landscape.

And so we should be really, really critical of the policies that are being written that will affect them. There are serious privacy and speech concerns with these laws. And you can dislike AI and fight AI while still recognizing that we need to write smarter tech policy that protects our civil liberties. So with that, I'd like to welcome Neil to Free Speech Friday. I, Neil, welcome. Thanks for having me.

To start off, why don't you explain what watermarking is? What does it mean when an AI model watermarks content? So a watermark is a less than visible, sometimes super hidden, sometimes less hidden mark that tries to identify the provenance or where content came from. The term itself comes from the stamps that you might see on fancy stationary that has the brand of the company who made the paper or something like that. But the idea in digital content would be to change the content. content in a way that's not obvious to the user, but which tells something about the content's origin.

And in the AI space, you might do things like change various metadata that's in an image to say that it came from a company. That's not exactly a watermark. A watermark tends to be independent from metadata. It tends to be embedded in the content itself. And so in an image, you might change certain pixels in a image in a way that you can't really see, but that a computer could identify as having been changed in order to mark where the content came from. What's really interesting about some of the new developments is that there are these new methods to watermark even plain text. And what that means that if you just cut and paste that text from your chat screen into a Word document, then you cut and paste it into a chat, and then you cut

and paste it into an email, the watermark travels with it. And this is a probabilistic watermark. We can get into all the details on it. And so it's not as definitive in any. way as maybe some of those other types of watermarks, but it is a way to mark text as having come from an AI source. It works much better on long text than it does on short text. In fact, it doesn't work very well at all if you're under 200 characters. That's the basic idea. The basic idea is to mark a digital piece of content with its source or where it came from or pass along other types of information in a way that when you make copies will survive. So Anthropic announced that they are going to start watermarking all the text generated by its models, including Claude, to comply with

European regulations. The EU passed something called the EU AI Act's transparency code, which just took effect last week. And this requires AI companies to watermark AI generated content in a way that can be identified. So what is this EU AI Act transparency code? What is the EU AI Act? And why is this happening? Well, the EU AI Act is a provision that was being.

begun, believe it or not, in around 2021, before Chat GPT was even introduced. The Europeans are cutting edge in certain ways when it comes to regulation. And I think it went into effect primarily in August of 2024, but various provisions of it have had different timelines. And the transparency code implements a part of that act that requires watermarking and the ability to identify digital content. And so the transparency code lays out some of the details about what that means and what it doesn't. And as you said, it went into effect recently. I think it also has some staggered

implementation effect. So some of it also goes into effect in December. And then there's another part of it that goes into effect in February. Like all European laws, it's pretty complicated. But the basic is that it requires certain types of providers of AI, generative AI, to watermark their content. I think when a lot of people think of the word watermark, we think of like a little image on a meme, maybe in like slightly more transparent text. Another way that I've seen people quote-to-quote watermark text lately is kind of like insert text in like white font so that when you copy and paste it,

that white font text is carried over and it's revealed that you use AI or whatever. How is Anthropic going to be watermarking their text outputs? Like are they changing the words? Are they putting little graphics somewhere that we can't see? How is it going to work? So Anthropic hasn't said exactly what algorithm them it's using, but the most common popular algorithms for this, including the ones that are running

in Gemini right now, you know, Google did this voluntarily quite a long time ago. They work by choosing from the plausible options of, you know, the way LLM's work is that they predict the next token. So if I say my favorite tropical fruit is, and then I ask an LLLM to fill in that sentence, you know, there's a bunch of options that it could put there. Some of them are going to be much more popular, so like Mango, some of them are going to be much less popular, like Lichy or something like that. And what this algorithm, this watermarking algorithm does is it replaces words that are plausible, but maybe that it wouldn't have used before in a way that is identifiable later, but probably doesn't make much of a difference to the user in many cases, right? And so there's a wide

range to say many of the ideas that we might ask an answer from a chatbot, or to generate text. And so it picks between those different versions in a way that lets somebody who has the secret key later look back at that text and say, oh, because of all these different word choices, I now know that it was generated by AI. And so it's in the text itself. It's in the words. It's not some sort of hidden characters. It's not some special specific words.

It's just a bias in how the LLM generates the text to meet the user's question. Two and five Canadians will hear the words, you have cancer. That's why every step and dollar raised matters. On September 19th, join thousands in Toronto for the Princess Margaret Cancer Foundation walk. Challenge yourself, friends, and family to walk 21 kilometers in support of life-saving research. Together, we can carry the fire and help create a world free from the fear of cancer.

Register today at pmcf walk.ca.ca. Get your tickets now for the dog stars. It's no surprise how fast civilization can disappear. Go! Out there, just kill or be killed. Jacob Allurety. There are people out there? They need our help.

Josh Berlin. Shoot first and inquire afterwards. Margaret Quali. Okay, tough guy. And Guy Peers. I'm up. The dog stars directed by Ridley Scott. In theaters, August 28th, get tickets now.

Hear that? It's your money calling. It wants a promotion. Elevate your savings with the Scotia high interest savings account. Always earn high regular interest rates that grow the more you save and invest. Conditions apply.

Visit scotiabank.com slash h-I-SA to learn more. Scotia Bank. You're richer than you think. This seems like something that people would welcome. I feel like tools like PANGram have become really popular. Everybody feels like their feeds are inundated with AI slop. So something that could help make, I guess, AI content more easily identifiable.

Seems like it would be a pretty popular policy. But I want to kind of dig into how this works and some issues that it might cause. So when you saw this past, what were some issues that stood out to you that people might not be noticing right away? One thing that jumps out immediately is that this is a message that goes along with text. When you're talking about inserting a watermark into the text that I prompted, that I'm trying to get the AI to create, for example, it means there's an extra message in there that I didn't necessarily ask for. And so when you're asking a computer to optimize for two different things, ultimately, the math just says it's not going to be as good at creating the thing that you asked it for if it's also trying to do something else.

And so the way they do it in text is very, very clever. It's not super noticeable for the average use. But if you're doing something like working on legislative text or you're entering text that you had created, but you're asking for proofing or something like that, the watermark does not really give any indication of what percentage of the text was AI generated or what percentage of it was human generated, like Pangram does, for example. It's a yes or no, basically. From reading the guidance on Anthropics site, it suggests that if you put text in there, it's going to market. And even if you wrote 95% of that text, then there were some typos that were fixed by the AI, it's going to now look as if the AI wrote the whole thing.

Or at least people won't be able to distinguish between those two. So people will really have to understand what this signal means. And that's a real challenge, I think. People, like you said, are looking for ways to identify AI text. And so the challenge here is that this will be both over-inclusive in that it will identify content that was completely human-made, maybe edited a little bit by AI as AI generated. And it's relatively easy to evade. If you're watching this video and you like my work, please support me on Patreon via the link below or buy a paid subscription to my tech and online culture newsletter at usermag.com. That's usermag.com.

I don't have any long-term brand partnerships. and a lot of my content is effectively demonetized. I've lost major brand deals for speaking out on certain issues and for challenging power. As you can imagine, advertisers are not exactly eager to work with somebody who covers a lot of the topics that I cover and talks about the things that I talk about. These videos I make are entirely funded by you

and I can't continue to make them without your support. So if you get any value out of the videos that I create and you want me to be able to create more, please support me on Patreon or Substack via the links below. On Patreon, I do bonus. this episode's monthly Q&A live streams and post frequent updates about my work. My substack newsletter gives you a bi-weekly roundup of everything that I'm seeing and reading and paying attention to online.

You can also get my newsletter on Patreon. Once again, the links to everything are below in the description. Every dollar of your support makes such a difference. This seems like it could cause a lot of issues when it comes to some of the stuff people use AI for, especially when the text has to be pretty specific. I'm thinking of like scientific research literature or something or, I don't know, know, information about sort of like pharmaceutical science stuff, legal documents, et cetera. I mean, I feel like it or not, people are using AI for a lot of like hyper-specific tasks,

especially businesses. I mean, Anthropics, huge business is actually B2B, which I assume this is going to apply to as well. Would that cause problems? Because I feel like swapping out a word here and there is not a big deal if you're just like using it to write your English essay. But if you swap one word for another word in like a scientific research instance, that seems like a bigger deal. Yeah, and I think especially in the sort of, you know, I'm a lawyer, so I see things from that angle, too.

And entire cases have turned on where a comma is placed in a statute or in a contract. And so these types of changes could have legal significance. They could have scientific significance. Now, the toolmakers try to make it so it conveys the user's idea accurately. And mathematically, they can do that in a way that keeps it tied to what they call the distribution of the overall model. but they don't really know how important various words are to any specific user. And so sometimes this watermark might make a better sentence for a user, and sometimes it might make a worse sentence.

But what it means is that overall, it's not doing only what the user was asking it to do. It's also doing something else. And I think that is the challenge, especially when you're trying to be very precise. Watermarking poetry through this method would seem sort of like breaking poetry. You know, the words are chosen very carefully for very specific reasons. And so even if you conveyed, you know, a rose by any other name would smell the same. Like, you swap in tulips. It's not the same sentence.

It wouldn't mean the same thing. It wouldn't have all the connotations that it does. And so I do think that there are certain applications where watermarking would ruin some of the value of the content specifically. And like I said, it could have real legal consequences. I know what my listeners are going to say, which is like, AI shouldn't be making poetry. They don't want the AI to do art, which I totally understand. And so that's kind of why to me, I mean, even if you accept the watermarking in these more creative uses, I do think like the legal, the scientific, and just like the precise nature of stuff, I don't know.

I was using AI to review a contract recently that I had made myself, which I was like, you know what, why don't I ask AI if I, like, made any mistakes? It turns out I made a lot because I'm not a lawyer. And I was going off Google to try to like make this contract. But, you know, you could imagine a world, as you said, if a comma, place, is somewhere different like that, it has legal significance. If there's certain sort of language swapped out, ultimately it's about these competing interests, right? We have the government kind of having their interests and then the users of their interests.

A lot more people are also using AI to create software and software development is also hyper precise. So I'm curious, I mean, is that ultimately text too? It is text and some of the ways they try to do it in code would be things like changing variable names that don't really change the meaning of the code. And so they'll try to do things like that. We call these types of spaces that you're pointing to. They're like lower entropy, which means that there's just fewer variations that the user would find acceptable.

And the fewer variations there are than the harder it is to watermark. It's why a long piece of text is easier to watermark and a short piece of text is harder because there are just fewer variations that the watermark could choose from to try to embed. And so these low entropy spaces, these spaces where every word, every comma,

every space matters, they are hard to watermark. And to the extent that you do watermark them, you are going to mess up the user intent. What about cybersecurity issues? I saw people talking online saying that like, hey, if you create this whole other watermarking system that only other AI can read, or only other machine learning systems can read,

you're also creating vulnerabilities. Now you have these other things. You know, if you're trying to develop, I don't know, something secure. I'm thinking of the recent hugging face incident where these open AI, agents were like communicating on message boards. And I guess this person on Twitter was at least theorizing, like, you could imagine a world

where AI is embedding secret messages for itself in these watermarks. I don't know. But are there also sort of security implications with this watermarking stuff? I think there are some potential security implications. It's hard to encode in a watermark a ton of information. These types of watermarks right now essentially are a yes or no. Like was it generated by this model or is it associated with this model or is it not? And so the idea that there would be lots of message passing is a little hard to understand.

But these watermarks do rely on secret keys that are held by the companies. And so if someone got access to those secret keys, among the things you might be able to identify is what account generated, what content. And that's more than just saying this was AI generated. Then we start getting into privacy implications. Because if we can then tie back to any person, you know, any content they generated. on AI, then we start to lose some of the anonymous speech that we might use AI for or other types of spaces. Those are all tradeoffs. And I think there are security concerns here, just as they are with anything where you're trying to pass or keep a secret from some part of the world.

The privacy stuff scares me. I saw a conversation about that as well. I feel like just the more that we're removing anonymity from the internet and creating these identifiable patterns or statistically significant nuances and differences between different users. The more identifiable information is out there. And I don't know, tech companies get hacked, information leaks all the time, or this could be used by governments. I'm thinking of the way that a lot of these governments have sought to compel tech companies to invade our rights and gain identifiable information about the content that we're posting

online or searching for online, et cetera. It seems like this would make it a lot easier for the company to have that information to potentially turn over to a government. I totally agree. And I also worry, you know, this same technique. Right now, the bias that's being put on the language model is toward just changing the outputs so that we can understand whether or not it came from an AI. But the similar technique could be used to select in a way that would be quite undetectable in any one sentence, but that over an entire body of text might be identifiable, you know, shift in ideological directions, right? That's not how these watermarking systems work now. But a similar technique, a similar sort of bias framework could be used like if the European Union wanted to say, we want to combat hate speech and we want to make companies just shift their outputs to be 5% more in some political direction. Like these types of techniques are the type of thing that you might use to do that in a way that wouldn't be obvious to the user at all.

Absolutely. And I mean, I was just doing some reporting for The Guardian in the UK recently. And this is something that the government would love to do. One issue that they have raised repeatedly is that people are getting misinformation, quote, unquote, from these AI systems. And the UK, broader Europe, they have sought to enact a great deal of control over the access to information and speech. And so I feel like just creating these openings where abuse could proliferate is definitely concerning. At the same time, I can imagine a lot of people getting on board with this because people

hate slop. So what is the goal? How do you see these systems being used? You know, if they're investing all this time and energy to put all these water, marking systems out, like, who's going to be checking the watermarks and in what capacity are we going to be using them? So that's not exactly clear. Right now, Anthropic is saying that they will make a tool available.

The EU law does require that there be some public compatibility between the various systems so you can identify. That raises a whole bunch of coordination issues. It's not quite clear how that's going to play out. The one thing that's interesting to me is that, you know, a lot of the examples that you talked about earlier about watermarks are ones where the creator of the content puts the watermarking, right? So you watermark a meme because you want other people to know that you're clever and you came up with it, right? That type of watermarking makes perfect sense and having technologies

that let you do that so that you can prove that, hey, this originated with me makes a lot of sense. When watermarks are ubiquitous and they're also confusing where you can't quite tell if it means that somebody entirely produced something with AI or if they did it in part, I think the whole system becomes a little less valuable to signal whether or not something is original. And I think that raises its own set of complicated problems. Yeah, I mean, to me, it doesn't make any sense to do this. Like, it's not going to make it more transparent. I, a huge supporter of the third party transparency tools.

I did a partnership with PANGram a couple months ago. I love the transparency that they bring to the space and the questions that they raise about how are people using AI in what capacity? Where are they using it, you know? But as you said, this watermarking tech is very crude. It doesn't tell you whether the text was totally written by A, it could have just been edited by it.

I wrote a piece recently, a freelance piece where the editor actually told me that she had put my freelance piece into AI to ask if there was anything that it missed. And I was like, oh my gosh, I totally also see from an editor's standpoint, this is probably a really useful tool, right, that they can use to check holes and things. So it's just so muddled anyway. Like, it doesn't seem like it's going to do anything. And it creates all these privacy problems. Like, who came up with this system? I don't know. I mean, the Europeans, they love these types of systems, right, where they see a problem and they come up with a technical fix.

a legislative fix. GDPR and the cookie banner is a great example of this. Hey, we're going to solve privacy by having, you know, clear consent from everybody. And now nobody pays any attention to them. It's, you know, it's kind of a joke. Like, they're ubiquitous. And so they don't, they don't serve any purpose. And I think there's a risk of this watermarking mandate becomes something like that, where basically eventually people just don't even pay attention to it because the signal is so low from it. Yeah. And in the meantime, it's like created problems that we didn't need to have. I mean, I talk a lot on my show about Europe's crazy tech laws. But is this going to affect Anthropic worldwide?

I mean, once they enact these systems, are they just going to enact it for users even here in the US? Like, is our content going to be watermarked if we use Cloud here as well? Well, unfortunately, Anthropics' choice here was to implement it worldwide. And so even though it's only mandated for European users, they have put it on their systems for everybody. And they've put it across not just their chatbots, but it's also their APIs and all the other ways

you might actually use Claude. And so that was a. choice of theirs. Maybe it's simpler for them to just comply worldwide, but I wish they had made a different choice and let different solutions emerge in different regions. I think in the U.S., there's even some compelled speech and First Amendment challenges that come with if the government tried to do this. And so it's very weird that the European Union is going to be able to do this in a way that if our U.S. government did it, we might be able to sue, but we won't be able to sue because this is Anthropics' choice to do it worldwide. That's a challenge. In a

problem. Yeah, I feel like it's very similar to a lot of the stuff that's happening with Meta, where the EU is taking this more like activist approach, passing these really crazy, often speech restriction laws and stuff, mandating content, takedowns, et cetera. And meta has just started doing it across the board where they're like, it's easier to just implement stricter sort of restrictions on content everywhere because to just police one country in the English speaking world, like the UK or whatever versus America, it's easier to just roll it out everywhere, which is scary, as you said, because like, we can't challenge these laws, right?

Like, this is, I would argue, completely against the first. I mean, it's kind of definitionally, it feels to me, again, not a lawyer as like compelled speech, but there's nothing we can do, I guess. I don't know. That feels scary. I mean, it's a challenge that this administration and some previous administrations have pushed back against some of the European tools that they're using to take, you know, billions of dollars from American companies. But that's the sort of extent of what we can do against them as users is, well, we

We can make the choice not to use the tools that adopt this. And we still have that choice. But as far as legal avenues, a lot of them are diplomatic and diplomatic pressure from the administration. And not every administration has been opposed to these types of measures. Sometimes they've been very cooperative because they know that Europe can do something that constitutionally wouldn't fly in the U.S. Two and five Canadians will hear the words, you have cancer. That's why every step and dollar raised matters. On September 19th, join thousands in Toronto for the Princess Margaret Cancer Foundation Walk.

Challenge yourself, friends, and family to walk 21 kilometers in support of life-saving research. Together, we can carry the fire and help create a world free from the fear of cancer. Register today at pmcfwalk.ca.ca. Hear that? It's your money calling. It wants a promotion. Elevate your savings with the Scotia high-interest savings account. Always earn high regular interest rates that grow the more you save and invest. Conditions apply.

Visit scotiabank.com slash h-I-SA to learn more. Scotia Bank. You're richer than you think. Want to talk ETS with the people behind them? Join me, it's Jean-Jean Jean-Cabouchard on the Upside Ticker Talk. We dive into what's moving markets with timely analysis, trends, and answers to the ETF questions you've actually been thinking about. Hit play on the upside ticker talk and elevate your ETF investing IQ.

Yeah. There's all this conversation in the U.S. around open weight models. The Trump administration's potentially thinking of restricting them. It seems like this would be perfect evidence of why we need open weight models. I would imagine that type of technology they can't be doing watermarking. You can get in and make alterations yourself to kind of the outputs. So yeah, I guess I'm curious like your thoughts on the open weight debate and whether you think this brings kind of like new saliency to those arguments.

Yeah, I think open weight AI is a competitive pressure on all of the large labs. I think it's a really important one. We are getting back into the game in the U.S. Meta just released a great open weight model. They keep coming out, and they are a check against these types of decisions by some of the biggest labs to perhaps impose extraterritorial effect to overseas laws on American users. And so Open weights, I think, is a great escape hatch for Americans who want to make sure that they can protect their own liberties and speech by controlling their own models.

Yeah, absolutely. Well, is there anything else you think people should know about this law or the greater sort of debates around this watermarking stuff that we didn't cover? For the vast majority of uses, these things won't cause changes to the content. But if you interact back and forth to try to explore ideas with an AI, there's no guarantees in here that it won't shift the end result to a direction in a place that you didn't expect it to go. That could be for the good. It could be for the bad. But what these watermarks mean is that the computer is not pursuing your prompt to the fullest. It's pursuing your prompt

and the European Union regulation side by side. And that's in every single prompt that you put out there. And so a bit of a strange world to be in. Well, Neil, thank you so much for joining me. It's great to have you on. And I really appreciate all your work and commentary in the space. Absolutely. I showed the same respect. And it's great. I'll see you online. All right, that's it for this week's episode of Free Speech Friday. If you like my work, please, please, buy a paid subscription to my substack newsletter at usermag.com. That's Usermag.com or support me on Patreon via the link below. I cannot continue to produce this show without your support.

So truly every single dollar makes such a difference. If you'd like to listen to add free episodes, those are available on my substack and Patreon. I'll be back next week with a brand new episode of Free Speech. Two and five Canadians will hear the words, you have cancer. That's why every step and dollar raised matters. On September 19th, join thousands in Toronto for the Princess Margaret Cancer Foundation Walk. Challenge yourself, friends, and family to walk 21 kilometers in support of life-saving research. Together, we can carry the fire and help create a world free from the fear of cancer.

Register today at pmcfwalk.ca.ca. Friday, see you then.