Hacked - Poison Pixels

Episode Date: December 16, 2023

How can an artists protect their art from being scraped by AI models? By turning it into a 'poison' that will corrupt those systems if it ever is. Our conversation with Shawn Shan from the University of Chicago about "Nightshade," "Glaze," and a suite of tools they're developing to help artists protect their art. Also a five minute intro about plants, deal with it. Support hacked on by visiting hackedpodcast.com to find our Patreon, or grab a sick visor, mug, sweater, or shirt at store.hackedpodcast.com. Learn more about your ad choices. Visit podcastchoices.com/adchoices

Are you a plants guy, Scott? Plants is in like green things that grow in my house. Yeah, you got it. I am, my wife is a plant person. I used to be. I used to keep tons of plants in my house. And I actually grew up in an acreage and we had like a two acre garden. Sure.

So I have lived among the plants. But I would say currently at this present moment, I am not a plant person. Okay. It's good to know. Are you a plant guy? I like a good fern. Okay. I got a little cactus on my desk.

I like a plant. Nice, nice, nice. So there are a lot of plants that will kill you. But few are as famous for doing so as Nightshade. Are we talking like Dungeons and Dragons here? Because it sounds like it. I mean, it's probably in Dungeons and Dragons. That's the thing about Nightshade,

a.k.a. atropa. Beladonna. aka the deadly Nightshade. It's very famous. I don't know if you know what it looks like. It's this little green leafed plant with pretty purple flowers and he's frankly quite delicious-looking little shiny blackberries.

But don't eat them because it is famously very poisonous. There are a lot of other poisonous plants. You got your hemlocks, you got your fox gloves, but Nightshade has the reputation. It's in the all. Odyssey. It's in Shakespeare. It's in Roman myths. It's part of Salem-era stories about

witches potions. Call it whatever you want, Belladonna Nightshade. It's like a shorthand for poison and death across centuries of literature and stories right up to, as you said, Dungeons and dragons. But Deadly Nightshade has this other association. The first part of the name, Atropa Belladonna, Atropa, refers to one of the three. three Greek fates, the people that would, like, cut the strings to end a person's life. The first part of the name refers to death, the poison part of the plant. But the second part of the name is Bella Donna, which in Italian means beautiful woman. And that is because this plant, this shorthand for death, also had associations with aesthetic beauty.

It was used as a cosmetic. This is kind of wild, but they would make eye drops out of it. Interesting. Because those eyedrops would make your pupils dilate, and that was apparently considered very hot at the time. Also, don't try this. There's a reason they stopped. But you have this one plant with these two deep, long associations, poison and aesthetic

beauty, death and art, which makes it a really cool name for the subject of this episode, which is a piece of software attempting to poison a set up. system built on a foundation of and arguably producing visual art. I'm just interested in protecting human creativity in some sense, right? This is, I feel like the companies or even government are taking a fairly fort-sighted view on AI, right? Of course, these are awesome. They are able to copy style generic images.

But what I see is if we keep going down this route, say, you know, they get better and the artist is getting replaced. And that more or less kind of mark the end. of human creativity in at least in some aspects. And it's unclear where are we going to go from there. That was Sean Shan, a researcher at the University of Chicago. And we had a conversation with him recently, didn't we? Lovely guy. Lovely guy. Great chat. He is part of a team working on a suite of tools for artists to be able to turn their art into a kind of poison, a nightshade, which is why they named it that, for generative AI models. You apply these tools,

to a piece of digital art, and the art looks normal to a human. But if it's ever ingested by an AI model, it will hurt it, like poison. Hence the name, Nightshade. Not only will the AI model not recognize what's in the image and fail to scrape the necessary information to imitate it, it's a poison in the sense that it will get it actively wrong. The artist creates an image of an elephant. the system doesn't just fail to see, oh, that's an elephant. It goes, oh, that's a little kitty cat.

And so the next time it makes an image of a cat, there's a slightly better chance that it will have a trunk. That's the basic idea. Does that make sense? I love this. I love this idea. I love that they did this. So interesting.

This was a fun one. I really enjoyed this conversation. So here is our chat with Sean Shan about Nightshade, their earlier tool glaze, which you'll hear us talking a bit about in the intro. poisoning AI systems, and David versus Goliath tech projects. Here, on Hacked. Sean, thank you so much for sitting down with us to chat about this project.

Of course. So I'm a layperson. Feel free to correct me as we go. I want to get to Nightshade. I want to get to Glaze and your team. But before we do, there was this sort of overarching question that snuck up on me as I was reading about your projects. when you train an AI model like Mid Journey or Dolly on an image, you feed an image into it, what kind of information is it getting out of that image?

How should we understand what it's learning about an image when you feed one into it? I see. Yeah. So at a very kind of high level, so this model takes in not only images, but they're also taking corresponding text that kind of describe these images, right? if you have a vengal painting, you will say this is a painting from vengal of landscapes. So what it does during the training process

is basically associate, all right, so wengal is the name, and next time I get a question for generating a vengal image, actually produce something similar to the image that I've been seeing. So specifically how they work they call diffusion model. I'm not getting into the details, but it is very powerful.

However, that each individual training points, they don't have as huge impact of the entire models, right? So if you're training data only appear only once, it probably doesn't have too much impact. But there are many images that has to be shared very frequently online. So also, again, memorize the model really to memorize pieces of that image and try to perhaps reproduce it when you prompt an image out. So that's kind of how this models works.

And then specifically, because I saw this term come up quite a bit in your research, what is style mimicry in generative AI? Is it the same thing as just feeding one image in? What is style mimicry for anyone that doesn't understand? Style mimicry is something more targeted, perhaps, is, you know, the cases where I go online, I see this artist I really like. I just want to get her painting,

but I don't want to pay him. So what I can do is I can use these AMO of mimic their artwork. And I can do this with just downloading some images from their Instagram or their website is need 10 images. And then they call it fine-tune this model. Basically just means you have a base model, you just train a little bit more on that additional 10 images.

And now the new model will be able to basically output arbitrary content from the same artists, right? Very much the same style as how the artists would paint them. Maybe the quality is not as good, but oftentimes we see this is good enough to replace the artists for many, types of commissions. Sure.

So that's kind of the style of your current topic. Interesting. So hypothetically, an artist who might have had to get a commission from someone wouldn't need to get that commission if that person was able to use one of these systems to create a piece of art that was to a lay person pretty close to indistinguishable. Exactly. And it perhaps is not no longer hypothetical. There are cases I think artists get replaced by these models.

So, like, it gets really bad. If you're like some artists today, they search their own name on Google, for example. Like the first thing that pop-ups is not their website anymore. It's like the model that make their style. And, you know, if I'm a customer, there's absolutely no reason for me to go to the artists, wait a couple of months, spent a couple hundred dollars to do that. I can just use a model to do that right away. Yeah, it's, I haven't, I have more stuff I want to get to,

but it reminds me of one of the first things I think most people do, you know, a year ago when a lot of people got access to these models for the first time, the first thing you do is you have it generate something totally abstract or just a concept. But inevitably, you come to that question of, oh, I'd like this artist. Could you do that same prompt in the style of that artist? And I think that's the moment most people realize the full implications of, you know, this tech we're just figuring out for the first time. Exactly.

So if a person wanted to, let's say, deceive that model, to create an image that the model would misunderstand in some kind of way, to create data that would, to bore the phrase, poison it to prevent such mimicry, how would they go about doing that? And I guess to get to your project, how did you go about doing that? Yeah, so I think for us,

so we, the product we call it glazed, disrupting kind of style mimicry step. So the idea is fairly straightforward. It's okay, the constraint we have is we can't change the piece of art too much. We can add some small changes. Hopefully that's not too disruptive. But what we can do is we can carefully craft these small changes to confuse these A.N. models, right?

So they are very smart, but these A.N. model has a very different way to see images compared to how us human see images. And we can leverage that, basically, that gap, to adding some small changes that are very small to human eyes, but are very disruptive to how these models see images. And so this is kind of how we did it is clear. we asked some small changes to us human is the same image, but the model see that image will say, okay, this is actually a completely different style from a completely different artist, so of course it would not be able to steal or do the style mimicry as you normally would. You use the phrase how they see the images.

Earlier on in the conversation, I asked, you know, what did these models see? And we talked a bit about the text associated with it, the way that they're tagged and this sort of human reinforcement element. But I guess when you're changing what the image sees without changing what a human sees, this might be too abstract a question, but what is it seeing? What is it seeing that's different from what I'm seeing? Or is there a way that I as a human will even really understand that? So I think one analogy I tend to give is you can think of this as a UV light, right?

So like machine learning can be UV light system. And of course, they turn a mass number of pixel values. So they see certain things we don't. really see, right? So in some sense, open UV light, see a lot of hidden sync there. That means we can snuck into a lot of changes on the normal kind of light frequency. But to us, it doesn't really change much. But once you open the UV light, like these models, you will see so much different changes. It was super disruptive to how these model can understand images. But going a little bit more technical, these models are basically functions, right?

they map raw pixel vectors into a bunch of high-level, not-very-light black box, features that they use to reason to generate different art. And the feature space can, some sense, be interpretable. Like, you know, Van Gogh images will be similar to other similar artist images. But beyond that, the models that are setting space is very hard for us human to understand. But because of that, we can also add in some small changes to really disrupt that space, we know the exact function that's being used to, you know, process images. Given that you're changing essentially imperceivable values to the human eye,

but there are still values being changed kind of in the image binary, how hard would it be to then train the AIs to identify those manipulations and bypass them? That's a great question. So, yeah, so since we released Glaids, there are quite a few kind of these type of attempt to train the AI to recognize them. So what I'm going to say is it is generally hard to do that without sacrificing its normal performance. So kind of without going to too much detail of this, so this is, so these type of small changes is a vulnerability that's kind of researchers have identified for a very long time, right?

These model have these problems that if you add some small changes, you'll very easily to confuse these model. So there are quite a few research kind of, you know, how do we make model robust? against these changes. And in general, after like five years of that line of research, it kind of agreed the fact this is very hard to do. It seems that this is some fundamental

property of this model. And so in order to be robust to these changes, you basically have to sacrifice a little bit of how your model performed. So we did some tests in our research. And in the case of generative models,

the sacrifice is quite significant. The reason really is just because you really have to be super precise to get a very high quality artwork. And if you change a little bit, you start opening some real artifacts and that's not very usable. So there's not like a global fingerprint to glaze that it can start to identify and just kind of remove or extract that malicious data from? Yeah. So I think there are people trying that. What we see has been felt or only working very specific. specific cases. And the reason is that we kind of proactively thought about this, so we add quite a bit randomness into this whole glazing process.

Think about the last time you heard a breach story on this show. It always starts the same way. Someone, somewhere saw something too late, an alert buried, a signal missed, an SOC that just couldn't keep up. Arctic Wolf set out to solve that problem by rebuilding security operations from the ground up for a world where attackers are already using AI. They created the Aurora super intelligence platform, a fully agentic system powered by the swarm of experts. Instead of single-purpose bots or lucky-guess LLMs, this swarm is full of deterministic agents that handle whole entire workflows. Humans stay in the loop and on the loop to validate the critical decisions and keep everything trustworthy, and all of this is just off running on their secure

operations graph. A constantly updating intelligence engine fueled by more than 9 trillion telemetry events every week and over a decade of real-world incident response. The system reasons on real signals and real context not synthetic training data. And the result is the new Aurora agent SCC. It's the first SCC that is agent led by design. You get agents that coordinate, agents that investigate, agents that respond at machine speed, and hundreds more that automate the repetitive work that normally buries human analysts. Arctic Wolf didn't try and bolt AI onto an old model. They rebuilt the model entirely. What makes it even more effective is how it works with Arctic Wolf's concierge experience.

The team brings customer-specific context directly into the platform so every AI-driven decision reflects your environment instead of generic assumptions. The automation frees your concierge security team to focus on higher value strategy and proactive risk reductions while the agents handle the grind. If you want to see what trustworthy, production-ready AI and security operations actually looks like, go to Arcticwolf.com slash hacked. Never feel like cyber threats are evolving faster than anyone can keep up? Last year, 2025 was nothing short of a record-breaking year for major breaches,

from sophisticated ransomware operators to AI-enabled attacks that turned defenses on their head. Organizations around the world saw headlines they never expected, and cybersecurity teams were tested like never before. But here's the thing. These incidents aren't just news headlines. They're learning opportunities. And that's why Arctic Wolf is hosting a live webinar on February 5th. diving into the most impactful breaches of 2025.

Their field CTO and security leaders are going to unpack not just what happened, but why these attacks succeeded, and most importantly, what businesses can do to fortify their defenses for it's too late. You're going to walk away with real insights into how threat actors are evolving, how defenders are responding, and what strategies can help you stay ahead of the next big breach. It's not fearmongering. It's practical, actionable, intelligence from experts in the trenches.

Register now at arcticwolf.com slash hacked. I'm curious to learn more. So you brought up Glaze, which, as I understand it, is sort of the artist-facing tool. It's the thing that you can put an image into to give it these qualities that are going to cause it to be misinterpreted by an AI. I also understand there's Nightshade. Can you talk a little bit about these tools where one begins and one ends, how they work? Take me through that. Yeah, absolutely.

So, nightshade is a direct kind of follow-up to Glaze. You try to improve, do something more. So, you know, if we take the AI company perspective, right? So if we look at this, and the worst case with glaze is, okay, there's some data I just cannot learn from. That's not too big a problem, right? Because there are just so many other art out there are not protected by glazed, and there's so many, you know, historical artists that can try those. So it's not too big problem.

So for nightshade, what we did is, okay, we can take this one step further is if you train on these nice shaded data, what happened is you will not only not be able to learn anything from these data, but you will also corrupt the base model that you already have. So corruption can come in many forms, but we're showing the paper that you can basically have the model to output a cat when you ask for a dog or all sorts of weird stuff you can have the model to do. And this will have a pretty big impact on, you know, how trustworthy your model is.

If you put out there, you can generate a basic concept, right? So that's kind of what I initiate that. Oh, cool. So Glaze is just about making it so it can't read. It's turning it into text in the language that it can't read anymore. Nightshade is like, no, this is actually going to deceive it. You're looking at an image of a cat, but what you're seeing is an elephant. So the next time we ask you to produce a cat, it might have a trunk.

Exactly, exactly. Oh, that's very fascinating. So you're poisoning the well, essentially. Basically. Yeah. Interesting. What would it take? either of these models for them to be really useful at scale in the real world.

Where do these models and, say, large social media platforms where a ton of the data that's being scraped to train these models are coming from? What role do they have to play? How does this go from being a tool that an individual artist has to upload an image into, get it back before they upload it somewhere else? How does it get more useful than that current state? Yeah, absolutely. So we are, well, we're fairly already kind of talking to all of this.

But so currently the model is basically, as you said, artists will download a tool and generate take quite a long time, so it's not very scalable. But we have been talking to our platforms. These are shared platforms. Some are very pro-AI, it's great. But there are quite a few of them saying they should protect copyright of their artists who share the image online. So we're right now working with one of the leading platform in the space, and I think they already start integrating blades into the platforms.

So every time they share a image online, you basically have the option of plays and or not. So that's kind of where we're going. And we'll also talk to quite a few entertainment and gaming companies. They're in a little bit sort of weird position. They want to use AI because you'll save their costs. But on the other head, they have a huge artist base that they don't want to piss them off. So I think they are a little bit tricky. So we're talking to some of them, we're talking to the right game a little bit.

but they're also interesting in integrating glaze inside. Also, the entertainment company has basically the exact same problem. They're saying, you know, Disney recently sent a letter to Microsoft say that you can't train our Disney characters, for example. So, like, they're obviously very interesting in protecting their IP, but as I'm clear, their true incentive was going to be the future for them. Yeah, there's a really interesting kind of tension that you bring up there where presumably some of these companies, would love the idea of not having to pay the labor cost of a bunch of artists to generate some of these assets. But I would bet more than that, they would be really angry that you can type Mickey Mouse into a box and have it produce pretty good images of their very valuable

intellectual property. And here you come with a product that goes right down the middle of that. Exactly. Do you feel like you're waiting into the middle of a, kind of a pretty long-term fight that's going to be playing out in that intersection of tech and intellectual property. Like you're sort of, you're setting yourself up to be right in the middle of that for the foreseeable future. Yeah, absolutely.

I think when we start a project, we absolutely does not expect this at all. But then, you know, once we released, I was okay, this is a real problem. People are facing, and, you know, everybody is talking to this space. But yes, I think right now we're more kind of committed to just be in this space to,

I guess we frame ourselves as, you know, provide technical solutions. when there's no regulation really in the space at all, right? Like, there are people active in the Hurtian. We're all waiting on the court cases who are all waiting for, you know, the legal action to take place. But they're going to take some time. So the technical solution really should be there to help some of the creators

to at least get by it for the second. Mm-hmm. You brought up regulation. I'm not, I'm definitely not asking you to draft up some regulation on the fly on this podcast. But like broad principles, maybe. Some sort of like nice to have, vague stuff, again, not asking you to write it.

What kind of things, maybe where do you think it's going to go in terms of regulation? Let's start there. I think it's going to be a little bit hard. I think there are just so many misalienies and tips in the space, right? There are, you know, of course, we want to protect labor.

We want to protect graders. But there's also the aspect of, oh, we have to build the next big AIS system before some other country does it. So the discussion around this has been very slow. We'll see how the court cases goes. But even though court cases went well, there will potentially be new laws from the Senate regulating AI

for better or for worse. And this is only the U.S. And there's, you know, the whole Europe. Europe was going pretty well with AI acts until recently they kind of stopped. There are a bunch of tension around that as well. And then there is Japan and China, which is much more pro-AI.

I think Beijing just had passed a law yesterday say you can copyrights, AI generate images, and things like that. So, so I, my take is even in the longer term, I feel it's never clear. It's going to be a clear cut, right? It's not going to just benefit artists completely. So I feel, that's why I feel like technical solution in some sense is useful, but also use some sense technical research can also push for certain regulations, right? Like, if we show, okay, these are how many artists are impacted, are carrying,

want to protect their tools or protect their art, that may have the implication of the ongoing legal discussion. Yeah, it's hard to say we need to pass some sort of law regulation that platforms need to do something to images hosted there to prevent them from being scraped without the consent of the creator if that tech doesn't exist. Yeah, like, you can't even start that discussion. Interesting. Yeah, or like there's similarly, I think there are a lot of platforms. I'm trying to say, let me compensate these creators. But compensating through a model is going to be very hard. How does each training data point contribute to a given generation and things like that?

So technically solution, I think, in the space is also very important. I can't help but just see the knock-on effects to have any kind of precedence set in court or legislation because, you know, we're kind of at AI, you know, 1.0 per se. And it's only going to get bigger and bigger. Like one of the things that I look at when I see stylistic mimicry, Excuse me. One of the things I look at when I see stylistic mimicry is like, you know, we're doing images now. We've got some text generating.

You know, I can have an AI, write me lyrics to a song. I can have AI modify an audio track that I sing to sound like any other artist. We're only a few steps away before I can just be like, hey, write me a Drake song about, you know, dancing in the flowers. And it'll be like, boom, here it is produced and out it comes. And it's like, you know, any kind of decisions that they make in the course, core cases today will impact all the future AI generated content. So it's just, I hope that they're weighing, you know, in the balance as how severe some of these decisions will be. Yeah. To the creative class anyway. Yeah. Yeah, exactly. Yep.

So you're not working on this alone. I'd be curious to know about like, kind of tell me a little bit about the whole team. And it's specifically what kind of, because this seems like such a new thing, what sort of backgrounds are people bringing into this project? Yeah, absolutely. So I think it's a little bit weird for us even. Like I say, we are a traditional research team. So I'm doing my PhD, doing my master here. And then we start out just as a research project. So traditionally, we kind of research in the space of security and privacy of machine learning system. So a lot of looking at how AI works and when does AI fail, we look at things, you know, how do you make sure AI self-driving car is secure? So we do a lot of that sorts of stuff. But then I think when

generative AI really start picking up, we start to talk to quite a few and then we see how severe the problem is like last year. So like, okay, we should steer our whole project, our whole focus into, you know, protecting artists. And the reason really is that we're kind of in a very fortunate position because we study these AM model, we'll study their variability for quite a long time, right? So we're like, okay, we know how to exploit these variability

as a protection tool for these artists. So that's kind of how we started. So the team is fairly small. As my advisors, my two advisors in the space, they has been doing kind of CS or commerce science research for the longest time. And then there is us, me and my two other co-authors, we just do with our PhD, but our background is more on privacy and security of AI systems and kind of share, kind of do more other generative advice stuff these days.

I'd seen two different numbers floating around for this in regard to Glaze specifically. I've seen a million, 1.5 million. Roughly speaking, how many artists have used these tools you're working on to date to protect images? Yeah, so we don't have the exact number of how many people are actively using it every day. So we only have the number of how many people downloaded the app from our websites. So I think in July it was 1 million. I say we got to 1.6 million as of, I say, last week when I checked. But this is just a number of full downloads.

They download the whole package, download all the resources, and perhaps start using it. But, yeah, we do not keep track of any say, after they download just for privacy reasons. But we also have WebGlaze. It's kind of a service we put up Farras, who doesn't really have a laptop, or does it have GPUs to run Glaze. For that, I see we have 3,000 active users, but we have a huge waylist. We haven't really get around to put people on the wayliless just because we don't have any of GPUs for the moments. But yeah. If somebody wants to support the project, are you guys taking public support?

Or is it just an internal project for the University of Chicago? I think we very recently worked through the university to have a donation platform. So you will go through the university, but a portion will come to us to continue research in the space. Great. So you've mentioned protecting artists. You've mentioned technical solutions being really important for figuring out the legal side of things as we move deeper into, you know, the AI era. Beyond those things, I guess just for you personally, why do this work? Why kind of take on this really long-scale battle with

these very well-resourced companies that have a huge financial incentive to keep these models ticking? Why does this matter to you personally? I see, just, okay, I think there are a couple of answers and there are typical answers. Okay, I want help people. Of course, I want it. There's so much. very rewarding feedback to artists and working with them just very enjoy that process. But also I think the bigger reason really is that I'm just interested in protecting human creativity in some sense, right? This is basically it.

I feel like the companies or even government are taking a fairly fort-sighted view on AI, right? Of course, these are awesome. They are able to copy style generic images. But what I see is if we keep going down this route. say, you know, they get better and the artist getting replaced. And that more or less kind of mark the end of human creativity, at least in some aspects. And it's unclear where are we going to go from there,

because as we see today, these models are not really able to evolve on themselves. They're mostly still feeding or mimicking existing art. So a sad future will just be, we stuck with the same type of art for, you know, hundreds of years and we can't go back because there's no more artists, there's no more art school. So I think really, really, want to push for that, just give artists some leverage in this negotiation to protect the human

creativity at this point. Yeah, you use the phrase protecting human creativity. And it's, it is funny how within about four seconds of using one of these things, you're sort of like left with these two competing feelings. One is kind of sort of like the technical awe. I'm sure as a very technical person, you really are like, wow, this is remarkable what you've managed to achieve. This is catastrophic potentially for human creativity, at least in terms of it being economically viable. There will probably always be someone wanting to pluck on a guitar in their living room. There will always be someone wanting to draw on a pad of paper. But whether or not it's a job, a career, something that you can make a living doing,

you're immediately struck by how big a threat this incredibly cool tech could be to that as just a thing people can do for a living. I think so when we started off, like I saw these two, I was like, Like, I was more on the other side. I was, oh, my God, this is amazing. I can't generate out, no, like, Darth's weight, eating sushi or whatever, like, whatever I want. And, and, but I think once we start talking to one more artists, that specifically, like, every time I gave some of these talks to, like, you know, a group of people, always there are parents come ask us, okay, should I still pay my Saz arts school tuition? Like, these are the questions we start getting.

And so, well, okay, this is very much the question that defines a human grade. Timothy in the future. So yeah. Okay. My last question for you for you, Sean, is where, where do you think this goes next? We're having this conversation again in five years. What do you what are the big moments? Where do you think this goes? That's a great question. I think it's hard to predict where AI is going to be in five years, but sure, but I think, I don't know, I feel like there will be at least some regulation in the space. If we say there's GRO right now, you know, it may not be great, but that there will be some. And I think a lot of our just want to fill the gap that regulation cannot catch, right?

Maybe they're able to take care of open AI as stability, but really not the random redditor online. So these are the cases, technology or understanding can really kind of take in place. But also, I think we are in this kind of a cultural shift with generative area, everybody using it. And just to understand how this impacts humanity, maybe for better or maybe for worse, to understand this space a little bit more and build. tools to help people to help steer AI to the place that we wanted to be.

Sean, thank you so much for your time. Really appreciate you sitting down with us and chat about all this. It's a very cool project, and I look forward to seeing where it goes next. All right. Thank you so much for talking to you. Yeah, take care. All right. Thank you, Jordan.

Thank you, Scott.