The Standup with ThePrimeagen - This Doesn't Look Good for AI

Episode Date: May 12, 2025

ssh terminal.shop

Today is the stand-up episode four where we're going to be talking about all the things that are important to devs. My name is the Primogen, the host. And today with me, we have... My name's CJ. Ha, ha, yes. Nice. And who else? Casey. Trash. Trash, you're frozen. Let's go.

Good enough. And so that is good. All right, frozen trash is pretty much the same thing as a normal trash. So today's first... Wait, I'm frozen? Yeah, well, you were. Today's first hot topic is going to be a court case that was recently tweeted. by Casey himself describing that AI and the training data for AI may or may not actually be legal to use.

And so what are the implications coming on down? This is with Ghibli Studio, which Casey specifically pinged in it. And so what are going to be the outcome? So Casey, since you open to this can of worms, why don't you guide us somewhere with this conversation? Wow, you're going right into heavy-duty stuff here. Yeah, yeah. This is the stand-up. This is not the mess around, okay?

We don't talk about our dogs here. All right, all right. So, no, this was not about Studio Ghibli. This is actually from a case. It's Thompson Reuters is the, are the people who are filing this particular lawsuit. Or I should say, the people who are worried that their copyrighted work is being infringed unfairly. And in this lawsuit, so just to give you sort of the big overview, I, everyone agrees that AI training is copyrighted. infringement, at least in the sense that they are taking copyrighted works without licensing them.

So there's no argument there. AI companies aren't going into court and saying like, what? No, we didn't use any copyrighted work. Like, no one's saying that, right? They're all saying, yes, we did. But their argument is that somehow this is okay, right? And so what happens when you go into court and on that footing, right? Your legal footing is we're violating all these copyrights, but it's okay.

for us to do it, you have to provide what's called a fair use rationale. And just to be clear about what this is, copyright is a worldwide thing. It's international. The U.S. is a party to something called the Byrne Convention, which is like a uniform agreement that copyright is respected everywhere, right? So it's not some weird, like, only U.S. thing. It's not some, like, finicky thing that only Congress decided or anything like that. This is a worldwide agreement that when someone writes something, they have the copyright for it. And they have to have, and they have this right to exclude other people from using it in ways that they don't like, right? Now, in the burn convention, it allows countries to make exceptions to this on very limited

grounds. So basically, a country is allowed to carve out some special uses where you can make a copy without the original author's permission. And in the U.S., what we have carved out, you know, over the years for that, is called fair use, right? Now, this doesn't just come from the burn Convention because copyright existed before that, but the burn convention is now kind of the backstop, right? It was in the 70s. There's been, people keep ratifying it, so it gets, you know, more and more countries get into it until now it's basically everybody. So in order to have a fair use exception, in order to say I can, you know, take these copyrighted works and use them without a license, you need to demonstrate that your use doesn't, you know, doesn't violate, there's basically this

four-pronged test. And the most important of the prongs is, you know, it's, you know, doesn't violate, there's basically this four-pronged test. And the most important of the prongs is essentially that you are not doing any economic damage to the rights holder, right? That's the very most important one. The Supreme Court has said it's the most important one. All the courts have consistently followed that guidance that, like, if what you are doing is creating an economic negative outcome for the original rights holder, it's probably not fair use, right? Like it's highly unlikely that it's going to be fair use because that's the main thing that copyright was designed to do is to give money to the people who've made the works so that they will continue to make them. So hopefully that all makes sense. I think it does.

Yes. So you said the burn convention, that's the thing in the desert that people go to every year? Burning Man? TJ, how long have we been planning this joke? Like, how long has it been? He thought about it as soon as he said, I saw it. his light bulb just going. I was just waiting for a boss.

I could make that joke, but I was going to pretend I didn't know for real. Sorry. Brian, you were, okay, no way. No, that's how it works. All the countries have to send their ambassador out into the desert. They all make a really big, weird thing, like sculptures, and people serve drinks. Everyone's high. And then they sign this treaty.

They don't know what it says. It's probably written in some foreign pictograph, like language thing that hasn't existed. You know, Sanskrit. grit. And that's how we do copyright now. And it's fine. Sometimes they get stranded there. We hope there's not a windstorm because then there'll be no copyrighted works the following year. Yeah. I mean, it makes sense. That seems to be on par with government. That's the only reason that the AI companies would think they could just steal everyone's stuff and then make money off of it. So to get into, this is why I said, you're opening a huge can of worms here because, like, I love studying law.

It's one of the things I do in my spare time for whatever. So, you know, I just, I'm into it. So anyway, The thing that I've been saying for quite some time, actually, is I don't really understand AI company's defense. And the reason for that is their primary defense has been that this is fair use because the thing that comes out of the AI is not going to economically compete with the things going into it. Now, that's kind of tenuous on its own. And we don't really need to go into that to talk about this particular tweet.

I think that's a pretty tenuous defense to begin with because they rely on a thing called Authors Guild v. Google, which if you read it, seems like it would have come out the other way had Google had their AI summaries at the time the case, right? At that time, that case was litigated. Google just had search results, right? And the little snippet from the web page.

It didn't have the thing that basically makes it unnecessary for you to go to, say, box office mojo to find out what the box office is. So I think actually Google would have lost Authors Guild v. Google today, and I think AI companies aren't appreciating that. So I think that defense is also very tenuous, but you can put that aside because maybe you don't think it is. Maybe you read Authors Guild v. Google differently than I do, whatever.

The thing that I've been saying is, who cares about that? The market is AI training data. We know that's a market. I know you guys have received this too. I have a YouTube channel. It's not even remotely as popular as Primes, for example. I've got emails from people offering to pay me money to license my YouTube videos for AI training. So we don't even need to hypothesize that their market exists.

It already does. People are already paying money for this content. So I'm like, look, it's an open and shut case. The market is the market for AI training data. They're not paying us and they owe us because we could be generating revenue from our content by licensing it to AI. companies to put it into their algorithms. And so I've been saying that for quite some time. That's been my general position on what this is.

We don't know what the courts will find, but other people have made this argument. New York Times made this argument in court. We don't know, well, made this argument in the initial filing. Their case hasn't percolated through the courts enough to get an opinion on that yet. But this first opinion that came out in February that I missed, and I only read it yesterday, actually holds exactly that. in this opinion, finding for Thompson Reuters that their copyright was violated and it was not fair use, the court said literally this. It said, there is potentially a market for this training

data. You are stealing their copyright. You're infringing their copyright in that market, and that could have negative ramifications for that was literally in the opinion. And so I thought this was a very positive sign because this suggests to me that courts are thinking that way, lawyers are making that argument and it's very possible that it will be found to be that way as more of these cases go through. So to me this was a very positive outcome because I think

companies should have to pay for their training data. And that's the entire long-winded thing, so hopefully everyone now has context. Take it away team. Wow. That was definitely some context. I feel like TJ, I can just see you the whole time you were bobbing, you were ready.

I missed the whole thing. I already have a lot of feelings on it. I I'm feeling a lot of things here. I have a funny question and a serious question. Which one do you want, Brian? Obviously, start off with T.J. Funny, which is the best funny. If they train on Trash's code, does Trash have to pay them or they have to pay trash? Ooh, sick bird.

No, no, for your code, trash. Yeah, yeah. Pay me. For liability. You damage to them, bro. They pay you, but you are liable trash. You are liable for the code, but they pay you. Only I can debug that code, baby.

Job security. When the AI company gets sued for the output, right? Like when it's like, look, this thing, you know, deleted all of our data. And they're like, well, I mean, it's just, we can trace it back to this training set from Trash. And I don't know what to tell you. Talk to him. Trash has been selling all of his data. He just cannot wait for this.

He's just, come on, get all my cover off. I'm kind of sabotage AI. Okay, my serious question is, though, like, so let's say codes on GitHub. Did I sign something? I mean, obviously I read the TOS. So we can put that away. Obviously, I read it. I'm just testing you, Casey.

Did I sign something to let GitHub, let people do this without my permission? So that's going to be a really good question and a really difficult question. I haven't tried to look into that because the reason of that is there's such a thing as a contract of adhesion, which is what you are effectively sort of quote-unquote signing when you agree to use a service, right, in a way. It depends how you want to look at. Terms of service and contract of adhesion are technically a little bit different. Like a contact of adhesion is more like the thing you sign when you go to get a cell phone, you know, or something like that. But it's the same kind of principle.

There are limits to what you can give away. You know, if the cell phone company had buried a little tiny clause in that giant contract that said, and also we can human traffic all your kids or whatever, right? It's like it doesn't suddenly, I mean, that's going to be criminal as well. But, I mean, you know what I'm saying. Like, they can't just put something arbitrary. egregious that would, you know, quote unquote, offend the conscience into one of these boilerplate things and then actually expect that to get them out of jail free when they go to court.

And so I would have to say that's a much harder question. I would say you probably, I would imagine you wouldn't have to worry too much about Microsoft's competitors being able to do it because the only people you probably gave an agreement to was them. So it might be training. co-pilot or maybe an open AI through some agreement, but Google would probably never be allowed to do it because Microsoft wants the competitive advantage

of all the source code on GitHub. My recommendation, and I have been thinking about doing this for a while, is I think everyone should probably be going dark these days. Everyone should be pulling stuff and putting it on their own website with a little notice that says if you want to use this for AI training, you have to pay me. That's, I think, the best practice. You know, let's say you gentlemen are distinguished web devs

and you know a lot of distinguished web devs, if you put up the AI-safe GitHub tomorrow, I would move everything that I have to it literally as soon as it was up. And I think a lot of other people would too, and I would also encourage everyone else to do so. So that would be a great competitor to GitHub. If anyone wanted to do that, I would even pay a fee to do that,

meaning I would pay for a good GitHub clone out there that would get me away from licensing Microsoft implicitly, Because like I said, I don't know if that. It will turn out to be something that they're allowed to do by slipping it in a TOS, but it might be, and until a court case, we wouldn't know. So the safer thing to do is to just not use it. So could you use an AI, an LLM, to generate a GitHub-like interface trained off of code on GitHub to then launch a private AI-free service?

Is that what you're trying to tell me? It'd be an interesting thing to do. but again, based on what we have seen, you may very well be liable for copyright infringement because of the way these court cases are going. Because if you weren't Microsoft and don't have the effective, like, you know, in the GitHub terms of service,

there's probably some clauses that say, like, you grant Microsoft, or, well, GitHub, but, you know, as a subsidiary of Microsoft, worldwide, you know, license to reproduce this work or whatever it is, right, because they need to for operating the service. And they'll just argue, well, co-pilot is part of the service.

service, so we train it, right? Other people don't have that out. They can't go to court and say, like, well, you agreed to it, right? And so, eh? So that's kind of a, yeah, I mean, that's going to be a gray area, even if things go the way this court case did and the way that I think they should, which is the air companies need a proper license, you know, who knows, who knows what that, what that looks like. Will anyone, will anyone using AI? Is there any sort of like trickle down to this where if people who are using AI, will they become also like tangentially liable for what they have generated?

So that's very possible, but also I think that really depends on people. So this will transition nicely to another thing I want to talk about. And you just tell me when we're ready to transition. Tell me when I have it. I have it. I'm ready. Hold on.

I want to say something. Okay. So we're going to, let's get through all this first. And then we'll transition. So to say like that, this, let me talk. Trash. Cheats. I'm going to bring this back to the studio Ghibli scenario,

because that's what everyone's probably familiar with, right? Where that was a whole big debacle. Yeah, Trash isn't checking out the laws, but he did see the pictures on the internet. He saw Sam Mawton and looking quite jibly. That you want to talk to about law, okay? No, you said you study it. You said you study it.

He knows who he's talking about. I said he knows my pirate all nine. That was not real. Okay. Riddle me this scenario. Okay. So we're going to bring this. back to art. Again, I think art is like where it gets really, really muddy. It's probably not as

cut and dry as like this Reuter scenario, which is like a very niche kind of area, I feel like. So riddle me this. I'm an artist. I don't really care if people use my work. I just want to be well known. I just want people to see my work. But as a human, I have inspirations from other people. Maybe it's Studio Ghibli. Maybe it's a number of other very popular artists where it's very obvious if I'm stealing their work. So when I draw, you're probably going to see a little bit of those details come out. All right, but I allow you to use my work for your training data because I don't care. I have a full-time job doing something else, you know, drawing is just my hobby. But when you're training on my data, some of those little qualities come out. Maybe I like Studio Ghibli eyes, you know,

maybe I like Dragon Ball Zabs. I don't know. So what happens if that comes out and then someone else notices it and you're like, wait a minute, that's Goku's six-pack, right? Will I be in trouble? do you get in trouble or is it fair use because I don't know I put like an extra shadow on one of the abs or something I don't know so how that work? You're basically talking about

is there a scenario where there's effectively copyright laundering so we take Studio Ghibli some other artists draw something sort of like it and then that artist gives them permission to train gives an AI permission to train that's what we're talking about just to be clear

Yeah. My assumption, A, that is the reason we have courts. In other words, for weird coroner cases like this, if someone really cared and was worried about it, they would go to court and a judge or a jury would look at that evidence and say, do we think this was really against the spirit of copyright law or not? My gut feeling is that would be fine, meaning that, you know, there isn't Studio Ghibli is not marketing educational materials for humans right now, and they are not marketing materials for AI training right now. If things go the way I assume, they will do the latter only. They will say,

okay, here is our pricing for AI training, because now that's a market, and we understand that. So AI companies have to pay that if they want to. If they don't then say also, if humans want to train on it. You have to pay us this, right? And therefore, these are educational materials, then no. And also that thing, again, all of this stuff is based on licensing. If they provided a license with their training course that says, also you agree not to draw things too close to our style, right? If they don't do that, then there has been no, like that causal chain is not broken. A lot of people get confused because they think for some reason humans and machines have to be treated identically or something like this. But actually, not only do humans and machines not have to be treated equally under copyright law, but humans and humans don't have to be treated equal under copyright law.

You can sell something for one price to professionals with one license that says you can do anything you want to it, and by the way, it's $5,000. And another one to students that says it's $50, and you can't use this for anything that makes money. That's completely legal. So again, you can copyright law is supposed to allow the person who made the thing to determine how it is used. And if Studio Ghibli wants to make special screenings of their movies where a big thing comes up on the screen that says no one can draw like this. If you are going to look at this movie and try to draw like this, leave the theater now because that is not allowed and your ticket price will be refunded. They can do that. Gane, is that contract of adhesion that we talked about earlier?

Yes. Do you guys remember when on Facebook, like probably 10 years ago, when everyone started posting that Facebook cannot use my posts for like anything and everyone just kept posting it as if it did anything and it was the funniest thing ever? I'm pretty sure my grandparents have been doing that recently. Actually, these posts are not allowed to be used by AI. My parents are just like, you do not have my permission. I'm just like, oh, my God.

It's too late, brother. Your parents need to put a price tag on it. That's what they're missing. Put a price tag on there. Say, like, in order to use this for training, it costs $1,000. Put it on there. Put it on there.

It might help you out someday. Here's my next question. Suppose they find out, okay, couldn't train on the GitHub stuff or like couldn't train because people had some GPL license on their code and it was in clear violation of the license as stated the way that they used to train it.

Right. Do they're going to have to delete the models? What is the next step? Let's say they find it out, right? Is it like, we got to roll these back and we're starting back at GPT 2.0? Yeah, so, again, this is, and it's important for everyone to remember all the things we're talking about are somewhat hypothetical. We know what happened in the Reuters case. It's a district court, which means there's two more levels to go, right?

Because AI companies are going to have to fight this because, you know, they're in a lot of trouble, potentially. So there's district court that gets appealed to us like an appellate circuit, right? You always feel like the Ninth Circuit or something like that. Then that gets appealed to the Supreme Court. So we've got we don't know what we're talking about right now, right? So just important everyone's aware that we're just saying hypothetically if this stuff happens, here's what's going to go. But we don't know that. So anyway, to your hypothetical, right, usually in copyright cases, the first thing you get is

injunctive relief. They have to stop selling whatever the things that are infringed, right? Right. So my assumption is that at the very least, they would have to literally rerun the training round. They would have to pull that

content. Let's say Studio Jubilee won it. They'd have to pull all the frames of all the Studio Jubilee things and rerun the $50 million training round. I don't know what they cost these days to run, but they're in the tens or I don't know if they're in the hundreds ever, but they're in the tens of

millions oftentimes is my understanding. What about distilations? Like things like, you know, like R1, one of its secret sauces was that, you know, it could verify some of its data against OpenAI and be like, okay, so this is like a great way to predict what's coming out next. Does that also, excluding if China were to listen to it, does that also mean that all derivative things that used O1 as part of the training set also have to then fall under the same copyright problem?

Potentially, but again, maybe it's too attenuated at that point. I don't know. And again, this is why I say, like, this is going to be something courts are going to be dealing with for a while. Because it's like that, you know, just Studio Ghibli see that, you know, well, and the other thing, too, is people always say like, oh, well, well, you know, that'll just ruin U.S. companies. It's like, no, this will apply to everybody. If you're trying to come into the U.S. market and sell something, they train somewhere else. If it was based on copyrighted data, they'll get the same exact injunction slapped on them, right? They can be taken to court just like anybody else.

And again, also, none of this stuff really applies to research, right? because that is actual fair use. So if you weren't commercializing this AI, you're not providing it to the public, you're just doing it in some lab somewhere for research and publishing papers on it. That would probably be totally fine fair use in all these scenarios.

If you're a government agency who's doing this for national security purposes, like opening, I usually try to use the national security thing. It's like, as if generating Studio Jibliab, which is national security, that stuff is separate work, right? Those are separate things, and they can be cabined off,

and they can use copyrighted works. But the instant you say, oh, hey, we're open AI and you can come pay to use this, Mr. General Public. That's the commercial product and you're on the hook. So, you know, if the court cases keep going the way they're going, all the things that we're talking about are plausible things that courts will be asked. Absolutely. And the reason I asked in particular about rolling it back is my understanding is,

generally speaking, they are using a lot of transfer like learning, right? They're taking weights that they already did before. they're using that as the beginning of the next round of training. Now, I don't know for like new models, whatever. My understanding is though, like when you do these things,

you do a lot of transfer learning. That was one of the things that they found out that was crazy about neural nuts. It's like, yo, we can train it over here and we can train on something else later and it does better. And you're like, what the heck? We don't understand why.

And they're like, okay, because of math. Sure. Sure. Math. Like magnets are real. It's true.

Sorry. And TJ learned about ICP just like two weeks ago and learned about magnets. And so he's just so... I still don't actually know what it is. But just the magnets are, that joke's too good for me. I love that joke. I have made it many times that everyone's like, what are you talking about? I'm like insane town posse.

Anyone? No? Okay. TJ literally just discovered it last week. So he's very stoked right now. But so that's kind of what, like, that transfer learning aspect is really... Because I wonder how far they would try and pull that back. Right?

Because like initially they were just literally reading all the code on GitHub for sure. We know because they would be like start typing out this GPL code. And it completes the rest, including source comments and saying it's GPL. Right. So it's like, okay, well, you, that's definitely against the license. So that part is very, I don't know, it would be interesting to see how that unfolds. Well, we also have Facebook admitting in public filings that they went to like, I mean, they effectively went to Pirate Bay to download stuff for their training.

Like, they knowingly violated, like, literally international law to get their training data in the first place. And so you kind of have this situation where these AI companies are completely on the hook. So the only question is what are courts going to decide about whether any of this is fair? And like I said, I'm optimistic because of the Reuters case. I don't know what will happen in the future. I'm optimistic that they won't, you know, can I get Kat on stream? There you go, yeah, cat on stream's fine. I'm allergic to cats.

Oh, I'm sorry. I'm sorry. Good. Sorry, trash. But no, no. By the way, the fun part about the Facebook one is that there's an actual email from one of the, like, leads that's just like, I probably shouldn't do this on my work computer, sweating, smiley face. It's like, oh, my gosh. It's amazing.

It's, it's nuts. I looked at that stuff and I'm like, what is happening, right? Like, they put that in an email. They like, yeah, Zuckerberg's like, yep, that's probably a bad idea, isn't it? Yeah. So anyway, you know what? Can we transition now? Yeah, this is the thing.

I assume that you want to do the Shopify one, right? No. I wanted to talk about this because, and you guys, you basically got there without me. That's the thing. I have this concept for the AI future that people haven't been thinking about. And it's all the things that you all just said, right? You were just hypothesizing. And I came up with a term for it.

that I want to try and make, you know, the way we sort of conceptualize this. I call it the taint. Okay. I like the taint. So React already does use this. So just so you know, the taint already does exist. Forsell might come after you. I think they came up with taint.

It might be the taint will come at you for copyright. Maybe we can work out some license agreement so I can use the taint for this. Okay. TJ is not participating in this conversation. It's possible that we are heading. into a future where everyone is going to need to know if their AI has the taint, right? If your AI has this taint because it's had the copyrighted work into it, right? And you can have like a lot of taint or a little taint as well, right?

So there's a certain amount of taint when we look at this AI that we're going to see, right? Taint's girth. Exactly. And so it's like, will AI companies have to pivot to demonstrating that they have, you, the least taint, effectively, or they are unlikely to have taint when you use it, right? I'm watching Teage. Teesh has like 20 jokes lined up right now.

I'm just like, I'm staring at his T's weird. He wants the jokes so bad, but he's just like knocking into it. No girth jokes, no length jokes. I'm going to miss 100% of the shots I'm not taking right now, let me tell you. So I would like everyone to think about this possibility that there could be like a future where everyone's concerned about the taint of AI. So there's a taint. Oh, I bet you there will be like some sort of judgment. Like if you can prove in some reasonable amount that your taint scores below 5%.

It's kind of like how you could, there's like a certain amount of rat feces that can exist within edible products. That they're just like, that's unavoidable percentage. The end, there are body parts and such in it and that's it. So therefore you can also have some unknown amount of taint factor. That's disgusting. By the way, Casey has just disappeared on us. Apparently he spoke a little to authority. authoritatively on the taint and now

now he's out. Taint AI. You're building it after this. Amazing. Can you hear me? Yeah, now we can hear you here. Here you again. Hey. The AI is fighting you. It hurt what you said.

So anyway, I'm in, like, I want this future where everyone is inspecting the taint. Right? They're trying to find, they're chasing the taint. They're like, look, we have to figure out because if we go to court, the last thing that I want, if I'm a CEO, I don't want to go to court and find out that suddenly I actually did have the taint, right? Because I, and I didn't know, like, we used some product, we licensed some, you know, thing.

And the taint, we ended up with the taint, right? This reminds me of a wheel time. We thought we were getting a coding assistant, but all we ended up with was taint. You just do like a smell test, right? Yeah, it's a sniff test. Yeah, this reminds me of the wheel of time. You know, Cyan was also tainted. You know what I'm talking about, TJ?

I was thinking, I was literally like, should we just have a 15-minute conversation about wheel of time in the middle of this? that would be good for stand-up. Classic stand-up. No, this actually is pretty interesting because there has to be a certain level of copyrighted material that the courts are going to say is okay to be within an LLM. Because there's just like some way that, like there's just going to be things that you will never know

because someone stole some work that you could not possibly ever attribute and it somehow got into it and therefore that's that. And now they're saying, okay, well, you can have some percentage, but you have to be able to prove that you're below some threshold. Well, it won't necessarily be a percentage. It'll just be like some number of people will sue you. And you'll make licensing deals with them, right?

Because what's going to happen is it's copyright assertion is like on an individual basis, right? It's like Disney goes and sues them and then, you know, assuming that they get a similar decision to Thompson Reuters, then one of two things happens. Either all of that training data is pulled and they have to rerun the models or they have to pay Disney. And my guess, and I could be wrong about this, but at least for the same. the big companies like Disney, what they're going to do is they're going to get a little that sweet, sweet mullah, right? So you're not going to have to pull Disney stuff. You're going to have to pay Disney their little AI tax, right? And Open AI is going to be taking some of those, you know, they won't be venture capital rounds by this point, presumably there'll be a public

trade company or something like that. Once they figure out how to turn themselves out from a nonprofit or whatever, they'll have to pay some of that, you know, all of that AI income or investment money, which is what they currently are running on investment money, they'll have to pay that to their copyright people. By the way, I would like to say that Trash is doing the most classic stand-up behavior, clearly

typing on his keyboard. I'm typing in the chat. Well, shut up. Okay, Trash, shut up. We're working here, okay? Focus on. I'm listening. He is director of community outreach. That is his job.

He's doing it. He's doing it. You lost me at the taint conversation and then my head just starts spiraling into like 50 memes. Yeah. Well, the good news is, is that we can move on from the taint. Okay, we do not have to continue down this road to this logical conclusion. Instead, we can pivot. All right. You guys ready for Topic 2?

Sure. Which is kind of really just like an extension of topic one. And so that is going to be this right here. Shopify has announced from the tippity of the topity that everybody must use AI. Toby effectively went on to say that he presented about it and called action for an invitation for everyone to be able to use it, but that was not enough. And then he goes on to say that people who used AI are not just hen X the engineer, but in fact, 100X the engineer in work being done. That's right. For every single three and a half day work they've done, they've done an equivalent of a year before they started using AI, which is quite a wild amount of work.

and then finally he lists out all the things that he expects you to be doing when you're getting or when you're using AI. For those that don't know a GSD as GSD just is a simple get shit done, that's like what it means. And so German Shepherd dog actually. German Shepherd dog. I was very confused when I saw GSD. It's a German Shepherd dog. If you had one, you would understand. I do not have one.

So that's probably why I'm missing a little bit of the context here. But Shopify has officially stated that you are to use it no matter what in every single one of these cases. Now, I don't know if there's any sort of, they didn't really list any sort of negative cases where you do not have to use it, but it went from a suggestion you should do this to now this is a requirement from here on out at Shopify. And one of their justifications was actually found in this previous paragraph, which is in a company growing 20 to 40% year over year, you must improve by at least that every year to re-qualify. So you need to improve, I'm not sure on what metric they're exactly saying, or how, they're suggesting that they rate this improvement, but you must improve at the same rate the company is in fact growing monetarily. Which makes no sense, by the way.

I literally have no idea what that is even supposed to mean in practice. I'm not sure he does either. I'll defend that line, even though I don't agree 100%. Pretty sure they're just saying the market for Shopify style sites is growing by this much all the time. so if we don't keep getting better someone else is just going to snatch those up because those are like brand new customers on the internet making Shopify sites

or like in their in their vertical the like Tam is just growing by 20 to 40% every year and they're like we're going to get surpassed by this right if we're not doing this then we're just going to get trashed that's why I thought they were okay so that you must mean the company because if you look at the thing it says in a company growing 20 to 40% year over year

you must improve at least by that oh maybe I misunderstood yeah that's what I'm saying, like, it just doesn't make sense. Is this a royal you or are you, the engineer? I was reading it as like you like Shopify's got to keep doing this because the market's growing there. No, no, no, no. You're saying, you're saying, yo person, you got to be 40% better.

Yes, because the rest of it is all about the individual using it, getting X percent better, all this kind of stuff. But TJ, I know you had some contrarian viewpoints on this, but I will say that one thing that I just really dislike about all of this kind of conversation is not that saying that AI will make you faster. For definitely a set of people in every single profession, AI is going to be a huge assist to their current skill set. I don't think that's, I don't think anyone can argue that.

I think the big problem is that we sell it as 100x. Because I just do not believe that three and a half days is worth a year of work ever for anything. Like, right, like you could generate shaders for a year and a half or for three and a half days. T.J. could. And I would outshader his shaders if you gave me a full year, 40 hours a week, every single week. T.J. is raising his hand. TJ, question. Thank you. Thanks for calling on me. Have you worked with some of the people at these large companies?

Yes. In fact, I have. I worked with Trash Dev. As you can see right below me. Am I below you? Well, yes, in this one, no, I'm below you. Okay, you get it. You're below me. Oh, he's on the lower floor because you're in the C suite. Let's talk about that taint again. Yeah, that paint. Okay. I think 100x. I'm with you. 100x seems wild. But I will say I've worked with people that haven't gotten anything done in a year, too. So it all depends on, like, what your baseline is, right?

Like, my baseline is not like your baseline. Like, they need to, like, have some metrics. Is it going to be lines of code, featureships? You know what I mean? Like, you can't, I don't know. I have a lot of thoughts here. So I'm just going to keep going. Yeah.

So one of the, one of the bullet points they have. And honestly, I thought I read the whole thing. But then when you started reading it prime, I completely missed everything you said when I was reading it. I guess I don't know what to read. So you had exactly. you in fact did not read it. Would you like me to put it back up for you? I in fact did not read it when he started.

I was like, oh, we had said that. So the one of the bullet points was before asking for headcount, you have to prove that you need it with like your current use of AI. Yep. Which kind of blows my mind. So like if we go back to the water tower, right? You know, I basically came in there and like bombed your project in like 30 minutes, right? You did.

So the way I kind of see it is like an AI is kind of more or less an intern junior dev where they're effectively a net negative until they onboard, get familiar with the codebase, you know, level up a little bit, right? So you spend most of your time babysitting, code reviewing that AI slash intern. So you end up losing productivity on your side to make up, to have the AI or intern do its work.

So that is like kind of counterintuitive. That would be the argument, I feel like, to, you know, you would need headcount because you need someone to basically be a full-time prompt engineer. and basically code review the AI stuff right before it just keeps shipping more stuff. Does that make sense? Yeah, but I mean there's a couple flaws to all this. Like, first off, on the TJ side of things, the people that don't ship in a year,

do you want them shipping to begin with? Right? Like, do you want them 100x more effective? Like, no, you don't. And so second, even with this intern thing, people always equated to an intern, right? AI is just like an intern. The thing about an intern is that day one, you're like, here's what you need to do, and you sink a bunch of hours into it.

And then day two, day three, day four, by, you know, by month, you're like, okay, hey, go do this project. And then they come back in two weeks and they have a pretty good set of project. And you're like, okay, here's a couple different things you need to do. There is no gains with an AI. The AI is as good as it is day one as it will be on day 60 right now, unless if there's some theoretical huge breakthrough that we have yet to see on the horizon, that's going to make it from a junior engineer into some magnificent senior engineer.

But right now, barring extreme breakthrough, you get to. the same thing. Whereas with interns, you get some sort of multiplier effect or at least some sort of additive effect to the work where they get to run individually because they have autonomy. Potentially. I think to be fair to the guy who wrote this is Toby, right? Yeah, Toby. Okay, so to be fair to Toby, if he's following his own guidelines, he probably, he might not even proofread this particular memo, right? He might just ask the AI for something that encourages the company to use AI and then it's send. Because he was, he was 100x. He didn't have time to read any of it. So it's possible that Toby doesn't know what it says. And frankly, that would be likely

because this is also the guy who sent out the premature optimization is the root of all evil. We should ignore a tweet. So like, you know, just if you could go back like last year and he was talking about how everyone needs to start thinking seriously about performance. And like, we need to use a lot of AI and we need to think seriously about performance all the time. They don't even, it's like, what are you even talking about? These are completely conflicting statements you're giving to your, you know, your engineers here. So I don't know. Like, it's, it almost reads like a thing where it's just like, yeah, I asked AI for a thing

that would encourage AI use. I hit send and then, you know, it leaked. Oops. Or something. But do you think there's a world where this actually is, I mean, do you think in our world this is actually a fine thing to mandate from the top down? Not if you trust your employees at all. because at least for me, if I was running a company of that size, I would say, the kind of memo I would send is like, look, AI is a new thing.

We would like people to evaluate it. If you find it useful, we'll pay for it. So go ahead and adopt AI practices if you think they're helping you. If you don't think they're helping you, you don't have to. I trust my engineers to make intelligent decisions or designers would be the other people presumably who are using things for prototyping. I trust you to figure out whether or not this is going to help you or hurt you and to. act accordingly. Saying like you have to use it everywhere just sounds like you don't really believe you have any good people. Because if you did, you wouldn't be saying this because you'd trust them

to make that decision. So you would mandate the evaluation of AI. You wouldn't mandate the usage. Exactly. You'd just say, look, you know, I would like our company to look at this. Please everyone take, you know, basically it's more of giving a permission, right? Saying look, everyone, it takes some time to evaluate AI. I would like you all to try it. see how it goes, maybe write it up a little bit so we know for each of our things, like, how did it go, like who had success, who had failures, and then I'll look at it and we'll maybe make some decisions about what we want to do going forward. That seems like a same thing. Just like, ah, AI is big and 20% everyone use it. Like it's just, I don't know, it just kind of comes

off as like almost like frantic or like, you know, I had too much coffee and or I don't know. There is another interesting bullet point where it says AI usage will be a part of your performance review. Exactly. How do you, let's just say I ship all the time even without the help of AI. Well, I get a bad performance score because I didn't make an AI commit. Or like, how do you even prove that I didn't use AI or use AI? Obviously.

If you would read the other bullet point is that you're going to share your prompts in the Slack room. And so it's probably prompts shared is really your impact to the AI usage among the company. My prompts would be so bad just like yours prime. Like, can you just make this work? You guys are not even chaos or math. Like it's pretty embarrassing. I've seen both of your prompt. That should actually, I'm going to start doing that.

That part's pretty embarrassing. Dude, I would love to see a room full of like shared prompts. And it just gets more and more degenerate. Like as you read, you're like, oh gosh, what, what? What with their mom? Right? It just gets worse and worse. Hey, if anyone's out there listening right now, all of my prompts, you have to pay me to train off of them.

So if you're in the audience listening right now, you better close your eyes and never hear me do a prompt again. because those are mine for a thousand dollars contract of adhesion yeah got them got them i've got some bad news for you guys on that because uh the copyright office recently released their guidance that says that anything that comes out of an i where all you did was put in a prompt is not copyrightable so everyone can just take that for free you can't even know anything about law it but this is the prompt he's crafting okay this is not the output they don't need it they can just take what it generated oh i see but to get the to get the peak performance you got to be using chaos Forbes and I'm copywriting that strategy.

So that's just too bad, Casey. Fair enough. I will say, okay, in, I will say that I have seen people use different LLM tools quite a bit, let's say, different levels of efficacy, right? And knowing when to use them and how to use them is not always intuitive. And also, like, sometimes it breaks your expectations and they're better at some stuff than you think they were going to be or not at other other things. So I do think there's something of saying like what you're saying, Casey, of like having

people try it. But even like saying an extended, like, you guys really need to like try it a lot of times to see because it is quite a bit different and like reaching for it in unexpected places. Like one of the ones that I used, I told Prime this one, like I was doing some awesome WM. Like not it's awesome, but like awesome is the name of it. And they have all of their docs and everything. online, but it wasn't like in a format that I could pull into my editor to get auto complete and everything. So I just like listed five URLs and told the LM to generate from

those like properly formatted stuff and put it in. And it would have taken me like hours and hours probably to do it. Right. And it did it within two minutes. Right. And then it like literally 10x how fast I could do the rest of the things. Because like now my editor works and I get auto complete and I can discover everything and I'm getting squiggles and all this other stuff right and I feel like if you're not trying LLMs out for different things you would never think to like paste a URL and just be like fix this right so there is something there where like I don't know exactly how that's going to play out or where that's going to end but there are things like okay you do actually need to try and use it if you want to get stuff out of it I don't think at least right now

you can just be like, I use it once a week and it really affects my like workflow. That just doesn't, it doesn't, it doesn't work for anything. You know what I mean? Nothing you just use occasionally is really going to change, you know, I don't know, except for ayahuasca, apparently. That's what I heard. Yeah, that's a one-time use change right there. Yeah, that's a vibe, TJ.

Okay. Yeah, exactly. That's what I heard. Yeah, I generally, I generally agree with that. There are a few things that when you use it, you're like, man, that is actually extremely convenient because I would have spent like two. hours trying to come up with a VIM macro to solve that problem and then attempt a rejects to edit HTML and like go down all the same routes everybody else has ever gone down in the

history of mankind just to try to fix that exact problem there are things that are fantastic and so you don't you definitely don't know that if you haven't a i'd hard enough so that's my only defense of like sort of the mandate aspect now would i put it in your performance review No, that ain't for me. Like, I, I don't want to, I barely want to measure anything in performance review. I would do performance reviews mostly off vibes. Like, I just hire managers who you can trust and hire managers of those managers who you can trust, however many levels you need.

I don't know how big Shopify is. And then you just like fire people if people get the bad vibes. You know, I don't like you can measure if they're doing a good job or not. You're a trillionaire. Okay, everyone desperately wants to know how to measure if you're making good software or not. and people aren't. So there you go. Maybe we use the AIs to evaluate whether or not you have been using AI

and then we can give you a completely objective in the sense that it's done by a machine performance review. There we go. We could also take a-I'm going to poison the data. Yeah, we could also take an AI and throw all your Slack messages into it with all of your coworkers and then be like, give me his vibe score, one out of 10. He's, viving with his teammates. People that aren't, get them out. Is there going to be a thing where, like, the trust and safety division of companies,

which is, like, you know, during the day, they're, like, trying to go through all of that company's public comment boards and pull out all the, like, things that are, you know, horribly demented that people are posting? And all of those are trying to just get sent to the AI team, where now those will be tried as prompts to see if they encourage the AI to do better work, right? So it's just like it's this pipeline directly from we had to stop 13 year olds from seeing this to this is what we feed directly to the AI because it works really well when, you know, but thank 33's comments that we had to remove from the public forum are all are getting it set up for the day. Dang. Well, I am just curious how this is going to go in a year because I'm just very, I think,

think the performance reviews, because out of everything, it's just like mandating people use AI in some sort of capacity at your job, it's like, okay, sure, maybe you could do that. But mandating it as a measurement of performance, I don't know how that's ever going to be acceptable or any measurable or anything. It just seems like you should always measure people how you've measured them. And if they're getting better relatively, then it's like, okay, this person's better. The end. It doesn't make it. And if the AI was doing its job, it will show up in that metric, right? It's like if the person is using AI effectively, they should be. be having good output. So you wouldn't really need to also check if they used it, right? Correct. They were doing 100 X? You'd know. You would know. Like, it will show up somehow. Your features, things should just get done really, really quickly because someone has

100 years worth of human work in one year of time at your job. There's a lot of that thing, too, where people keep saying these ridiculous multipliers, but their output doesn't appear to change externally. So you're just like, I don't see like Shopify producing 100x more stuff. So what happened? You know, like what's going on exactly? So it's kind of weird. Like shouldn't we see it?

100x would be like you've rewritten all the software in the world this year, but just your company. That would be 100x, right? Yeah, you didn't do anything like that. But you didn't do anything like that. dependencies, right? Like you just rewrote the stack. I wish the like, for in my experience,

it's been much more like, instead of saying like I'm, I'm 10x better at whatever. Like, AI has let me do things that I would normally not do at all. Like, because it would just take too long to like learn some base knowledge about this thing. So I can say like, hey, do these.

Oh, that's not really right. Kind of go here. Okay, now like summarize this. Well, right. And like,

I could start something and like complete it that I would not have started otherwise. But it's not like the AI like wrote all of it or anything. You know what I mean? It's that like it enabled me to do that, which is there's no comparison. It's not like a. 100x more or less.

It's just like, oh yeah, I could do this project that I couldn't have done before. In a sense, that's a million X. But in a sense, it's plus one. Like, okay, so I feel like that would be such a better framing for a lot of these, like, CEOs. Like, you can take on a more ambitious project. You can try something in a new area. We can try and, like, fix something that we didn't know how to fix before because you can, like, ask the LLM how to solve DNS problems that you couldn't figure out on your own before

that were 37 replies deep in a forum. Sweet. Now you can solve that. But like it doesn't make me a thousand or a hundred X better. I can't write all of my software for the year and a week. Like that's not happening. And it's something in which, yeah, because when we did the towers of Mordoria stuff, there were a few things in which I could have spent a day or two just reading the love docs and just purely experimenting on how to draw, how to like do the things right in that.

Or we could just generate a whole bunch of stuff and then try to fix it as we go, which in some sense, it just means you. understand the basics of it. I understand how to use it. I don't understand how it works, which I think is a fine thing when you're attempting to learn something for the first time. Or you want to iterate on an idea and you want to see different versions of it. Like it, like it's important to be able, it's a valuable thing to be able to try something out, throw it away, try something new, throw it away, try something new, throw it away. Oh, that one was good. Oh, that could have taken me three months to figure out before, but this way

it took me three days. Okay, cool. Then maybe I delete all of the code and now I right from scratch by hand like a like a ludite or something but uh that iteration speed's still like important and useful especially like you're not going to guess product market fit right on the first one for anything basically right like if you are then you you are also if you already a trillionaire right if you can be the guy that always guesses everything right for pmf congrats go work at one of these countries and or companies and deploy billions of dollars of capital and like succeed where others have failed. That's not me. I can't. I have to try it to find out if it's good. Correct. Hey, speaking of that, we're going to do, because we're way over time, but we're going to

just do one quick thing. There's this video going around right down the internet, oh, sorry, which is Quake. And Quake was entirely generated by an AI. This entire gameplay that you're seeing right now is generated by an AI. It was 100% trained on only Quake, and now it made Quake, and you can run around and shoot bad guys, sort of. If you turn your back on them, they've never they never existed to begin with. Sometimes they die standing up because that's just the way it goes. But, you know, it's just, it is all generated. So, Casey, and speaking kind of of of this fast iteration cycle,

hey, we're going to be able to move really, really quickly. Is there something to this kind of technology? Let's just say it can improve by 10x or 100x or some of these big magical numbers. Do you think there's actually a real value in the gaming market to be able to see or experience features not yet created as fast as a prompt, just to see if they're good enough? Or is being able to just simply see something play out actually real to you being able to say that's a good mechanic?

So, you know, I'm really the wrong person to ask. That's a very designer-specific question. And the best I could say is in that form, like the form that they're doing there, probably not. Because when I've seen great designers work, and there are very few of them, but I've had the privilege of seeing, like, a couple of people who I would consider to be exceptionally good designers do what they do. There's a lot to the specifics.

You know, like, a great example would be go listen to. So I mentioned Dragon Sweeper one time on Prime Stream. There's a game called Dragon Sweeper made by Daniel Ben McGui. He's also the person who made Storyteller, which was a big hit. One of Netflix's top games on their service, actually, trash. and he's an excellent designer and you can go listen to there's a podcast of him where he describes

how he designed the game and he designed it very quickly it didn't take him very long all things considered but the kinds of things he had to work out to make the game fun and to experiment with are things like exactly what is the algorithm that determines how these pieces are placed

on the board when the board is generated or things like that and it's just like So a thing that improves that iteration time would be useful. This thing that Microsoft is showing doesn't seem like it gets at that. That seems like a very superficial thing that they've created that wouldn't really help any real designer because we all kind of already know what that thing. Like if you're just training a thing on Quake, it's like what are the design changes I'm going to make that this thing can represent properly that actually give me that information? They're the kinds of things that we would call a producer knob in music,

a thing where you just kind of like pretend it does something so that higher-ups can feel like they did something. That's what this looks like. It's like something that some higher-up could say like, make the shield more green. And the AI doesn't like, oh, that was great. And everyone's like, thanks, Bob, you really turn the project around. And then they go back to doing the real work, right? So I do think there's room for something that helps game designers iterate. this doesn't look like it to me,

but if you really wanted to know, you'd have to ask some of those top-notch game designs. Ask Jonathan Blow, asked Daniel Ben-Mir-Gui, ask some of the people who are really excellent at doing this kind of stuff, and see what they say, because I don't really know. Okay, yeah, because I think one of the things that's probably missing from this context

is that they did this other kind of release of it, which is copilot for your games, or I forget what they call it, it could have been co-pilot. We should talk about that, that's a completely different thing. Okay, well, they would just, it's the exact, well, in my head it's the same thing, meaning that it takes your game,

records you record thousands upon thousands of hours on you know this game and then you create a model off of it that can generate your game and then you're supposed to add in these features by kind of prompt maxing it into being like okay i want my game but i want like surfboards and then it makes surfboards and go oh that wasn't fun looking so then you add in new stuff i think that's the entire goal of this project because i believe this is along the exact same veins which is how close can they accurately portray your game make it playable and then add new features that don't require any programming, just a simple prompt. But the copilot for games one was the thing that's like clipy for games. Like it comes up and you ask it like, where's the wood on this map? And they're like, oh, you have to go towards the forest to the northeast. Like, you know what I mean? That's what copilot for gaming it. Co-Pilot has a lot of meetings these days.

There was some Microsoft Game Studio AI that is doing what I'm describing, which is thousands of hours, recording of a game and then attempting to replay it and show you. Like, this is your game on features. Yes, that was a different thing, I think, not co-pilot for gaming, but I could be wrong about that. Maybe you're right. Maybe they're both called co-pilot for some reason. I mean, everything's called co-pilot from Microsoft. It's a good point. Maybe I've got the name wrong. Maybe it had a better name. I don't know. We probably have three separate AI game related things called Microsoft Copilot for Games. You're probably right. Or a co-pilot for Microsoft games and games for

co-pilot Microsoft. And so they're all different products. Yes, by Xbox. By Xbox. For Windows. You need the game pass for that one. As many are as you can in there. Yeah. Yeah. So anyway, that's all I could really say about this is I don't, I don't, the specifics are what matter to real game designers. And so this is just not specifics. These are more cosmetic changes that they're showing here. Things that the game designer would not really, like, they're thinking about exactly what will happen if I change a little bit the way we respond to like this mouse down.

Or what, like, that's what they're working at to feel, to get the feel for it. Everyone knows what it looks like if you walk around in 3D. It's not hard. like that's not a prototyping problem we've ever had, right? Not for a very long time anyway. So it just, this doesn't seem to be at the level of experimentation that they're working at typically in my experience. But again, they'd have to say, not me.

I'm the wrong one to answer. Okay, okay. So maybe in 10 years, this thing could be outstanding, but today from your brief survey, it does not look like it necessarily fulfills the need. It might even be outstanding today if they were training it more on the kinds of things that game designers actually work on.

Right. Like the nitty gritty, right? Like I said, this is very producery, right? It's very much like, should the Valkyrie jump a little higher because it looks cool or something, right? Like, it's not asking the question of like, what are the gameplay ramifications of the Valkyrie? It can't tell you, oh, the Valkyrie will win 20% more matches. Right? Yeah. And that is the thing that they could be doing. So, you know, maybe they'll get there soon. Because that's the kind of thing, you know, just we analyze billions of matches. And then we can tell you like, oh, if you, if you change. the way that this thing worked where you have to go to the black king bar costs 20% more at the shop it has this effect on games that's something that game designers could use for like balancing multiplayer let's say right this doesn't look like that to me maybe it does that and they just didn't say so but it doesn't look like it can do those sorts of things i could be wrong okay was that a dota two reference it was oh sick all right nice that's tj's love language dude we can go play that at burn convention that'll be sick at burn convention we'll all gang roche that's the only term i know sweet okay cool yeah that's all i needed all right we'll make no further jokes about the burn convention because they're

all inappropriate from here on react miami the whole like burning man thing kept coming up react miami is the burn convention of of developer conferences gosh i don't even know what you're talking about you know what okay never mind yeah don't it was funny no year ago stop talking about burn convention All right, hey, what's your sign off? Who? Anybody, you gotta start going. My name's Casey. Trash.

And the name is the prime. Take us away, prime. Why would you just interrupt me like that? It was coming out. Dude, that was so good. It was like bam, bam, bam, bam, and I was coming in, and it ruins everything. If you're here watching the video now, Flip took it out. Flip took it out.

It did not exist. Join us next week for another predictable disaster here on the stand-up. Next week, we have a great topic which you have suggested. the docs, Casey. Yes, we use docs, by the way. We're the stand-up. I actually wanted to, I think that's just hilarious because I think I have like an internal itch that one day, if I ever was forced to work for an evil company, I would do the exact

same thing. And so it's just like, I have always wanted to, yeah. Anyways, what? Yeah. We'll scratch that it. We'll scratch that it next Wednesday. The name is the foreshadow agenda.