CoRecursive: Coding Stories - The Bitter Lesson: The history of reinforcement learning

Episode Date: June 13, 2026

I've been trying to understand how machine learning actually works. Not use it, understand it, down to the ifs and loops. How does a program built out of plain conditionals get better on its own? So late one night I sent Don a paper. Three words in the title: reward is enough. The claim is that all of intelligence, the whole thing, comes down to a system maximizing a reward. Don thought that was far too reductive. I wanted to pull it apart and see if it held up. We backed up through the history to find out how far "reward is enough" really goes: B.F. Skinner training pigeons, a backgammon program that taught itself, the Go move no human would have played. It's a story about machine learning, and what that leaves for the rest of us who still do it by hand. Episode Page Support The Show Subscribe To The Podcast Join The Newsletter  

Hi, I'm Adam Gordon Bell, and this is Co-Recursive, and today I have here again, Don. Hi, I'm Don McKay, and I'm here again. So I've been texting you again. Yes, yeah. And it's always late, too. It's never like, you know, it's 10 o'clock at night or something. I know. Sorry. I mean, when you're like an old man like me, I'm like, that's late. It's too much. Yeah, you're like, I eat at four. I go to bed at eight. Um, yeah. Yeah. So I've been, I've been trying to understand, I guess, like, machine learning. And that's why I sent you some thing. So, uh, yeah, what did I send you?

First you told me you didn't have whiskey anymore. And I was like, that's a crime. And then you said, okay, I've been continuing going deep on AI and ML stuff. I'm trying to understand it at a level of like, could I write a program that's just normal ifs and loops and whatever, but yet it can learn. And I think I found the key. It's this paper. Reward is enough. Silver Singh, pre-cup, Sutton, 2021.

I want to explain it to you. It's sort of simple and simple my knee quotes. And, yeah, because like, okay, machine learning is fancy and cool. And, like, we're software developers. We should be able to understand it. But it feels like magical. Yeah, it's like existential, right? You're like, oh, I don't know.

I don't know how it works. So, yeah, I was like, oh, okay. Here's the important line. Intelligence and its associated abilities can be understood as subserving the maximization of reward. What do you think? Subserving is such a weird word. Yeah, you haven't heard of subservient individuals.

Well, I guess, okay. Maybe in that context. It seems like he's saying that as long as you have a reward structure, you can create intelligence, which I think is a little bit reductive, in my opinion. I mean, it's incredibly reductive, right? He's saying it all reduces... It all reduces down to the maximization of reward, right? And furthermore, saying intelligence and its associated abilities can be understood by subserving

reward, which means that everything about our intelligence is just serving some kind of reward structure. And I think that that's kind of an... I don't know. Yeah, it's very reductive. Once you reduce something to that level, you lose the fidelity of the statement, right? So it reduces everything, but has a cool title, right? Because that sentence is very complex. But the title of the paper, right?

Reward is enough. You kind of get what he's saying, right? If you wanted to create an artificial intelligence, reward is enough. That's the only structure that you require. Everything. Yeah. That's all you need is just that one thing, right? Like I've reduced the entirety of intelligence, of intellect, of culture, of...

I mean, I guess he didn't say culture, but everything to do with what makes something intelligent and smart and able to take action in the world is just reward, right? Press the lever, get a piece of cheat, basically. Okay, so, yeah, Sutton, he wrote this book on reinforcement learning, and, like, him and his advisor, like, kind of invented this field. Yeah, so I just want to pull it apart, right? That's what I want to talk about today. I was like, this is a great thing for our kind of stack trace format, which I asserted

before and maybe still needs a better name that, like, we take something and kind of, like, back up the stack frames until, like, you get, right? We have this really grand statement. Like, can we pull it apart? Can we get to somewhere? No point or exception. Yeah. Turns out it was all intelligence is a null pointer exception.

Gone the bonkers. Okay. So it's 1988 and Richard Sutton is finishing his PhD at the University of Massachusetts. And his advisor is named Andrew Bartow. And Bartow, he actually comes from B.F. Skinner was his background as a academic. Do you know B.F. Skinner? I do not. So interesting enough, right?

B.F. Skinner, he taught pigeons things, where they would, like, press a lever, and they would get, like, a little piece of thing. Okay. He's the person who, like, operationalized how to, like, train animals to do things based on reward. And, like, his theory was also that all of human behavior is based on, you know, going after rewards,

like, the same way your dog wants a treat. It's kind of a weird background. Like Sutton is, like, a computer scientist, but he's working under this guy whose background is like this behaviorist and who like taught pigeons basically and side note

Skinner is super interesting probably not a topic for the podcast but Skinner in the in World War II he built things for the US military any guesses what they would be? homing pigeons yeah so he built

a missile guidance system where they just put the pigeons yeah with they put the pigeons in the missile I did actually see see like a video about that yeah insane right Like he had very reductive view, as you would say. Like, he reduced everything to like, well, yeah, let's just, we need to hit this target.

Why don't we just train a pigeon? Yeah, they were good at it. And like he also had, or like one of his students built, I wrote this down, this guy who worked for him named Tom Verhe had a pigeon product or project at the company. Is it? E. Lilly? Is that how you say it? The drug company, E. Lilly. I think it's, I think it's Eli.

He had this project at this company where basically they just had pigeons do QA. So like all the parts are going down the aisle and they just have pigeons there who like are trained to look for the broken ones and pack them out of there. So they tried to basically, they replaced human workers with pigeon and like they got the whole thing work and then they canceled it because they thought it would be like a PR disaster. Although I did hear a story before that they canceled it because it was very demoralizing. Like imagine you're at your job and they like they like, They fire the guy, like further down the chain from you, like further down the... And just replace him with the... Replace him of the pigeon.

Like, he... So it didn't take off. But the same idea, right? You just give a simple reward. The pigeon learns like, oh, this is a defective part or this is a defective pill and knocks it off. Anyway, this is behaviorism. Anyways, that's where the Sparta guy came from. And Sutton's working under him, but the whole computer science thing at this time in 1998

was very different, right? It was AI, but a different approach to AI. Expert systems. So expert systems, they sometimes call it good old fashion AI. Expert systems was this idea of AI was like different than today. It was if you could interview a doctor and figure out how you assess a patient and you can just like write down all those rules and then ask people about their health. And then it's like a doctor in a box because it's like a just like a flow chart that you can

travel through. I mean, it seems kind of basic right now. But it had this idea, you know, if we can get all the rules for all of intelligence written down, right? Here's all the doctor rules. Here's all whatever. We can replace. We can have intelligence just in sort of if statement, whatever. So Sutton's not that, right? He works for this behaviorist, different approach. And he publishes a paper called Learning to predict by the methods of temporal differences. And it's a super, it's a super cool idea I'm going to show you. Because this is how I started to understand machine learning, right?

X's and O's. So this is, yeah, this is Tick-Tac-Toe, right? And this is how he explains the thing, right? So if we, if we play Tick-Tac-Tow, you go first, so you're your X, you can pick which position you want. But this is all the possible states of the entirety of the game in this tree. It's just a giant chart with a bunch of X's and O grids and what the current state of them would be, very similar to like a chess moves, kind of like mapping. It's kind of like a tree, right, or a flow chart. So it's like, his idea is, okay, there's Tic-Tacto. He figured it out there's like, I think it's like 725 possible state.

that you can be in, right? And then, like, at the very end, you have a result, right? So, like, if we played this game where you get three Xs across and I do nothing to stop you, because I don't understand the game, then he gives that a value. So, yeah, he's assigning values to win conditions so that, you know, that's what you want to get. That's the ideal state. It's like, there's no magic. It's like an array of seven hundred twenty-five values. It's a game. And, like, you put an X, and I put an O. And so it's a tree going down. And at the end, I win, right? So here's this trick. This is, this is like the whole of the thing. And then all he does is this, right?

So this one must be negative one because Adam went. And then so this one, he goes backwards. And he's like, okay, this one is like negative point one, right? And then this one is negative 0.01 and so on. Going backwards and saying like, well, those must have all been pretty good because you want. They're not as good as the part that you want, but they're kind of good. So it's basically like walking backwards. It's like how people develop superstitions in a way, right?

So you trace back the path to how you got there and you assign values according to how much you want to incentivize that path based on the end result or desentivize the path based on a losing condition. So at the end of the day, when every possible move has a value, you'll know how to respond according to always going towards the one that will give you more of a score. Exactly. This is this whole paper. I think you just nailed it, right? Yeah. So whatever choice you make, it will then evaluate all of the other choices that it's learned. and then it will have a score and it will go with the one with the highest score. So if I go play, so that was play one game, right?

If we go play 10 games, like every time it goes through the tree, it decides if it wins. So it's like, hey, play a whole bunch of tic-tac toe games and see where you want and where you lost. You don't even have to know how the game works. You don't even need to know what's good and what's bad. You just need to be able to play it many times so you can start building it like a model of scores. Yeah. And if you do like enough times, you start to know what the values are. like what's a good move and what's a bad move

this is his whole thing right this is like his early paper and I love it because it just explains this simple version of machine learning right it's learning to play the game just by doing this working backwards stuff and like there's no crazy stuff right it's just a bunch of array values and you're like updating it's like if

if at the end of the day you get told you did good or bad then you just like figure out during the day what were the things and work backwards yeah that's amazing and like that's his that's part of his like reward is enough right like all you need to do is assign a score to desirable or undesirable outcomes So what happens next? He moves up to chess. Not yet, but if we're going to get there,

four years later, 1992, somebody picks up his paper, and it's this guy at IBM Watson Lab, and he does backgammon, which was like kind of an interesting, backgammon people are super into backgammon, but like,

I don't think it has the allure of like chess, right? It doesn't have the pedigree of chess. You know, backgammon involves gambling often, too, like you're betting on it. Like, I don't know. Like old British men play it, I guess. I don't know who.

I know my mom plays. backgammon all the time. But it is like a complicated game, right? And it has some interesting... So the guy who decides to work on it, his name is Gerald Hassaro, and he's at IBM Watson. And he just takes that rule, right? Which Sutton calls the temporal difference update.

But there's a problem with backgammon versus Tick Tick Toc. Maybe a problem versus every single game in the world versus Tick Tick Toctoe. Do you know what it might be? They're just bigger. Like Tick Ticktoktoe is so small. Oh, you mean like the possible outcomes? The possible oakums is just like astronomically larger, right? So like Tic-Tacto, it's great for his paper because there's like less than 800 states.

But yeah, do you want to guess on the backgammon number? I have no idea. So this says it's a hundred quintillion possible states, which are, so that would be possible game states, right? It's if you played all possible games. If you needed to make this array to hold all the values. Of every possible position in every possible game. Yeah.

So it would be, no, that's exactly what it is. it would be as many grains of sand on the beach of Earth. You can't even enumerate it, right? You can't make an array or hash table. You'll need a lot of memory. Yeah, it's just not possible. And then, like, to play that game forward and backward

and update those all, like, it would take too long. Right? So he does something different. He just, he uses a neural network. So you can imagine that we have, like, in our tic-tac-toe, we have, like, a function, say, that takes in the, like, nine values,

like, your state of, like, X's and O's, and then it returns that number. He does that, but he doesn't actually, you know, come up with that much memory. He just puts this neural network in the middle that somehow returns a value, right? And that way he can shrink the problem. So the network that he makes is tiny.

It's three layers in a neural network. A neural network has, like, kind of floats in it, and then some conditions that make it either return, you know, a high value or a low value. I don't super know how it works, but he's using way less values than there are possible states. Right? And the idea there is it's kind of like compression, right? If you need to store all the values of the game state, but instead of having just a bazillion values, you say like, oh, we only have this many, then it has to figure out like what the duplicate parts are. It has to like pattern match. It's kind of like a JPEG compression, right?

So all these games are similar. We can update this little spot at once. It starts to see winning patterns from losing ones. It just doesn't have that much memory, much like we don't. So it has to find ways like, oh, this is similar to that other time. Maybe this is like that. But he does the same thing, right? Have this thing play backgammon against itself. And then when it gets a good result or a bad result, you know, update the value in the neural network.

So what happens? Like, how can it something actually get better when it's just playing against itself? You know what I mean? Like, it doesn't know backgammon. Like, how's it going to get good at it if it's... Well, you get to know all the moves, right? If you haven't played it before and you play against yourself, but you know the rules. Like, you have to at least know what the rules are in the game.

Then when you play against yourself, you start to recognize what to do in response to certain moves. So this is a side project that he plays and he has this thing. Like it's 1992. Computers are still pretty slow, but he's got off in the corner playing backgammon against itself. At this time, like, IBM Watson is this prestige lab of the AI era, like for this expert systems of like writing down all the important rules. Like, this is the preeminent place in the world. But this guy's like doing his other thing, right? he's taking the Sutton guy's idea, having it run.

Well, they're all busy off working on more important expert projects. I think you can guess what happens next. Like, after they let it play with it, like play against itself for a number of, I guess, days, months? Like, how long did they spend? Hundreds of thousands of games against itself. Hundreds of thousands of games. Imagine they would have moved on to something else.

Well, I mean, maybe they did. But, I mean, he's a researcher, so he wanted to write a paper about it and whatever. But anyway, so he brings in, he brings in backgammon players. He brings in expert. and they play it and they lose. So this thing has gotten better at Backgammon than the real world players. And Backgamon does have chance in it, so it's not totally definitive, given enough time. Because, I mean, this is that guy's point.

Reward is enough. Like, you get to the end and, oh, I lost this round against myself. Well, what did the, you know, what was the difference? And how can I, like, it's the same. It's just a bigger scale of that Tick-Tac-toe thing, right? It also just did interesting things. Much like Chester's this whole theory behind Back-Amin, and like when you should do this and when you should do that and what you should do in these situations.

And it does different things. It hasn't read those books. Like it's the computer program. It only knows the data that it's acquired by playing itself. Yeah. And so it does things playing them and they're like, and then it beats them. And they're like, oh, wait. So it beats them because it's unconventional.

Rather than trying to imitate humans and develops its own sense of positional judgment by learning from experience in playing against itself. And that's why it was so surprising to these backgammon folk, right? They're like, wow, this thing plays unusually, and it's beating us. Right. So they brought in this guy, Bill Roberti, two-time world champion, top three player alive, who had written books on backgammon. And he's like, he says, like, yeah, this thing is better than we are.

And we were wrong about things because it beat me in these ways. And so he's written all these books about how to play backgammon. And so he starts updating his book, right? He's like playing against this thing, learning new moves. She's like, oh yeah, we got to change that because it just beat me this way. So in its unique experience of like not being exposed, it's almost like, as you said, not being exposed to humans has actually benefited it, right? Yeah.

It's learned extra thing. It didn't learn the game framed in any kind of way. It just kind of learned it through its own experience. Yeah. So that was cool. They beat backgammon. And then, like, you would think the next thing that happens is this takes the world by storm, but didn't.

They just forgot about this idea. They just forgot. Yeah. guess for how long? I'm guessing they forgot because like nobody plays back in and so it didn't grab headlines to like your layperson but for how long I'm

I mean if it's a successful project and I wouldn't imagine it would stay dormant for that long. Yeah so 20 years. Okay, that was way out. I mean they had a much bigger project they were working on which was deep blue was like beating the chess expert and using these expert systems and like the researchers who were building this amazing chess thing

They were like, I don't know, that's just backgam and it's weird. Like, of course it works with that method. Like, because it's, I don't know. Who cares? Like, yeah, it's like saying some, it's like a result they didn't care about, right? They're like, we're not interested in this way of solving things and we're not interested in this game. So they, uh, they just dropped it for 20 years. Yeah.

Like those researchers went on to other projects. Yeah. I mean, I'm sure some people kicked this idea around, but this wasn't like the hotness, right? It's like there's probably somebody out there still really in the like, uh, macromedia cold fusion. But, like, that's not, that's not the thing, right? Like, maybe it had some cool ideas, but like, who knows? And so deep minds, they published this paper in nature. Yeah, this is from their paper.

Perhaps the best known success story of reinforcement learning is TD Gammon, a backgammon playing program which learned entirely by reinforcement learning and self-play and achieved a superhuman level of play. However, early attempts to follow up on TD Gammon were less successful. So it's not that they were successful and then they got shut down. It's that they were less successful when they tried to expand beyond backgammon and they're like this is a dead end. I guess so and everybody gave up on it.

So Sutton himself in 2017 gave a talk where he was kind of saying like WTF. Yeah, temporal difference learning, temporal difference learning, temporal difference learning. It's a method for learning to predict. It's basically the center, the core of many methods you know about. Q learning Sarsa, TD Lambo, deep Q. Natchezer, T.D. Gammon, the world champion backgammon player using deep reinforcement learning from 25 years ago. Oh my God, 25 years ago. deep reinforcement learning 1992.

I don't think you gave it very good emphasis, but like, he's pissed off, right? It's like, he's the guy. You get where he's coming from, right? He's like, I had this thing. Yeah, I mean, if he's, if he's probably pissed because he's like, I did this stuff like 25 years ago and nobody gave a crap.

I mean, how often does that happen, like on a very much smaller scale to a lot of programmers in the field, right? Like, listen, our database, like, if we follow this trend, it's going to get too big. There'll be no instant size that can hold it. And like, it'll destroy our business. And I've worked on a solution.

It's just like, if you give me three days, like, it'll save us in 18 months. And they're like, three days, though. No, you could be working on this other thing for three days. Yeah, forget about that. And then 18 months later, they're like, so we got this issue. Can you like whip that up like right now? And that's when this guy's like temporal difference learning. Like, I've been telling you.

I've been telling you. That's with 25. Yeah. So then because what's happening at the same time, right, is this deep blue thing. Deep blue. Do you remember the Deep Blue story? I do, yeah, I remember. What happened with Deep Blue?

Give me the summary. That was the one that played chess, right? Yeah, and it started winning against Grandmasters. There's this big project to beat the world champion at chess as like publicity for IBM. And it was like part like, oh, they had these very big supercomputers. But a big part of it was this expert systems, right? Do you play chess? My dad always wanted me to play chess and he like taught me, but I would just lose to him.

And then I think once I left, like after university, I learned. And then occasionally, like, when I'd go home and visit him, I would play chess against him. At one time I beat him, but I think he made it let me win. But that's like my ex. Like, he, like, from my perspective, like, chess has a lot of ranges. From my perspective, he was just way better than me. But chess has so many levels, right, that I'm sure, like, there's tons of people, like a grandmaster, like, nobody can beat them.

But when I learned chess, there was this idea of you, certain pieces of certain values, right? So it's like you're sitting there and you're playing chess and it's like my dad's going to do this move where my pawn takes his pawn and then I could take it back with my knight, but then his other bond could take my knight, right? And you know like, oh, the knight's more valuable. It has a value of three and like the pawn only has a value of one. So like I shouldn't do that. Because then like you're basically like adding up the value of each side. And so like a big part of playing chess is like this idea. It's like you have these pieces and you know what's valuable and it's like, okay, if I make this move, then what will he do and what are the value? So the

The interesting thing is like the, like they've looked into this, right? Grandmasters actually don't play that many moves ahead. So they might play like three or four, right? But the thing is that the moves that they consider are always just the best move, just interesting. Yeah, like they have names for all those like, different opens. Yeah, and they have the open books, which is like, yeah, all these people who've played forward, like the first, whatever, like four or five moves on a chess board. Like they've all been exhaustively done and they know like which one is better and which one isn't. And so Deep Blue used the strategy, right?

So it had this giant open book of all, you know, the good ways to move forward. It had this giant book of end games. Like if it's just down to these couple pieces, how exactly do you win it, right? And then instead of what I was saying of like, I just add up the pieces and determine the value. I mean, I guess not instead of. Like it had something like that. But like IBM was throwing all this money out. They brought in all these grandmasters and they came up with like,

the most accurate way that they could come up with to, like, assess. At this point in this game, right, what value, like, how good am I doing, right? And so what I described is like, whatever, three or four rules for adding a point. Like, guess what theirs was? 20. It was 8,000. Oh, I was not. Like, 8,000 rules.

I don't know why you ask these, like, arbitrary questions, like, different answers. Well, no, because it's good, because you said 20, and I think the thing about 8,000 is it's absurd, right? but this was this expert system idea, right? It's 8,000 moves in advance? Well, so the 8,000 isn't how many moves in advance. That's just how they figure out for any given game position, how valuable it is, right? It's like you get plus one if you have this piece there and like just 8,000 different rules to say, like the equivalent of our number on tic-tac-toe that's like, this is a 0.7. Scoring system.

It's the scoring system for evaluating any specific chess move, right? And then, like, as you said, right? Like, you know, we play this and we think through, like, okay, if I move here, you move there, right? And then, or that state will use all those rules to figure out if this is advantage. But then the other thing, yes, was like, deep blue was just like, I can look through a bazillion of these, right? So I have my 8,000 rules and then I'll just play it out really quick, right? This was in 1997, Peters weren't as fat, but it would play 200 million positions per second. And like it was this big boon for IBM, but also like for AI.

They're like, oh, we hit the pinnacle, right? But at the meantime, like Sutton, the Tick-Tac-Tow guy, right? So he's working on a text. Yeah, he just keeps saying that whenever you talk to him. He's like temporal, what is it? Temporal. It was temporal difference learning. He's just screaming about that in the background.

So he writes a textbook, right? Him and Bartow, who's like the behaviorist with the chickens or the pigeons. Pigeons. So reinforcement learning and introduction comes at 1998. eight and now we would say it's a very influential book people use it and it's important and whatever but then it was like whatever like IBM is the thing and deep blue and nobody care right everybody's working on something else right it's still like he's like the cold fusion guy like doing his macromedia thing everybody's like dude what what are you doing um but there's there's somebody paying attention right

the thing that ibn really figured out was this searching forward right it's like we're moving really fast through this game and we can figure out, you know, chase down the tree of possible moves and find the best one. Um, where the Sutton guy was like, you know, just play it to the end and then we'll figure out who won and it worked backward. But so somebody, uh, looks into this in 2015. Dennis Habiz and he, he founded the company DeepMine. So this is years later in 2024.

Um, he said this. We bet on generality and learning. So those were always at the core of any techniques. we would use, that's why we triangulated on reinforcement learning and search and deep learning as three types of algorithms that would scale, be very general and not require a lot of handcrafted

human priors. I mean, he's an expert in the field and I'm some guy. But yeah, so deep mind is this company and they're going to turn out, like, they're like, we think we can reinvigorate this, right? Like, we think this is important. And so they come up with these deep queue networks. Deep Q network is just the same as this Sutton rule.

It's this idea of like working backwards. and applying these values, right? And so it's the same ingredients. There is Dutton's update rule, and then there's a neural network, and then there's just a programming method, right, to play games and put together data so that it can learn.

Network with like three layers, very small. But in the meantime, right, while all this stuff was happening, we talked about it in the LLM one, actually. Like these dudes, like Hinton and stuff at U of T, they figured out how to make really big neural networks and learn all this stuff. And so they decide to do Atari.

Basically, we want to build an AI that can play Atari game. So they have 49 Atari games. But they have this problem, right, where you think of, like, if we do TikTok toe, it's like our state, you know, into our neural network or our array is easy, right? It's like we have nine fields, like either filled out or not, right? But how do you send in, like, where we are in a video game, like, into something, right? Like, it's hard to make a function call that's like the current state of the, of my video game.

Yeah, because there's too many factors. Yeah, but just do the simplest thing. Well, I mean, like, you could do something like X, Y, Z for, like, coordinates. Yeah, so they did, they took whatever the resolution of an Atari screen is, and they blocked all the pixels. So not, they wouldn't send in every pixel value, but they would send in, like, say, like a five-by-five block.

Every five-by-five block. And they put it to gray scale. So they took out the colors because, like, that doesn't actually matter usually for... Okay, so they, like, reduce the resolution and then send it. the pixels in. It's like asking it to learn. It would just like, but it can't see the game. It can just see like, you know, 0-0-1-1.

So the software they built, they made it play like a whole bunch of Atari games. And so it gets, its reward thing, right, is just the score. And what its input is, is like all of the pixels. And then Atari has a joystick. So then it's just like, it can decide. It gets in all the pixels. And then based on his joystick, it can just like decide, right? I'm going to push this way or push that way.

So this is their, this is like, this was pretty famous, I think, when it came out. So this is one of their videos. It's blockbreaker, like a, like a very early Atari version of blockbreaker. So you've got your, you've got your paddle at the bottom, you've got, you've got your, your ball in the middle, and you've got a rose of blocks at the top, each of them in different color. And so this is after it's played a hundred times, right? So it's played a hundred times and learned some stuff. And it's just trying to. Yeah. So the, the goal of it is always just to prevent the ball from slipping past you. So you've just got intercept it. Yeah, but it has to learn that because all it really knows is the score. It got something. It's basically randomly moving the stick around. Okay, here's 200.

It's more like I would play. Could occasionally, it like randomly moves the stick. Okay, now we're at 400. 400. It's getting a little bit more accurate. It's catching it. It's getting it every time. It's moving it to intercept. Yeah. Okay, so it feels like it knows what it's doing, like it's learned. And okay, then the, what is the overlay here says, at this point, the agent finds and exploits the best strategy of tunneling and then hitting the ball behind the wall. It's learned something after 600. It's learned a strategy of trying to get in behind all the blocks because then. So it must be, so yeah, because it's counting the score, right? So it's like if I can do this, then the score will go up more without my intervention. Yeah. So it's figured out this idea of like drilling a hole

through all the blocks and then it doesn't have to do anything, right? The ball just bounces around back there and clears thing. That was like a huge deal. Like there was a paper in nature and And it wasn't just that game, right? After that, DeepM gets acquired by Google, which is nice because Google, much like IBM, has just, like, a bazillion computers. The thing about this type of training is you need something that can, like, play the games over and over. Right.

You need the hardware. You need the hardware. And so that is when they decide to tackle. Right? Because I feel like if they were, like, cool, we have this method. Everybody said it was no good, but we made it work. And, like, we're going to tackle chess. Nobody would care.

Like chess has already been vanquished with this other tech. Nobody wants to hear about that anymore. That's old news. Yeah, we've moved on from that. So chess has already won. So they decided to do Go. And the thing about Go is it's incredibly hard. So at that point, there was no good AIs at playing Go or very few.

The reason that goes interesting is because of like this kind of exponential explosion. When you play chess, like there's maximum like 35 moves you can make each game. And as we said, with Go, there's much more. 150 possible moves at any point, which means if you're trying to like play forward and build of that tree, it's just astronomical. It's a very large board. You have a lot of pieces that you can place in lots of places. And then it takes a long time to play and it's not till the end that it kind of all resolves. And it's like, did you win or lose? Human players probably haven't even moved those moves, right? Because there's so many. But humans are good at pattern recognition and

somehow, you know, they learn to play this game and there are champions and they're good at it. But like, it doesn't seem to fall to our normal techniques of like, oh, let's map this all out. These reinforcement learning guys came in who had been listening to Sutton with like his temporal different. Yeah. What is it again? Temporal difference learning? Temporal difference learning.

And they listened to him, right? And so they built their machine. But there was this guy, Remy Kulam, he was an AI expert and he had built the best go program in the world. I think I have a quote from him. So the quote is, I think maybe 10 years, but I do not like to make predictions. So he said it would be 10 years. But then 22 months later, they had beat this, right?

AlphaGo beats the best go players. That's awesome. But like, what is it? Right? Because I'm a software developer. Yeah, you explain like the problems with, like, we can't play the game to the end. Like, we don't know what a good state is. So like, how does it make the decision then on what?

How did it quantify vibe? Exactly. This is the question, right? And so I built a version of wordal. You know wordal? Yes. I think everybody knows. was word. Yeah, I never really played it, but it's my example. I built like alpha wordel,

basically. So if you haven't played the game, it's a, uh, is a five letter word that has been chosen that you don't know what it is and you're trying to guess it. And you can put in a word and it will evaluate your guess and tell you each letter, whether that letter is in the word and in the right spot. It's in the word, but in the wrong spot, or it's not in the word at all. And given that feedback, you have to then pick a new word. And you only get a finite amount of guesses. I think in this one there's like six. And then after six guesses, if you haven't figured out the word,

then you lose the game. Here's a wordal game. We're looking at, let's guess. So I don't know, what should we get? So I'm going to pick the word slate. When you play wordle,

you can do the same thing as in chess. You can say, like, okay, if I play this move, there's not an opponent, but I can get back a score, right? And so like if I get back and none of them match, then what I've eliminated,

not very many positions, but some. Because, I mean, there's a finite amount of decision. It's quantifiable now. Yeah. So you can kind of play it forward in the same way you go with chess, right? So we're looking at a wordal game, right? And when I pick, I'm going to pick Slate as my guess.

And it was completely wrong. So it scored them. And none of those letters are in the word. Yeah. And so there's only a finite number of wordal words, like in their set. I guess there's 2,315. So, like, I played Slate and nothing matched. But actually, that eliminates a lot.

So this is saying, in my little app I made here, before this, there was 2,315 possible words. Yeah, because you eliminated A and E, right? Which is contained in a lot of English words. Yeah. And then I can play this forward, right? So say to go back to go, right, at any given state in the game, I can tell how good I'm doing just based on how many possible words are left.

And so that's like the one component that's hard and go, right? The other component is the playing forward. So now that we have the slate position, I can pick a new word. like let's say if I pick this birch but before I select it right so for any given word I can do the same as in chess right I can search forward so I know like if I played the word birch oh I see then you just be like oh if I get this result then yeah that's because the um the result here has a finite amount of possibilities yeah okay so this is kind of this is my dumb down version of how the alpha go works right

so it has this neural network right and in the neural network the the weight in the neural network are basically from playing the game through all the way to the end, the same as Tick-Tac-Tow, and it figured out what won, and so it updated thing, right? And so for the game, for its playing, it thinks this word arose is like its best guess. Probably it played that word at some point in one. And so it's like, oh, this is awesome, right? So this is its best plays. And then it has this value, which it calls position strength.

So for Wordl, it's saying, my position strength is 3.644, which is, basically it thinks that it will win in about three yeah about three guesses or four three or four so this is uh these are like the two important parts of the machine learning right it's like it needs to know how good it's doing this is kind of like the chess points right and then this is kind of like what move to make and so it thinks that we should do a rose how do we do a row but then the other important thing is we can we can play it forward in our head so before we submit our moves we can kind of, okay, if I pick that, what are the possible options, right? So it thought this arose was best based on its original playing.

But then when it plays forward from that position, it decides like, no, irate is better. Starting word. Yeah. Like, I'm going to pick irate. So basically, AlphaGo, they've combined these two ideas, right? So deep blue was this one idea, sort of let's play forward into the future and figure out the values, right? Like, anytime it's our position, we can kind of spread out and try to go. quickly through all these rules and figure out what the best thing is.

With this, which is like, then once we get to the end of the game, you know, we'll learn from that and we'll update these, which is kind of like Sutton's like Tick-Tac-Tow. Does that make sense? It's like they came up with this idea that we'll have a neural network that tells us how good each position is, but how will it know that? Because we said, oh, and go, it's super hard to know actually if this is a good position or not. And so they said, like, well, we'll just, we'll do the Sutton trick.

we'll play a game ago and when we get to the end we'll say like hey if we won all the places that that happened are good so is this what sets them off and he's like several different learning yeah several difference learning but like that would take forever right like the to do that because there's just so many go games so you have to constrain how many games you play maybe well so you combine it with this other idea which is the search right so when you're playing your game forward right so i played 10 games right right Right. And I have, so I think like, these 10 random moves I did are good, right? And then I'm going to play my 11th game. And before deciding what move to do, I'll play forward. Yeah, you play it, quote, in your head. But we're doing this two-stage learning, right? It's like when we get all the way to the end, we say all these things must be good. And then when we're playing in the future, we don't wait until the end to decide if it's a good move. We play it forward. And if it looks like any of the other ones that we did before, then that also must be good. Like if a piece of it matches one of the winning scenarios, you're like, oh, this is a good one.

This must be good, right? We're learning faster. It's like we don't have to play all the way to the end of the go. So it's this double layer thing. It took me a long time to understand this. Yeah. I mean, obviously, it hasn't learned everything, right? Obviously.

It's playing like things that are not correct. But it can play this situation forward, right? And you can keep playing it, right? It can keep learning. And it will get better. And in fact, like, you can do this whole thing. Yeah, you're like increasing the resolution of, the prediction.

Wordle's not that hard. You can actually just figure out the perfect wordal move by walking through every game. It's somewhat like tic-tac-toe and that it has a bottom. So now it's running a whole bunch of games and it's learning more.

He won. Yeah, you got it in the fourth guest. Not bad. Not bad Alpha Wurdle. This is their idea, right? And it took me well to understand this. It's like these two things. Like one is playing against yourself

or in the case of Wurdo there's not another competitor. It's just like they're just playing. But it's still the same idea, right? It's like thinking through the moves and updating it, and then this idea of searching forward. So they use, it's called Monte Carlo tree search. But basically they're randomly playing forward in this tree of words.

And which moves they decide to play forward in are basically the ones they think are like, oh, this seems like a good move. Let's play it forward. He'll moves and see if it is. It's like limited foresight. With some randomness, because it's like you don't just want to play the moves

you think are good forward and see if they are. because like you start off knowing very little and you're probably wrong. So you have to kind of like... You have to fail a lot. Yeah. So they do this and they train and then, yeah, they're going to play this big game against this Lee Sedol, who's the champion. Yeah.

So they win the first game against him. At this time, it's inconceivable that they win at go, right? Like, obviously we know that they want, but I was part of this thing called the Human Judgment Project at the time. And it's like they tried to train expert bet on outcomes. Yeah, I remember you talking about that. It was because we, um, we both read that book. book about super predicting. Yeah, super predicting, right? So I tried to join that guy's project and like you

could bet on these things. And this was one I bet like I was like there's no way. Like they've determined it will be like a million years before they're good at this. So I bet like no, Google will lose. And then they won the first game. Um, like they crushed them. And then during the second game, I don't know what's happening. Like they're playing at the moves and I don't super understand things well. Um, but AlphaGo is playing white and Lysidol is playing black. And it's pretty early in the game. The position's pretty open. And the, and the, commentators, because it's like a live broadcast thing, they say that Lysidol is doing quite well, maybe even a bit ahead. But it's still anyone's game, and it's only game two, so they don't really know.

So then AlphaGo plays a shoulder hit on the fifth line, which I guess is like, oh my God, a shoulder hit. On the fifth line. No. Yeah. And so people don't really know what to do. Yeah. So it doesn't mean anything to me.

Like, I don't know what a shoulder hit on the first line. But Go is like a 19 by 19 board, and people put their stones down around. Yeah. And the basic rule you learn. right like it's just like a basic rule is in the opening in the middle you play on the third and fourth lines you don't play in the fifth line and so alpha go played on the fifth line which is away from the center of the action and i guess that's for the end game yeah i mean we'd have to look up go terminology but like if if if if it was doing that there would have to be a um there would be

a factor there that would throw off the human competitor because they've never played against somebody who did that the same as the backgam and when they yeah it's like what's going on what are you doing. So Lee Siddol, who's like, he's the best at this. And so he only plays other, like, amazing, the best people. Yeah, what do you think he does? I mean, I think that he would probably, like, he doesn't have a strategy for that because he hasn't played against anybody who's made those moves. So it would, he would have to train. There's no counter for it. It's funny because we know nothing about go. So it's like, but I don't understand the rules, but I assume they each have so much time to make a move. And what Lee Stoll does is he, he just gets up and leave. He just left.

Yeah. He didn't finish the game. He just walked out. Like, it's a lot. It's a televised thing and he's like, like, why did he get up? I mean, I'm sure he was just shocked. You didn't know what to do. Like, you know, wouldn't you just keep playing? Like, if I was playing chess against somebody and, you know, they made a weird, unconventional move, I just like, well, I mean, I guess. And then I would just be playing with my own strategy. Yeah, and it's a live broadcast thing. And he left for 15 minutes. He just walks off. And it's not like they have AlphaGo. Why don't you entertain us? Like, no, that's a computer. So the commentators on

the live broadcast, they don't really know what to say. Like, they're basically like, I think this is a mistake. I think AlphaGo is off its rails. What should we do? And so an important thing, right, is AlphaGo has these two systems, right? It has the one that tells it the move it should make, right? And then the other one that we saw that kind of like plays forward what's going to happen. Right.

And so this weird move that it made, the thing that said like what you should do next basically said you should never do that. Right. but then they played forward the rules and all of a sudden it looked really good and so it was a rule, it was like it knew the same as we did or at least it all knew like this is not a conventional move but then it played forward and it was like, hmm, interesting.

This is a good strategy. Okay, and here is the architect of AlphaGo who is actually David Silver. He was running the team at Deep Mind. Here's what he said was happening at that moment. The professional commentators almost unanimously said that not a single human player would have chosen move 37. And then we found out that AlphaGo said that there was a one in 10,000 probability that a human player would have played that move.

So it went beyond its human guide. Then Lisa Dole comes back and he sits down. I guess he isn't gone. And they start playing. And like, yeah, the Go games are long. Like, this was considered early games, but it's move 37, right? But over the next 50 moves, everybody starts seeing what AlphaGo is doing. Right?

Like, this move starts to make sense. and it starts to be like a dominant factor and they're like, oh, I get it now, right? It starts crystallizing in everybody's mind who understands go, like... So is that like a move now? Like now people like play to the outside? So he loses the game, right?

And yes, I mean, I don't know for this specific game, but I know that much like backgammon, the rules shifted, people start being able to play against these machines that have new techniques and then they learn from them. Right? It's like we were based on our own strategies that we learn. over time when we had these biases, but the computer's out playing random games and discovers new

things that we didn't know, right? AlphaGo wins the match 4-1. So at least it all won a one of them, right? Four-one's pretty decisive, though. Yeah. And that was March 2016. But people aren't willing to give it up, right? And so this fact comes out that, okay, AlphaGo did actually have some human input into it. So there is this online thing called the Casido Go server. And, and It's been around since the 90s and people play Go on it. Yeah. How many was it? Millions of games of humans playing that had trained off.

So people were like, well, you said the machine has beat us, but maybe it's just remembering, you know, user 5, 6, 3, 9 played Move 37 on a Tuesday in 1993. And it's just remembering it, right? You see the same things with LMs, right? They came up with a strategy to overcome this, which was, let's do this again. Let's build this again. but we won't include the humans playing. Throw it out, right? We'll start again.

So it's like the same machine as Whirdle, right? It's like if in my Wirtle, I originally played forward and I played a whole bunch of games, and then I start training it. They're like, oh, maybe it learned from me. So then they do this again. They call it Alpha Go Zero. Which is a good name, like, zero because it's like zero human knowledge.

So now they play their Alpha Zero against the one that beat him, and it crushes it. So they've built an even better one. And so this, the whole thing, playing all these games forward, It only took three days because they're Google and they can just run so much in parallel and play all these millions of games at once. Which is, it's kind of crushing, right? Because you could dedicate your life to being good at Go. And we're like, well, we took this thing. Didn't even know what Go was.

Is that pigeon? Three days later. Yeah, no, it's just like we're giving it and it's like don in a box. So it's like running at the super fast speed in parallel. And it learns. Here is how DeepMind announced it in the nature cover in October 2017. A longstanding goal of artificial intelligence is an algorithm that learns. Tabula Rasa, a superhuman proficiency in challenging domains. Starting Tabula Rasa, the new program AlphaGo Zero achieved superhuman performance winning 100 to 0 against the previously published champion defeating AlphaGo. Tabula Rasa was also a video game that Richard Garriott tried to, the guy who made Ultimal Online, and it didn't really get off the ground and then got scrapped. Do you what it means? Blaine slate, I think.

Yeah, yeah. Blank slate. Blaine slate. But yeah, it's like, because it knew nothing, right? It's like, it started from absolutely nothing. Okay, so they removed the human games and it worked. But then they're like, hey, we should remove more thing, right? Like, what more can we remove?

What do you think? What more can they remove? So they did the Atari move. They basically removed the rules of go. It doesn't know how to play the game. Much like my wordal guy. Yeah, it doesn't know that you can't play the same letter that you've already discounted. So they're like, what if it just all it can figure out?

like it just gets whether it won or lost. A score, yeah. And then it's like figures it out the same way the Atari thing did. Yeah, so it just needs a win condition. Yeah. And then once they built that version, they had it play a whole bunch of games. So they train it on go. They train it on Shogi, which is some sort of Chinese chess like game.

They train it on chess. And so then they have it, like once they built their chess one, they have a play against stockfish. Stockfish is like the most powerful chess thing at that point. that, like, Graham Master's learning it, and it's got all this custom stuff, and then, like, yeah, it crushes buckfish. And, like, they're making a point, right? They're, like, they're, like, doing the Sutton thing.

I mean, I think they're also drumming up publicity, but they're like, this thing knows nothing. Like, all it knows is whether it won or lost the game. And you can build the most complicated thing in the world, and you give me some time on Google's compute things and a couple days, like, we'll crush you. They're making a very clear point that they can just learn it all, right? Yeah, so that one was called me.

mu zero, like MU zero, because it doesn't know any rules. Like they've removed even more. But the meantime, since all this has happened, Sutton comes out and he writes kind of a manifesto. I feel like it's always good when you can say, I'm writing a manifesto. Sometimes are good, sometimes are bad. You could go unabomber.

Yeah, it's risky. So he writes the short essay March 2019 that he posts on his personal website, and it's three pages long. And he calls it the bitter lesson. And so for Deep Blue, right, the problem was that they had to have that evaluation function, right? They needed to be able to say, like, how good is this chess position? You will hear people talk about the people who are big onto AI right now and who are building LLMs and all that. You'll hear them talk about the bitter lesson all the time.

And they'll say things like, are we sufficiently bitter lesson pill? Basically, like, you know, in the matrix where it's like, take the two pills? it's like, have we sufficiently taken the bitter lesson till, right? Like the bitter lesson that this guy wrote has become like the important rule that all these people believe in. Right. And so here's how his three pages that becomes this big thing on his website opens. The biggest lesson that we can read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective and by a large margin. General methods, right, that leverage computation, that's,

his whole point. I mean, it's a larger essay, but he's saying, none of your clever methods, right? We don't need 8,000 rules for how to evaluate a chess thing. We just do this tic-tac-toe thing. Just let it go. Like, it has a reward, right? Like, let it go, right? But he means, like, large. When he says, like, a large margin, he means, like, extremely large. Like, whatever you got, however clever you can be, like, building a complicated system, it will be beat if we can just figure out a way to get a reward signal and just turn the crank. It's basically saying, stop being clever. Yeah, because like computation has reached such a level that if you just can provide it with a reward condition,

then it can play out massive amounts of scenarios and create its own kind of guide, right? So you don't need to actually provide it with any rules. Your rules might actually be constraints. Move 37 or whatever it was or the backgammon thing that people are like, that's not the way you do it. But they were right because like Deep Blue had all these rules, but then, you know, stockfish had less and was stronger than Alpha Zero had like no rules and like crushed it because it's just it can learn all the rules itself, right? It's just a simple algorithms,

but they're these learning like meta algorithms. They win, right? And so he was just, he was salty about this. He, he calls them sore losers and he wins a Turing award. And then he starts using this method everywhere, right? He uses it for, uh, to get good at speech recognition. So there was like 30 years of building amazing speech recognition systems, you know, learning phonetic rules. And then people just come up with this deep learning method. Just like, hey, we learned how speech works by just throwing lots of compute at it. And then right in the middle, in a paragraph about go, he says this thing that kind of summarizes this whole episode, right? This thing that I'm trying to get at.

So read the next quote. When a simpler search-based approach with special hardware and software proved vastly more effective, these human knowledge-based chess researchers were not good losers. They said that brute force search may have won this time, but it was not a general strategy, and anyway, it was not how people played chess. So learning by self-play and learning in general is like search in that it enables massive computation to be brought to bear. Search and learning are two of the most important classes of techniques for utilizing

massive amounts of computation in AI research. Yeah, because he's saying, like, a program can make its own data. Like, we talked about this when we did the LLM episode, like, oh, they're like, we're running out of data. But like here, it makes his own data. You just like let it run as long as it can figure out whether it's one or law. But you still have to, yeah, you still have to provide it with good and what's bad, right? Like what's a positive and what's a negative condition so that it knows how to evaluate. And that idea, right, that came from Sutton. Like they were taking his tic-tac-toe idea. I mean, you have to squint and say like, well, maybe math. Like they're like, oh, math could be like a

tic-tac-toe thing, right? But just randomly guess and we'll learn backward. But this is his idea, right? He's finally, he's in, his brilliance has been recognized. There's like 20 years of objections that this, of his idea of self-play, right? But the bitter lesson, the reason he called it the bitter lesson, actually you have the quote. Tell me why it's called the bitter lesson. We have to learn the bitter lesson that building and how we think we think does not work in the long run.

Which is super confusing, building and how we think we think. But what he's saying is like, don't try to like reflect. Don't ascribe your human conceptual models onto the AI because it doesn't. think the way that a human brain does. Or it doesn't need to, right? You don't need to record the way the best chess player plays chess. You can just let it figure it out on its own, right? Like, don't give it how we think we think.

Just let it run. And it's bitter because, because, well, you want to, you know, you're the greatest AI person or software developer and you build this complicated system and all these rules and it's getting better and better. And he's saying, like, no, actually, don't be clever. Just throw compute and a reward at it. And it will be guaranteed better than whatever you can come up with.

It's a bitter pill to swallow it to say, like, whatever cleverness you can come up with a human will not be as good as us just figuring out where the reward signal is and giving it unlimited compute, right? Like, it's kind of saying like, let the machine win. Like, the machine is better at this than you will ever be. Stop trying to teach it things. Just let it learn. So then October 2021, an artificial intelligence journal, he publishes this thing that I texted you, right? Reward is enough. And so the people on this, one is Sutton.

Right? David Silver is the other person on it. He was the guy, the lead architect of AlphaGo, and Alpha Zero and Alpha Mu Zero, I think. But yeah, the last guy is Sutton. And it's the same Sutton that's all the way through. And in Reward is enough, they make this claim like even larger. Do you want to read that? Yeah. Reward is enough to drive behavior that exhibits abilities studied in natural and artificial intelligence, including knowledge, learning, perceptions, social intelligence, language, generalization, and imitation. Now he's, now he's, now he's definitively saying, like, reward is not just enough for beating somebody at chess. Reward is enough for knowledge, for learning, for perception, for social intelligence,

for language, for imitation. He's basically saying, that's it. Like, all of, um, all of intelligence is contained in beating something or reward and seeing what it learned. Yeah. I mean, again, it's still reductive because we do undertake some endeavors, not for rewards. Like, I don't think there's a tangible reward to figuring out what dark energy is, but it's something we haven't discovered. How would you ascribe a wind condition to that? And Skinner, so Skinner was the, Skinner was a psychologist and

his methods got left behind a lot, although they were powerful because, you know, you think of Freud and like wondering why people do things. Like, Skinner never wondered about that. He was like, no, you press the button, you get a piece of grain, right? It's enough for the pigeon, yeah. But he had nothing to say about individuals and how they worked because people to him

were like a black box, right? You give a reward and then the pigeon does the thing. And people aren't pigeons, right? We're a little bit more complex. But the interesting thing is these AIs are black boxes as well, right? Like, we don't know what's going inside the neural net. Nobody care. So it's very interesting how it actually follows from this guy. He's like, I don't care what the pigeon thinks about it. It's like I trained it to do the bomb guiding. It's like, I don't care what the neural network does. Now it's just better at chess than you will. And I don't know, it's many years later. And he won the Turing Award. And he

he wrote this, and it's affected the whole world. But he went on this podcast about AI recently, 2025. He went on the Dwarkesh Patel podcast. Yeah, I know. Yeah. And so he interviewed him because he's like, oh my God, look at AI so big and you finally got your comeuppance. You know, what do you think? Like we've all finally learned your bitter lesson and we're building these things. Will you read it? Large language models are about mimicking people doing what people say you should do. They're not about figuring out what to do. They have the ability to predict what a person would say. they don't have the ability to predict what will happen. So they thought they learned all his bitter lesson,

but he's saying like, no man, you built an LLM by consuming all of human knowledge. That's not Alpha Go zero, right? You actually just built something that's pretending to be human. Like his way would be like... It starts with nothing. By his thinking then, we would be at the big blue era of AI. It's really good at doing all these things, but it's encoded all this human stuff.

And the next step should be like, can we get rid of all the human stuff? human stuff. Because right now the LLM isn't going to do the Move 37, right? It's trained on human knowledge. Yeah. It's also trained on like what it thinks you want. Like I think that he's on the right for like a like a general artificial intelligence because he's talking more about like don't teach it the content. Teach it how to discover the content.

To learn. Teach you how to learn. This guy who worked with him and who worked on the AlphaGo team, Julian Schnitzer, he's at Anthropic, working on LLMs, building large language models. And he said, after that statement, he said, like, oh, no, you're actually wrong, Sutton. Because as we discussed the other time, these LLMs are, they have these RL loops in them now, where they're like teaching them reward based on math and based on whatever. So he said, no, actually, Sutton, you're wrong. These LLMs are doing exactly what you're saying.

I guess they're built on human knowledge, but we're still, we're giving them this a reward loop and they're learning. Okay, but now I'm going to go dark. this whole thing, right? Because, like, I feel like this goes further than all of that, right? Because he's saying, you know, reward is enough. Well, enough for what? Saying, well, it's enough for everything. Like, I think that there's room for nuance there. I mean, most of the things we do for reward, some things we don't have a discernible reward, right? Like, we eat to stay alive. We work to make money so we can be alive, right? But there's, like, pursuits that don't have any kind of discernible reward or reason why someone would do them.

It's true. But I feel like what he's saying is, like, if you can come up. up with the scoreboard, which is basically what the reward is, right? It's like you need a way to score something. Then the computers will beat you, right? And so it was easy to come up with a score for chess, and now the computer's B you at Chet. It used to be this image net. It was in the last episode, it was like identifying animals and images and computers couldn't do it. But then they built this big data set and they were like, it has to guess and say, like, is this a cat? Is this a dog? Once they had that scoreboard, computers just learned and got better at it until they're way better at identifying things in images than humans are.

But now there's like Swaybench, right? It's like a benchmark for programming tab. It's like you pull down this task and it's like an open source bug and you have to solve it and then it scores you on whether you're good or bad. The problem is that's a benchmark that now a computer can do its reward game on. And so like the larger bitter lesson is like we're in trouble. If there's a reward signal like the Sway bench, like a computer would be able to dominate us at it. the dark version of reward is enough, is that reward is enough for a computer to crush you

at what you do. And that's the bitter pill. So for 70 years, the job was to be clever, right? Like, I'm the best bummer at solving a problem or I'm the best software developer. But I feel like when I try to extend his analogy, it's like Sutton's whole life says that these things are actually clever without us. Now that we've built them, like the advantage of being clever is kind of escaping us. the what's valuable isn't what you know anymore. Like it feels like the world's changing. I don't know. It feels bleak, right?

I mean, it could. It could be. I mean, rewards are also dynamic. They change a lot. Sometimes, you know, like what is today's reward might be different tomorrow by being able to redefine it. I think the thing is like when we, we work on something, we define the rewards. It comes from within our own kind of minds.

It has yet to determine its own. rewards. So that was the show. I know. I exhausted Don. So he walked in not knowing. Actually, so I walked in not knowing what subserving meant. Like I had just never heard that term. Oh. Yeah. Subservient I had heard. But I think we figured it all out. Whether we agree with or not, I think is a little bit undecided. But yeah, thank you, Don, for letting me go through this all with you. Here's what I would say about Sutton. I think that he's right. If you can come up with these rewards in this box system, the computer will win. But as humans, we just come up with new boxes.

And so this guy who did the backgammon, right, after the computer beat him at backgammon, like he put out a new book, right? Or you invent a new game. And the humans, you know, the way it is now, like we're setting the limits of what these boxes are, right? And so I think, like, you just have to keep learning and growing. If you try to compete head on head at a limited game

with the AIs that we have right now, you will be beat. Yeah. No, worry about it when it can create its own box. Yeah, yeah. You just make a new box, right? I'm not going to try to rate SQL, the most performance sequel anymore. Like, I'll just ask Cloud Code. It'll be better than me.

But I'm building the things. Yeah. Yeah. So until next time, yeah, stay curious, learn new things. Thank you so much for listening.