Algorithms + Data Structures = Programs - Episode 300: Sean Parent "I'm not writing code anymore."

Episode Date: August 21, 2026

In this episode, Conor and Bryce celebrate the 300th episode fo ADSP by catching up with Sean Parent!Link to Episode 300 on WebsiteDiscuss this episode, leave a comment, or ask a question (on GitHub)SocialsADSP: The Podcast: TwitterConor Hoekstra: LinkTree / BioBryce Adelstein Lelbach: Twitter | BlueSkyAbout the Guest:Sean Parent is a principal scientist and software architect for Adobe Photoshop. Sean has been at Adobe since 1993 when he joined as a senior engineer working on Photoshop and later managed Adobe’s Software Technology Lab. In 2009 Sean spent a year at Google working on Chrome OS before returning to Adobe. From 1988 through 1993 Sean worked at Apple, where he was part of the system software team that developed the technologies allowing Apple’s successful transition to PowerPC.Show NotesDate Recorded: 2026-08-19Date Released: 2026-08-21POD GODCppCastcpp.chatSean Parent - Papers and PresentationsADSP Episode 299: Do We Need Humans in the Loop?NVIDIA CCCLcel-rsStepanov PapersBob TarjanTarjan's strongly connected components algorithmSoylent Green is people!Claude CodeThe Last Emperor - A GentlemanJonathon Blow on the Quality of Software (Software is in Decline)Lean Programming LanguageDafny Programming LanguageCreator of TypeScript: 10x Faster Typescript, Why AI Won't Replace SWEs | Anders HejlsbergautoresearchIntro Song InfoMiss You by Sarah Jansen https://soundcloud.com/sarahjansenmusicCreative Commons — Attribution 3.0 Unported — CC BY 3.0Free Download / Stream: http://bit.ly/l-miss-youMusic promoted by Audio Library https://youtu.be/iYYxnasvfx8

Now I'm back and finding that I'm not writing code anymore, which is an existential crisis for me. For me, this, you know, hit pretty hard because Dave Abraham's, who was a member of Software Technology Lab on my team, he really had a pretty strong reaction, anti-AI reaction. And I was like, you know, this is going to destroy the industry. I have done very little with the book because who's going to read it? Right, right. What's the point? AI is going to be a bigger change.

And I think just saying, you know, I don't want it, I don't like it, is not a productive stance. And start to think about what is the role of people in society where we are no longer the most intelligent thing on the planet. You know, right now I think our politicians and leaders are woefully ill-equipped. I don't think they have any idea of what's coming. I think we just have a small glimpse of what's coming. Welcome to ADSP, the podcast, episode 300, recorded on August 19th, 20206. My name is Connor, and today with my co-host, Bryce, we celebrate our 300th episode by catching up with Sean Parent since the last time he was on ADSP at the tail end of 2025. We chat about AI, how Sean isn't writing code anymore, how it's affected his workflow, its implications for society, and more.

Wow. All right. We should. Will that make the final cut? Maybe, maybe not. So I guess, yes, we should launch. Should I actually share? I wasn't sure if I was, I launched it in case.

Maybe this is a good way to start. I see, Sean, you've got your, you've got your CPP concert. I was just thinking earlier about what I'm going to do with all of my conference shirts. And I have this, my dad used to be a windsurfer. And he would go to all these windsurfing regattas. And my mom had this quilt made. from all of his t-shirts that I had as a baby. And so I was thinking about something like that.

But I have so many conference t-shirts, it would be like 12 quilts. But then I'm thinking I have this big white ceiling above me. If I could somehow attach the quilt to the ceiling, that would be super cool. Or make like wallpaper out of all my conference shirts. It would probably be pretty easy just get some wood frames and then stretch them out and staple them to the wood frames. and then just tick those a lot easier than having a quilt made. Yes. Knowing Bryce,

he is going to contract someone to get that quilt made, though. A lot of the same, and it would also, you know, cut down on your echo. Yeah. Yeah, that's true. He's got a pretty good mic, though. But what I was about to say before that was here, I'm going to share an entire screen.

I've been teasing this for months now, but seeing as it is a special episode 300, we've got back the most recurring guest of all time, probably the most asked for guests of all time, Sean Parent. And speaking of Sean Parent, hopefully I'm sharing the right screen. This is still yet to be released,

but this is an emulator version. I mean, I've been using it since February, however many months that is. But every single time I have a new idea, you know, I just keep on adding it. And this is a podcast player, podgod, soon to be released on Android, and then after that iPhone.

But the beautiful thing is here. Sean and Bryce are looking at this. We've got ADSP, of course. But then we've got one of the many personalities that you can go and subscribe to. What is a personality? It's like a meta podcast that curates all the interviews across all podcasts. So if you click on Sean, obviously we have on almost Monopoly on interviewing Sean. And I guess episode 300 is going to be here when we release this on Friday.

We're recording this on Wednesday. How does this work? So is this? Wait, wait, wait, before we have questions, if you scroll all the way back, The first, I'm not sure. So I actually, when I was communicating with the AI at one point, I was like, you know, and if we're lucky, well, actually, I'll discover some interviews with Sean Parent that I don't know of, but the only ones that came up with were the ADSP ones and then two CPP cast ones and then a CPP chat,

which is a now retired or deceased podcast. And it's pretty fantastic folks. So I would say it might be out by the time. You're listening to this, but that's not true. I checked. And it typically takes one to three. sometimes upwards of a week for Google to approve apps. And to be honest, there's a couple things I'm still trying to add.

Anyway, so if you are a listener, and this is going to kind of undermine the point of trying to grow our listenership, if you actually are only here for these Sean Parent interviews, you will be able to go and unsubscribe from ADSP and just subscribe to the Sean Parent personality. And anyway, so. So I have a question. How does this personality feature work? Is it AI powered or do you somehow index guests? Is there some way to look at like all podcasts and see who appeared on a particular podcast?

Is there like metadata for who's in the episode? Yeah, we can reserve this for another topic because I don't want to bore Sean and we have questions to ask him. So the short answer is, I wish it was that simple. It's very fuzzy. And in the midst of the mechanism that curates this is basically a Gemini. endpoint that once it has a short list of potential candidates, it feeds the title, the description, and says, is this actually an interview? Because in the world of podcast, there's a ton of people that, like, just put Elon Musk in the title in hopes that it shows up in, like, search results.

But they, they are not interviews with Elon Musk. And there is no, like, 100%, what do you call it, like zero false positives way other than getting like an AI to check. But we'll, we'll kick that down the road. Anyways, so maybe not by episode 300, but maybe by like 301, 302. If you want to go and unsubscribe from ADSP, if you're only here for Sean, you will be able to do that now. And with that, I mean, Sean, I guess a question, first question, did we miss any, is there like an interview with some Microsoft podcast from like 2008 that we don't know about that this missed? Or is this from what you can tell, a full curated list of all your interviews? I think there are some video interviews. That's probably all the podcast interviews. And there's some interviews in writing. So, but search my papers and presentations page. And you'll probably find them there.

That is some future turf that I'm sure YouTube would come after me is I saw a couple. One of the personalities I follow on here is Boris Churney, the creator of Claude Code. And I saw, if we can go to him, I saw that there was a couple interviews or like panel discussions on YouTube that I was like devastated. I was like, oh man, like how do I, but they weren't, they weren't in a podcast RSS feed, which means there's no way this is going to pick it up. But that would be amazing if it could go and like find articles published by someone or like YouTube panels and then like text to speech the articles. Although like now you're getting in like, I don't know, copyright infringement territory. And that's why YouTube would be upset. But it's an idea for exploration. Anyways, we will leave. a link to a future availability of this app, but also to the link that Sean just mentioned.

If you want to go check out any of his podcasts or articles or interviews on YouTube or past talks as well, we will leave links in the description. How has it been, Sean? I don't think it's been as long as the last time we talked to you in between. It's still been a couple months, a few months. So how has it been? It's been pretty good. Let's see what's gone on since then.

I took a sabbatical, so that was nice. kind of got out of here for seven weeks. So didn't think about work for a while. Although just before I left on sabbatical, I had installed Claude Code and started to kick the tires. Now I'm back and finding that I'm not writing code anymore, which is a bit of an existential crisis for me. So wait, what were you using before?

Because I know that we talked to you about how you had like resurrected some docs that you would always want it to do, but, you know, it was just you're not going to waste, cycles on that because it just wasn't saying for your buck, but then with AI, it accelerated. So what were you using back then? And I guess that is what you're using now. Yeah, so I was using cursor. And so it was largely at the stage of, you know, you write a prompt and, you know, it does the thing. And, you know, and had mixed results with it. So I had some good successes, like you mentioned resurrecting old docs and some things that were fairly

structured. But it wasn't nearly at the point where I am now where it's like I've got a whole workflow set up with cloud code, multi-agent workflow, and I give it a task, and it writes the spec, writes the plan, writes the code, does the code review, gets in an argument with a GitHub co-pilot about the code review, completes the code review, submits the PR, and moves on. So, yeah, it's my workflow at this point is largely it, automated. and I was thinking about, I've got about 34 open issues right now on my project. So I was thinking about, do I just, since I have the complete workflow for each thing, but right now I'm just manually kicking off each task,

am I at the point where I have enough confidence that I can tell, uh, tell Claude, hey, so you're not stepping on yourself, bucket this into about three work trees at a time, uh, where each of the work trees is working in a separate area. You know, implement the issue, see it all the way through. to push it to the code, pick up more issues, and just grind through all the remaining issues on the project. So this is fascinating because literally episode 299, I think I ended up calling, do we need humans in the loop? And Bryce and I, I was driving, which I shouldn't have been doing, but, you know, I had got behind schedule.

And we were having basically an argument. Bryce was saying, you know, for CCCL, it wasn't possible to like, I think, I think actually it was a 10x more issues or PR. issues you were saying it was like 300 open you're saying 34 and also too should we mention what what's the project i'm assuming it's one of the stu lab you know there's adam there's eve there's a bunch of things which is the one that you're working on right now yeah so on github right now it's it's it's github slash stu lab slash cell dash r s r s because it's written in rust and it sells c l which stands for common expression language. And I started on this project. The Adam and Eve projects have underlying them a common expression language in C++. And I'd started on this as kind of

rewriting that infrastructure, but making it statically typed and writing it in Rust. And doing that. And I was doing that with cursor and making general progress. Now it's evolved into a re-implementation of property models, aka Adam, sitting on top of the cell infrastructure. And I've got a demo slash debugging application for looking at property models. I've got a language extension for for BS code that does you know formatting and syntax highlighting. There's a language server. So it does real-time diagnostics to your typing. A new planner for the probably haven't talked too much about Adam here, but it's a multi-way constraint system. And so there's a new planner for that, which was solved a long-standing problem with the atom system,

where the solution before was to manually hint it, but you had to know a lot to manually hint a particular class of problems to avoid a contradiction and had actually stubbed my toe trying to solve the planner on that with Alex Steppenoff, and then we invited in Bob Tarjan, who's a train award winner, because Alex had a correct observation that this somehow intersected, connected component problems,

and get graph theory and targent wrote the algorithm for that. And we couldn't come up with a solution to the planner. So I had this kind of hacky, hacky manual hinting that you had to do for the planner. And plot solved that. Wait, wait, wait. Hold up, hold up. You just said that between yourself, Alex Stepanov, who arguably should have won a Turing Award, and then another Turing Award winner who, did he win the Turing Award for Graph, like theory? For graph theory.

For, yeah, yeah. So you've got, you know, basically one and a half Turing Awards, three individuals couldn't solve it, and Claude code. What model were you using? Fable, Opus 4.8 or? No, it's actually, what is it, 4.6, I think, is all that Adobe's running at this point. Wow.

Wow. And it was interesting because I, you know, the prompt I gave it was to go, I said, you know, this seems like it should be a solved problem somewhere. And so it was like, review the literature and see if you can find a solution. And if you can't see if you can build off some existing solutions because there's lots of planners for constraint systems out there. And so I thought, you know, maybe I've missed something and maybe it's a known problem. And I just just haven't found the answer.

And so it went off and came back and said, well, it's not a known problem, but we can assemble some known algorithms and solve it. and including Targent's connected components fell in there. So the intuition was kind of right. And I have it set up, so it does its work in a work tree. So it did the work tree to solve the problem. And it was like, there, I'm done. And it pointed out while it was doing it, it said, oh, by the way, there's these three issues I spotted in your existing planner.

And so it came back and says it's done. And so I kick up my demo app and I write some test cases and kick the tires. And it's horribly wrong. It's horribly broken. And it was broken in the same way that we hit problems with previous attempts. And so I thought, well, that's very depressing. Okay, abandon that work tree and effort. And I'll go fix these three issues in the existing planner.

And doing that, I found another issue and fixed that issue. And then I went back to Claw. and said, okay, so we've got this failed attempt in this work tree to solve this problem. So let's implement the hinting system that I've already built, which we know is a viable solution. And do you have any suggestions to clean that up? And Claude literally came back and said, well, why would you want to build a partial fix for this when we have the complete fix in this other work tree? And so it was like, well, it's not a complete fix.

Here's a test that fails. And they came back and said, yeah, but at the time that I wrote this, the code. I didn't have these four additional pieces of information from your incorrect planner. Knowing that, I can fix it. Do you want me to? And it did. So it's been, I don't know, two and a half, almost three weeks now since it completed that. And I have not found an issue with it. So, so. Maybe it's just me that, you know, I, of course, find the AI to be very intelligent, but maybe it's just me that constantly has issues with it. Cheating. misbehaving, doing weird stuff.

It's, you know, I have the saying, LLM is people. And if you know, though, it's a reference to Soylent Green, you know, if you ever read the book or saw the movie. So I think I must have read the book in school. Right. So the short premise of Soilent Green for the listeners who may not know is that the planet is dying. And the only food source left for mankind is algae,

the oceans. And so the government is harvesting algae from the oceans and creating a product called Soylent Green that they're using to feed the people. And the protagonist in the book discovers that this is a vast conspiracy, that the oceans are actually dead also, and that Soilent Green is made from corpses. And so the population will continue to diminution in mankind is doomed. So a happy upbeat story. So what do I mean when I say LLN? This people, Well, a few things. One, it's distilled from human knowledge on the internet, right? It's this, you know, grind up people knowledge, pour it into a system, and out at the end, you get some results. And it's, it's, you would expect things to, to reach some norm, some kind of average of intelligence in, in, in, in such a system.

So, so it seems to, I think, exceed average because it's got a huge amount of knowledge, but it's not particularly bright. it just knows a lot of stuff. It's got a big brain and not a lot of wisdom. And so using the LLMs is a lot like a management task, right? What would you do if you were building a team of junior engineers? Well, you wouldn't say, I'm going to tell you what to go do, and you're going to run off and you're going to write a whole bunch of code, and you're going to come back in a day.

And I'm going to feel like my only choice is to accept that or dismiss it. and if I tell you how to fix it, you're going to go off and make the same mistakes again because you don't know that much and you're not that bright. That's not the way you're going to run a team, right? You're just going to frustrate yourself if you're managing a team like that. Instead, what you're going to do is you're going to start to build in process. You're going to say, look, for everything you write,

I want you to write a contract on every component. I want you to write a set of unit tests that are only looking at the contract and the interfaces. I want the person next to you to then review all of that to make sure that it's correct. And then I want all those tests to run until they're passing. And you're going to build up a process, right? And you're going to start to say, okay, well, you know, you're kind of going off in the weeds here. So let's start by writing a spec and let's discuss what a good spec looks like and build out that process.

And, you know, Claude Code, that's kind of the magic of Claude Code, right? It's got a lot of process built into it, and it's got a system with skills where you can augment existing skills and you can expand on the process. I have a lot of issues with Cloud Code that I would like to see addressed, but it's a good kind of 1.0 release of how do you build an engineering workflow? And just like, you know, if you're trying to build a product with a team, you're going to start to run into limitations. You're going to start to run into scaling issues. You're going to start to see, oh, look, if I'm making this thing too complex and I have to reason about a very large system and understandable, if I change something over here, it's going to permute something over

there and you start to break local reasoning within your system. So kind of, you know, the secret to building a scalable AI system is the secret to building a scalable team, right? You want to make sure that you preserve local reasoning within your code base so that you can have the code reason about things in isolated pieces and still be able to assemble them as a whole. And you want to build up layers of abstractions that when it's working on the next layer of your system, it doesn't have to know all about the implementation of everything under the hood, which is why you want contracts on all of your functions and all of, and basically you're documenting your whole system as you go.

And the documentation isn't necessarily for people. It's so that the AI doesn't have to reason about that. code, it can just look at the contracts, you know, and build your system up. And, you know, constrain languages like Rust, I think have a lot of value in this space as far as helping to guide the system to be correct and enforcing local reasoning at a compilation level and, you know, building it up. So so far I'm getting pretty good results. I am even on my little green field project here, you know, starting to run into scaling issues. You know, I'm noticing I've got about eight or nine components,

and occasionally I've got to do a piece of work that spans all of the components, and on those pieces it does the worst job. Is that still on Opus 4.6? Still on Opus 4.6, yeah. I would highly encourage convincing Adobe to let you jump to 4.8 because...

Or just you can use GPD 5.6. It depends on what you have access to. But I remember at some point, There's kind of these naysayers that I hear on different podcasts that I listen to, and they say, oh, like the marginal improvement is decreasing. And I mean, it depends on your task, but one of the best things that these latest, like, incremental from 4.6 to 4.7 to 4.8, although mostly I've been on Codex models, is that the issues that you have disappear, not necessarily the code gen or anything or what it's

capable of changes, but like the things that you run into where, like, you, you, you ask it to do X and you just know that it's one of the few things it's not great at or you have to hold its hand. Those things like more and more just disappear. And I like the the boundaries of what I think it can't do, like they just grow a little bit. And it's still, you know, like I said, the code that it's generating anyway. So I'm not sure specifically for your case, but. Yeah, I suspect that that is true. I mean, I even notice, you know, if you look, Claude Code updates itself practically daily.

and they're relatively minor changes, but they accumulate fairly rapidly, right? And that with, you know, building out your own skills, it's like, yeah, I see there's just this solid progression of improvement. And like I said, just the last couple days I've been thinking, well, I'm really not in the loop much in my project anymore, right? I've almost automated my way out of it. And so, like, do I have enough confidence? You know, maybe I'll spin up a fork and just be like, hey, go, go complete it. Just like, I think I have issues open to get me to 1.0 for a release on this.

So it would be like, yeah, just grind through all of those and wrap this up. Is there a way that you would be confident that it actually did it? Like, is it a matter of you testing like four things that currently, you know, whether it's Adam or cell RS aren't capable of and that if you just like, it's like, you know, the stable sort is at the top of, you know, a bunch of other things. If you just go test that, you know that everything leading up to it like 98% chance or 99% chances working or would it take some time? No, at this point, I've got the core planner done.

And so that's kind of the core of the constraint system. And so the rest of it is features on the peripheral, right? right and so just you know a good set of unit tests which actually automated the writing of those i think cover it and i've got enough test cases and stuff in there that are already stubbed with like okay i can't can't light this up until i turn this feature on right you know at this point i think even if i were to do it with the process i'm following i'm probably maybe a month or two away but if i just let it loose i don't know maybe three four days i have experienced on a few times if I just let it loose on all the issues,

that for a lot of my projects, it doesn't get it right in the first try, and that might just be that my issues are not well-specified enough or there aren't sufficient testing. But, yeah, I think that the problem, in a lot of cases, I've not sufficiently specified the problem, and so I can't just let it loose on all the issues because it just make a mess of things.

Yeah, I mean, a very honest. thing that I found is most of the time, well, you know, so I built into my rules that I say, because a lot of times it will do like a multi-phase implementation of some larger task. And in doing that, maybe I stop it before it completes all the tasks because I want to do some other things first. I will have an open GitHub issues for the remaining phases it didn't do. If it finds things while it's working that are out of scope, but it thinks should be fixed, I have Claude open GitHub issues for those. If I spot something, I will typically ask

Claude to open my GitHub issue with a small prompt, and usually that's in the middle of working on something, so it understands the context around it, and I'll write a one-line prompt, and it will open a three-page issue and be like, you know, and quite well written.

It'll be, you know, I identify this problem, and here's where it is in the code, and this is why it's a problem, and do a full explanation on it, just because I pointed it out. So my GitHub issues tend to be, you know, very verbose and detailed, but I'm not even writing those. Yeah, that's fascinating.

I'll be interested to hear how, like, whether you're able to get there with just the agents, because maybe I'm just doing something wrong. I mean, there's a few things. Like, you know, so I'm working in a green field. a relatively small, narrow project that's got, you know, 20 years of academic papers written on the topic of which several I've written. So it's a relatively well-defined and relatively narrow, narrow problem. Now, you know, one voice in the back of my head that I have is,

is my original premise on this, you know, it's kind of, you know, property models now have been around for 20-something years. Why aren't they in widespread use? You know, if they were in widespread use at Adobe, they would eliminate somewhere between 15% and 30% of the code in our products. So it seems like it would be a big win. But I've found that they are difficult for people to author, and it's a difficult space for people to think in. And so it takes kind of a different mindset. And so my premise when I started to revisit this was, well, if I, you know, build

a better engine that requires, you know, eliminate some of the manual hinting that's going on. I mentioned one instance of it. There were two instances where kind of manual hinting was required. If I could eliminate those two instances, if I tighten the constraints a little bit by making it statically typed instead of dynamically typed, and I added better tooling around it, you know, if I put a language server on it and then group in a, could visualize the graph for the constraint system, then maybe people would understand it and it would be a significant uptick.

But now, it's like, I don't think people will be authoring this, because I don't think people will be authoring code. So the whole thesis and premise changes to, this is a higher level way to describe a problem that is much denser than the typical imperative code for describing this type of problem. And so if we build a DSL for this, can the AI is more efficiently author inside of that DSL and have better outcomes and lower token cost?

So I've been doing a lot of work recently on evaluating programming language efficiency and the efficiency of DSLs versus more mainstream languages. And one of the main takeaways is that there's a very large incumbency bias. So, you know, if you've got some new DSL or some new framework or something that even if it's better and more efficient, it might be less token efficient than the ancient, crafty existing thing that's been around. for a long time just because there's more literature on it. There's more knowledge on it. The model was pre-trained on it. The model was post-trained on it, et cetera. Yep.

Which is a little sad, a little surprising. We have all these cool like Python DSLs for authoring Kuta kernels. And Kuta C++ did surprisingly well. The most interesting thing is that the JIT compilation times for Kuta C++ were actually comparable to a lot of the DSLs. But I have a theory about that, and my theory is that that's because of the type of code that they're writing. So the agents write, they don't use templates. They don't use abstraction when they're writing their C++ code.

They're just like, I'll just specialize for everything. I'll just have simple functions for everything. I'm going to write C style code, and I'm just going to write a lot of it. And so, yeah, it's fast to compile because it doesn't have any of the things that make C++ slow to compile, like template instantiation. but there's just a lot more code. And you would think at some point that the code would get to be enough spaghetti code that would be harder for them to work with. And maybe it does, and we just haven't run the experiments out far enough.

But, yeah, they're surprisingly good at writing awful C-style code. Yeah, I mean, this goes back to the LLN's people premise, right? Well, it's really interesting that you say that, because the other finding that I've had is that all of the things that we perceive as being difficult for humans to deal with. And all of the developer productivity concerns that we have about like diagnostics, stuff like that, it's all an issue for agents too. Like two of the biggest sources of token inefficiency, aside from reward hacking,

two of the biggest. Well, actually, including reward hacking for one of them, diagnostics is a big one. In particular, diagnostics that happen late. The later the diagnostic, the worse it is, If you only find out about the failure at runtime, then, like, you know, the model goes and, like, writes out a prototype of the code. It's like, oh, let me see if this compiles.

Oh, it compiles. Okay, I'll flush it out a little bit more. And then it keeps going and keeps going. And then eventually it goes and runs it. And then only then does it learn that it wasted all that time because there was a mistake. Or, like, you know, it's something where it passes some of the tests. It passes some, like, lightweight tests, but then you run it with the full test suite. And, oh, no, that didn't work.

It's like, okay, well, if those were all errors that we could have caught at compile time, it would have been easier. And ironically, the JIT compiled languages, the JIT compiled DSL languages that are often easier for, you know, higher level, maybe I should say, not necessarily easier, but they're higher level. You have more of those types of failures with that type of language than with a statically compiled language like C++. And presumably things like Rust would also be good. the other big source of token inefficiency, at least in our line of work, is race conditions and concurrency. And there's two factors to this. One, of course, just concurrent programming and debugging is hard. The bigger problem is that they're flaky. So the model goes in it, it writes the code that introduces the race condition, and it runs the tests, and they pass, and it's like, great.

And then it keeps working. And maybe you've got like a parallel swarm of agents that are working. And then like, you know, some of them start having failures. And they're like, oh, that's weird. Where did this failure come from? And you could end up having it waste just as a human would, waste hours trying to track down this flaky, you know, race condition. And it's, there's something very rewarding, redeeming about seeing all of the things that people that we who have cared about developer productivity for years have said that these things matter. It's, it's very, it's nice to see those, like, reflected in. the actual token efficiency.

Vindicating is the word you're looking at. Vindicated. I feel very vindicated. Here, I'll show you, I'll show you one thing. I just happened to have this up. This was, this was not, not prepared. I'll show you this one thing, which was, this was, this is the auto research log for the, the cub histogram auto research run that ran for like six days,

130 hours. And all of these yellow lines are commits where it did some form of reward hacking or cheating that was found out retroactively. And all of the blue dotted lines are places where it found race conditions. So there's like an eight hour period. There's one eight hour period at 50 hours. And there's another couple hour period early on where it introduced some race condition and then just wasted enormous amounts of resources trying to get out of it in like realizing, oh, I made this mistake.

oh, I need to go back and, you know, like roll back all of this work and then all the work that I did that built on top of it was maybe invalid. And it's just like a, it takes a lot of effort to recover. Yes. And yeah, well, you're somewhat, you know, making my point, which is, I think, just like people, right, right, there's this incumbent bias where it's like, oh, well, it's doing a good job with C++, but then, you know, you wouldn't be having race conditions if you were writing in Rust, for example.

Right, right. That's, yeah. That's one of the things that I certainly saw in some of my results, that in abstraction layers where you're not explicitly doing the concurrency, where it's sort of automated from you and these sort of higher level functional paradigms like tile programming, you, you know, you don't end up spending time on these errors because they just, they just don't come up. The other thing, the thing I've been thinking about today is how do we make models better at this and like how do we come up with like a benchmark suite for this and my idea is well let's let's come up with like a set of tasks where the model is given some code and the code has a race condition in it already but the model doesn't know that and we give it some other tasks to do in the code base where it's going to become hampered by this race condition and it'll have to go and find the race condition and fix it and you come up with like 10 tasks like that but the challenge is that, like, I was thinking, oh, we should call this racebench. But the thing is, if you call it race bench,

then the model's going to see in the name of the thing that it's called racebench. And it's going to be like, oh, I know what's going on here. Yeah. Yeah, instead of benchmarking, I mean, where I would like to get to is get, you know, an engineering process app that I can package up a complete engineering workflow into and deploy it at scale on team. And then I can change that workflow and permute it, right? Right.

I can be like, like, you know, dial up and down the detailing contracts or, or change the rules for how unit tests are written or change the rules around specification and deploy this at scale enough so that I can be measuring teams and saying, okay, you know, how much code is this team generating and how much of the code is coming out of their product? And now I can start to do testing where I can say, okay, I'm going to deploy this on various teams and see, does their productivity go up or down, right? Are they getting better results at lower cost or not? And, you know, it's the same thing as doing like, you know, a medication study. Right.

I think tools around the software engineering process and in particular tools and frameworks for that help us test and ensure quality in figuring out how to quantify quality and measure code quality are going to, it's going to become a really important industry. That sounded like a compelling pitch for a Sean Parent startup. I would buy shares in that in that and that Sean Parent start. of. Yeah. Not, you know, I don't know, maybe Anthropics should hire me and I will help them build the product.

I have the expertise in the engineering process side, but not in the, how do you build the LLM harness and do the training on that end. So if I were to do a startup, I would have to hire good people, which are hard to come by with that expertise. I've started getting more into that world on the training side of things recently. Yeah, it's very interesting process of like thinking like you end up thinking a lot about like how are you, how does the model learn and then like how are you going to teach it and like what thing, especially for post training. Like what, you know, just in the same way that you'd think about how you would teach to humans like, but just sort of more at a scale and industrialized. Like how would I actually distill these sometimes intangible things that I care about into something that I can. you know, make testable and then like trainable and like some loop where I can tell it, I can give it a task and I can have it go practice that task a lot of times. But it's tough because like you have

to think about all these things that maybe we've not verbalized or written down or formalized. And like, how do we do that? Yeah. Yeah. My approach on that so far has been, you know, just keeping an eye kind of spot checking and monitoring things. And when I find something that I find objectionable telling Claude, okay, you know, this is, I find this objectionable, this is why. Can you turn that into a skill or a rule? Let me review that. Now apply that across the entire code base, right? Catching all the things that I didn't catch.

And now let me look at the diffs. And, you know, I might throw it away a couple times and be like, no, no, no, that's not what I meant, not what I said. Right? But review the dips and be like, okay, refine that a couple times until I'm happy with the result. I'm like, okay, now we can commit that PR, which is just an update to the rule set and move on to the next thing. It's funny that you say that to you because there's a theme across this discussion, which is like, you know, initially, although you made me think that it worked the first time. But when you had cloud code, you know, tackle the connected component, you know, plus plus problem, it failed at first.

and then you said, okay, well, you didn't do that successfully, but then you came back to it, and then luckily it just said, oh, well, like, if you had given me this extra information, like I could have used that. And the same thing here, right? Like, you're asking it to go and create a quad skill and then, you know, applied across the codebase, and then you iterate. And I feel like a lot of folks, they expect this stuff to just work. Like, it's just like, you know, one shot or zero shot or whatever, you tell it, and then it's just perfect.

And, like, I find that so shocking because that is not how humanity, like, any points in time, whether it's engineering or mathematics or just working with any people, whether it's like a career or profession or you're just having a conversation about like, oh, you know, what should we do with the backyard? You know, it's not like you ask a question, there's an answer and then boom, that's it. We're done. It's like a back and forth. And I feel like a part of the reason that I am so successful with these models is that it's not that I've suspended disbelief. It's just that I know that like if I work well with these tools, like I can get where I want to go, way faster than I could, like before.

And it is not necessarily going to be perfect. But like whenever it fails, I don't think, like, why is this not working? I think to myself, yeah, I probably didn't communicate that as clearly as I could have. Like, you know, there's a quote from a movie, I think it's called The Last Emperor. And it's about like the Last Emperor of China, Poo-E, I believe.

And there's this great quote and it says, a gentleman cannot mean what he says if he does not say what he means. Words are important. Why are words important? If you cannot say what you mean, your majesty, you will never mean what you say. And a gentleman should always mean what he says. Like, I always whip that out every once in a while when, like, you know, my shema and I, my wife, you know, she says something. She's like, oh, you know what I meant.

And I was like, no, I don't know what you meant. You know, you literally said something. And then expected me to not take the semantics of what you said, like, just, oh, wow, you know, why do you have to be so, like, hyperatological? And I'm like, I just listened to what you said. Anyways, we love each other. We're doing great. baby's on the way. But the point being is that, like, communication has become so important with these AIs. They do take what we say literally. And even sometimes it actually reads between

the lines. It knows that what you meant is actually different than what you said, because, like you said, it's trained, you know, LLM is human. It's trained on so much text and books that is reading between the lines. And it's, well, you know, if the person said X, what they actually probably meant is that, you know, A, B, C, D, and then you get X. And so I'm going to assume that that's the case. and then you watch it thinking, you're like, holy F, like, that is crazy. Like, it knew what I meant, even though I didn't say what I should have. God, I'm just pitching you all on all of my ideas today. We're going to tell you to start a second startup?

Well, this is less a startup, but I was thinking about, like, how do I, how do I collect all of those things, all those, you know, design sensibilities? And like you could, I'm thinking both about how you can do this in like a synthetic way. How can I generate these design sensibilities from a model? But also, how can I collect it from humans? And that's what led me to the idea of Tinder for code. So it's like an app where you get shown some snippet of code. And you're not looking at it for very long.

And you're deciding, like, is it a good code or bad code? And then if you decide it's like either way, like you leave some feedback, maybe it's not even text. Maybe like, you know, there's some. list of options. But the idea is like it's going to show you like, you know, 10 lines or like one function. And it's going to be like, is this good or bad?

And I don't know. Just it's like they came up with it as I was working out my mother earlier. And I'm like, hmm, maybe this could get me the data I need. And there was some leaderboard, but I don't know what to call it yet. I'm sure you two could give me a good name for what we would call. Just steal, Zuckerberg, it's hot or not, except it's for codes. Yeah. I've been wanting to ask since you said at the beginning of the podcast, Sean,

that you are going through like a little existential crisis because, you know, join the club. And then you also said, I don't know, 30 minutes later, that you don't think people are going to be authoring code. And anyway, I'm just, I want to get your, your larger thoughts on, yeah, like those two statements. Expound. I want to know more because like I spend a ton of my time talking to my wife about like, this is going great. I'm very excited, but like, let's play this out. And like, I am sure I will be employed if I want to be employed, you know, several years from now. But I definitely do not think what I am, you know, what my job looks like today. I think it's going to be vastly different. And I'm in a great position. But like I just, you know, we have these philosophical discussions where I'm just like, what is actually society even. I tried to have this discussion with Bryce last time. But he missed what I was trying to say. And it's just like, you know, what is the point? You know, like it hasn't come.

out yet. It'll come out probably the next episode about like I was I was talking about conferences and committee meetings and like are they actually like do they serve a purpose and like maybe the purpose is that like it's actually just nice to meet with people in person like yes you know you're wearing your CPP con shirt you go to a conference you do learn something but like really humans crave like social connection and and like you know if a conference was like just the learning aspect and you don't have the hallway track and you don't get to have conversations with people like that's just like a you know, a module that your boss sends you to like stare at for a couple hours. That's not like fulfilling. Like it's much better to go and, and, and have real conversations with real human.

Anyways, and so like I'm thinking like if our whole industry falls apart, like, and we're not going to conferences is like, you know, I'm going to just be buying gnomes and putting them in my backyard and pretending to have like a fake conference with like my backyard gnomes? And anyways, over to you on everything I just said in existential crisis and not authoring code. Yeah. So, so for me this. you know, hit pretty hard because Dave Abraham's, who was a member of Software Technology Lab on my team, he really had a pretty strong reaction, anti-AI reaction, and it was like, you know, this is going to destroy the industry. And part of his answer was, was he got involved with

the band, playing music, and retired. And so he and I were working on the better code book in making some solid progress. And I've, since Dave retired, I've done very little with the book because, one, I don't have a collaborator on it every day now. And two, there's this who's going to read it. Right, right. What's the point? The first couple of chapters make a really good prompt. So I have my Claude rule to say, go, go read the start of the book. Wait, this is, this is like you already dropped too massive. First of all, I didn't know that Dave had, did you know, Bryce, that David retired? Yeah, I think we talked about this. Definitely not on this. This is news to me. This is the first, well, unless if I have a bad memory,

But definitely... You are getting old. So it sounds like partially, if not entirely, the reason that Dave retired was because of AI and where he thought the industry was going? No, I think his primary reason was he had an opportunity to play his music full time. And that's always been a passion of his. So I definitely think that was his primary reason. But I think a secondary reason was just... the same existential crisis of,

he viewed it as, you know, he had built his career on trying to help engineers write better code. And, and what does that mean now? And so I spent my sabbatical, giving a lot of thought to that problem,

and came to a somewhat different conclusion than Dave came to, which is at the end of the day, I don't, yeah, there's Dave Abrams, his website, it's great. Yeah,

you know, we'll see, he's got some of his, It's really good. We will have to have him on, and hopefully he'll, even though he's retired, he'll be happy to come and tell us his thoughts. There you go. But yeah, on my sabbatical, I spent a lot of time thinking about this issue. And the conclusion I came to was, was I never cared about the people in that equation.

You know, the reason why I've pushed for better code is because I want better code. So back to the LLM as people. If I can get the LLM to write better code, then that's pretty fulfilling. I'll tell you what I think that people like us will be doing, which is I think that people like us will inevitably gravitate towards developing models that are good at coding. Like for somebody who's an experienced software engineer, experienced in programming language and library design and all of these things, somebody who's more on the systems end than on the applied end of software development,

I think like it's just natural that we're all going to gravitate towards model development. I think there's some truth in that. I think that, you know, and from a societal standpoint, that creates an interesting dilemma, right? Those of us with experience will be employed at least for a while as we put ourselves out of business. you know, I also talk to a lot of people in their first comment is, well, software engineering looks like a solved problem. So what are you going to do next? And I'm like, do you have to understand that what software engineers do is we solve other people's problems? So if software engineering is a solved problem, so is everything else, right?

It will just all tumble. I would say that I think that coding might be a partially solved problem or close. Software engineering, I don't think we're anywhere near to that being a solved problem. Like, show me, show me, show me, show me an AI system that can ship software at scale over the course of multiple years without a human in the loop. I think we're pretty far. Sean has literally said he's about to fork his cell R.S. And say, go solve the outstanding issues. And, you know, if we're not there today, we are very close, Bryce.

I know your experience is different, but like three years from now, there's no way that we don't get to a point where large software, or I don't know, maybe Sean, you might disagree. You're not accounting for the long horizon behavior. And as me, somebody who's been exclusively focused on longer horizon behavior, as soon as there is misbehavior, it very quickly becomes a feedback loop. And without really, really strong processes and validation and guard rails, over time, the system becomes more and more incoherent. Yeah, but that's true of people doing engineering too, right? Like, I have people who are like, you know, oh, look at this horrible code the LLM wrote. And I'm like, crack any file in Photoshop and you will find it.

Right? It's a 30-year-old code base that's been built incremental. and it is at the point where I would say in many ways it's kind of losing coherence, right? There's nobody on the team who can keep the entire project in their head anymore. There hasn't been for 20 years, right? And so I think that it's true of LLMs and it's true of people. And I kind of go back to the failure points are the same. and if you expect to get good results out of an LLM without putting in place the whole process,

you're going to be kind of in for a world of pain, right? Right. And yeah, I don't think that having that whole process in some shippable form, like I said, I think Claude Code Codex or some of these other systems are 1.0 system of that, right? We are just in the beginning. I don't have a way even in cloud code to specify a complete engineering process and then deploy that at scale and say, you know, follow this process. But I think that's going to happen really quick. I think there are a number of companies that are working in that direction.

I'd be shocked if Open AI and Anthropic and the other big AI vendors, Google, Gemini, are not seeing the same thing and going after the problem. Right. Like I was just reading an article lately that was like, oh, you know, chat GPT is not that smart because we tried to have it run a business, right? And we let it be the boss of the employees. And it did a horrible job. And I'm like, well, yeah, sure. You know, if you hired some random kid off the street and you said, you run this store, we'll just ask you questions you answer. That would be a disaster. And that's effectively what they just did, right? It's like you need the person who's run the store who's got the experience to set up what is the process for running the store and build that into a framework for the AI to follow. But yeah, I don't see that we're going to hit a limit. And I think society, you know, I keep saying between 20 and 40 years, that's still kind of my back of the envelope evolution, although I think we're going faster, so probably closer to 20 years. We're going to hit a point in 20 years or less where the AI can do a better job at any job than a person can. And what does that mean for the societal impact? Right.

Right. And it will do it cheaper, pretty much. Yeah. Okay. So, so from picking strawberries to building rocket ships, right? I think it will be doing a better job. So from my standpoint, I'm like assuming society doesn't completely unravel, I'll retire in that time frame. I'm out.

I was going to say, so Dave's already retired. Sean's going to retire. So it sounds like just try to get yourself financially in a position where you can retire and you'll be good. Yeah, I hope the economy doesn't take so badly that it doesn't destroy your retirement. Yeah, on ST Lab right now our challenges, Dave retired, Mark Hambur retired, Nick DeMarco took a medical leave and then announced he's not coming back. So my team of five is now a team of two, myself and David Sankle. So, hey, that's a pretty good team of two. Is David, uh, talked about retiring?

No, no, David's not, not talking about it. David, David has like 13 kids and they're about to start entering college age. So I don't think he's going to be. retiring anytime soon? Well, I was on, it's not quite 13, but while I was on 12 or 11 or something. It's 10. It's 10. But is it, but it's 11th coming? No,

the 10th just came. So while I was on my sabbatical, he had his 10th. So, so he's got stepping off beat by one. So Wow, stepping up has nine kids. Is that right? No, I'm sorry, stepping off his eight kids. He's got stepping off beat by two.

Wow. So, I had no idea. Yeah, yeah. You got to catch up, Connor. You need seven more kids. I mean, we've talked about definitely having two, but there's, I don't know. I think honestly, when I think, I don't even, when I think of eight or ten kids, like, Shima and I have had conversations about how many we want. And I think about it from like a getting on a plane perspective.

Like, if you have two kids when it's like a big plane, it's pretty good because you can sit one parent on each aisle. and then you each directly have access to a kid. As soon as you have a third, you're going to, like, one parent's going to have two kids on either side and then the other one's going to be off. If you got eight or ten, like, I guess you're just not flying,

because that's like, that's impossible. Well, presumably they're spread out a little bit. So that's where you get to be underwatches and the other ones. Yeah, that's a good point. I didn't think about that. I used to ask David, like, when he was like on kid number seven or eight, I asked him like, oh, like, isn't it hard?

And he's like, you know, once you have like a critical mass of them, they sort of like the older ones, take care of the younger ones. And it all sort of works out. I wasn't thinking long enough down the path. Yeah. Anyway, so you're just going to retire. I mean, it's interesting too because like, well, I'm not retiring soon. Yeah, yeah, not retiring yet.

No, he's got, he's got to do the Sean Parent, AI, process, software engineering harness startup. Well, yeah. I mean, right. I would like to finish this run at property models and play that through. And it's kind of part of what I've viewed as a larger vision, which I refer to as declarative forms and this notion that the complexity of software at scale becomes structural and not algorithmic.

And so you want to have systems in place where you can describe the structures of systems and have that be a solved problem. So you want to be able to do that declaratively, and that's what I refer to as declarative forms. So you want to start programming at a much higher level when you're building things at scale. And I kind of say, you know, I think within, you know, a typical desktop product, you've got somewhere between a half dozen and a dozen declarative forms in there, and you want a system, you know, you can think of each declarative form as being this declarative DSL that describes it. You want those systems to all be similar syntax and to interoperate.

So I can be like, okay, well, I've described the parser for parsing the file, and that's going to map into a schema for a database, and I'm going to attach a UI, and that's going to have a layout engine, and that's going to have a behavioral engine, and you're going to have this set of DSLs, and they're all going to be the piping in kind of a functional language from one to the other to communicate and build your complete. system. So, so, you know, I'd like to, property models for me are kind of, you know, the first step in that, the first piece of that system that I want to build. And so I'd like to carry that out. I do want to finish the book. I'm not sure what form it's going to take now. You know, maybe at

some point I'll have the energy to pick it up and continue where we left it. How close to being done do you think it is? Oh, it's, we've got, you know, two solid chapters, two chapters in progress, and the book is slated to be probably eight chapters, right? So it's somewhere between a quarter and a third of the way done, I would say. Maybe for episode 400, we can launch the Sean Parent book tour. Here we go. Maybe so. It's tough because my key collaborators on that were Dave and Nick.

and I'm trying to convince Sankle to partner with me on it. I'm like, I have learned over the year that I'm much like Alex, which is it is very hard for me to stay focused on writing anything because I get started on it, and then I see all of these tangents, and then I've got to go build a bunch of code. So I need a collaborator to make progress. I have a similar problem.

So distracted by Lint, basically. I was also going to mention, have you heard of, have you heard of Lean? When you were talking about the property model and the constraint system, I was thinking about, you know, formal verification is like all the rage these days. Yeah. And everybody's talking about lean. I haven't really looked at it that much, but, you know, it's this language for expressing proofs, but also kind of just a programming language, too.

Yeah, I haven't played with Lean yet. It's on my list of things to play. with, I have played with Daphne, which is a similar programming system. I like Daphne. There are problems with trying to build systems at scale that are formally verified, but they're nice in the small. I think LLMs will help with the scalability problem there. I've had good luck with Daphne with LLMs completing proofs for me, which is, you know,

I'll get stuck and it'll be like, oh, hey, you just need to put this assertion in and you're good. And I think that's because in any of these systems, the LLMs have only trained off of correct proofs because that's all anything ever, you know, nobody's code into GitHub that doesn't compile. And within a formal verification system, compiling means it checks. And so the LLMs have like only ever seen correct proofs. And so know the rules for how to get there. Although that may not be true because I think there are some models specifically designed for theorem proving. Like I think Deep Seek had a approver model and presumably they did reinforcement learning. And so during the reinforcement learning process, the model would have written bad proofs and then learned, oh, you know, here's how I can write a better one.

That actually, I would not be surprised if that's in, to some degree, in the, like, you know, post-training recipe for the big frontier models, too. Yeah, I'd be a little surprised because I don't know that they've gotten to the point yet where they're doing, you know, post-training on proving systems correct. But maybe they're starting to, at least on the math front. Yeah, I would guess that specifically that they're doing it now because of this, like, arms race in. cracking the world's math problems. Yeah, yeah. I lost my train of thought. Where were we going with this conversation?

We were just talking about the... Yeah. I think really the question was about the applicability of formal verification methods to these property models and the constraint checking. Oh, yeah, definitely true in that, you know, I've been doing a fair amount of... I haven't formally verified the algorithms behind them, But the idea with property models is they prove a bunch of attributes on your system.

And one of the tasks that I have outstanding is to build an analyzer where everything that I can't prove it directly in the compiler, I've got this growing kind of analyzer task, which is I want to build a model checker after the fact that we'll do a deeper analysis to verify the correctness of your model. Yeah. And I think a lot of these systems fall into some form of a multi-way constraint system. it's just that the shapes of those

tend to be very different. So a layout engine is a multi-way constraint system, but it's trying to minimize a set of constraints. And you have scheduling multi-way constraint systems where you put a strength on constraint so you can say, well, this person doesn't have to be at the meeting. So I'd like them to be, but if you can't find a slot,

break that constraint and let them out. Property models are a multi-way constraint system, but all the constraints have to be satisfied. and the strength applies to the values within the system. So you can override a value and derive it or not based off the strength of the values. So a lot of these systems have similar structure, but to solve a particular problem in this particular domain, you have kind of this larger set of constraints the layers on top.

And that's also where I think a lot of people get stuck, right? They try to say, well, I can just build a generalized constraint, solve all these problems. And the answer is probably not, right? It's, you know, in the same way, like, or you make it touring complete. You know, you reinvent protocol. And it's like, well, sure, you can, but is it the right system to solve the problem? Yeah. So, yeah, but that's what I'd like to continue to do.

You know, the question is, how much runway do I get at Adobe? Like I said, my research team is down to two of us. We haven't gotten back fills. We're in Titus's org, which has gone through some significant reorganizations. So, you know, I might end up back assisting the Photoshop team or something. And go back to my evening project, we'll have to see. Well, hopefully, hopefully we don't have to have a third resurrection of the SD lab. Yeah, yeah, yeah.

I think we're in many ways, this is kind of the end of the second resurrection. So, yeah, that's unfortunate. Hopefully the rejoining. Did that work? It did indeed. Yes. This is a perfect way to segue back to when you were saying, open up any file of Photoshop, you know, and we can see it's like,

I went on this rant in the last episode about how, you know, I referenced this little clip all the time, Jonathan Blow, who's a, you know, game designer. He in the middle of a 60-minute talk went on like an eight-minute tyrant. of like everything he noticed that was wrong with like software he was using and he was he said he planned on doing it for a week but then like he ran out of like space in his deck in like a day and a half of just all the different things and like I've I've told a few people this over the last like month is that now that AI is here and it can like do everything and it's it made me

realize like how bad software is because now I'm just like like you can just get the AI to fix it and we have internalized dealing with poor quality software, like all the time. But fixing it will introduce new problems. That's the thing. It's like when Sean was talking about, like when he was working on CloudCo getting it, he said, oh, by the way, you know, I also noticed three issues. Like if you just open up, you know, Claude Opus, whatever your version is or Codex,

and you say, I've noticed an issue. Also, by the way, can you just like take a look and see if there's any other things? And it'll go and like say, oh, I noticed you're using this open source thing for cross-platform mobile development and it has a bunch of known issues and I went and checked to see if you were exposed to those and I noticed you were like it might have been might not have been causing issues but I went and had and like coded in some safeguards just to make sure you don't hit that stuff it's like no programmer I know has the capacity to do that like in a week let alone in five minutes like it just went in like consumed everything yeah but fixed my issue sometimes those sometimes those safeguards

are like the wrong safeguard no no no no GPT 5.6 doesn't let me down. What are you going to say, Sean? Yeah. First, I agree with the sentiment. It's like I can't use any piece of software for more than 10 minutes without, you know, having some gripe, right? This is wrong. This feature is broken. This thing drives me nuts.

And for the record, I might clean it up a little bit. Sean's audio stopped working and then he dropped out of the call, rejoined the call, and magically his audio started working again. Thanks for talking about. Software sucks. And Dave Abraham had what he referred to as A-Block, Abraham's first law of computing, which is computers, no damn good. Oh, yes. So just for context here, I've had the LLM start sending PRs for my new histokash algorithm to Cub.

And it's broken it up into four PRs. The first PR, I just earlier today gave the check of a, approval after and the GitHub PR thread it has 433 comments because I've been iterating on it for two weeks of cleaning up this like plus plus 1800 minus 800 lines of code diff and I didn't read every line of code you know I'm mostly skimmed and then like read its comments but it took a lot of iteration to get it from its initial PR to something that I think can land. Now, the people who are the actual Cub maintainers now have to sign off on it, so it'll probably be a more review after that.

There's probably things that I missed. I definitely think... I would look at this and say, you know, first, that's a huge PR that no person would want to review. And second, it's like, what are your rules that built that PR? It's like, is there a spec behind it that got written first? are their unit tests for it they got written first? Why wasn't it phased out into a multi-phased implementation at that scale?

But I think plus 2,000 minus or plus 1,800 minus 800 lines of code is not too bad. It's not too large. That's definitely in the category. Like everybody knows the anecdote of like the three different sizes. And it's like, you know, if it's too small, you get too many comments or whatever, blah, blah, blah. But like if it's big enough, people are like LGTTM, you know, you can just. You can ship it. And I'm sure our engineers won't do that.

We got a great folks over on the CCCL team and they will review it. But definitely it's like the bike shedding. You know, you change one little thing and five lines of code and you've got like 10 different people weighing in on. I'm like, well, I don't know about that. Right. So, yeah. So that would be the thing of like, like, you know, why did it get to a PR?

What was your process? They got it to the point where a PR were you still had mistakes on it. And all of your criticism in there that you went into that. PR, did you put that back into the system as rules of some form so that these mistakes don't happen again? Yeah, I mean, a lot of it, a lot of it was just not doing things in a way that was consistent with the unwritten coding guidelines of the project. Well, then that's a place where what you want to do is say, you know, go analyze the coding guidelines for my project and write the undocumented ones, right? Yeah, like, it got like a lot like, the latest round of things.

was just like inconsistent naming choices and stuff like that. But like the biggest problem was just like the way that it did the custom, like the tuning policy was just completely wrong. And like a mess and spread across multiple files and not structured. And there were a lot of things where like it added like a new constructor for something. And I was like, why do we need to have a new constructor for this? And like it added like a new mode for something. And so then this like class template now had like three bulls.

where you needed to understand like the right combination of them equaled like one mode and another combination equaled another mode and I'm like, no, no, no, take out those bullions, just give it like an enum that's like, you know, you explicitly tell it what the mode is.

And just a lot of like, like, the way that the abstractions were designed. A lot of the way that abstractions were designed was just not good. And this, I agree, the whole reason why it took so long is that our coding guidelines aren't sufficiently written down and we don't have sufficient ways to actually measure all of those design

principles and like I would like to I would like for there to have been a way for a deterministic tool to have flagged these problems like it doesn't need to it like it can have false positives but a lot of these things are things that I think a deterministic tool could have could have flagged as like this is poor quote quote right my normal process for that is is to say okay so you know analyze this and try to distill out what the set of rules are. So at least we have a document where they're written down. And then that gives you a place to start to make corrections. Right, right?

So now at least you've got a point in the system where you're not having an ongoing argument about, well, this is wrong. Why are you making the same mistake? It's, you know, it wasn't written down. Well, I don't want to take the time to write it down. Don't take the time to write it down. AI write it down. That gives you a concrete point where you can say,

go add this rule or fix that rule. That's a good idea. I will try that. Yeah. And then like CloudCode has this thing called doctor, and you can use that to kind of distill it down. And one thing the doctor will occasionally suggest, and I have it augmented with my own rule,

it's like if there's, you know, if you've got instructions in here on how to do something, and you could write a tool for it, like Python or something and save me tokens, go write the tool, right? somebody at invidia has a skill for that it's called determinize where it looks through your your skill and lives out the deterministic parts of the parts that'll save you tokens i listen to i literally just tweeted out and whatever promoted on lincoln a clip from an interview

with anders houseberg creator of i believe both typescript and c sharp and potentially a couple other programming languages and like i feel like this is a thing that anyone that spend any amount of time. Like, it's an unlock. Like, and the way he phrased it is, you want to park the stochasticness and then like unlock determinism because these things are non-deterministic, or he would use the word stochastic. If we just let AI loose on, you know, what is it, like a half million lines of code we have in the old compiler or somewhere between half a million and a million, right? And asked it to translate it all. Well, I don't know that that would absolve us from then having go in and carefully examine.

every line that came out of it to make sure that there were no hallucinations, right? Unless you have 100% perfect test coverage, you probably still got to go check all of that. Now, I think a better approach, quite honestly, would be enlist AI to write a program that helps you translate from TypeScript to go. Because at that point, you can part the stochasticness in that program, and then you can get deterministic behavior whenever you run the program, which means you get the same transformation every time you run it, right? That's always the thing about AI that people forget. It's not deterministic, right? So you can't really trust that it's going to do the same thing twice. Anytime you can get it to write a thing that will then do something deterministically, like one, it's no longer costing you any money, but two, like you're unlocking determinism and like avoiding the non-determinism.

And like that is a huge unlock. Like I have a ton of workflows that now just are just scripts that just run that like initially AI wrote. but now like AI never touches it and it just is humming along and opening PRs for me and I click merge buttons and like I reviewed every once in a while just to make sure but I feel like it's small things like these that I don't know like if you're trying to get people that are less technical to use this stuff they're not thinking like they don't like my actually my my brother-in-law is like trying to write some software for his company and he works in like the H-FAC business and I think to him

I had to really explain like he's just like oh let's just get AI to do it let's get AI to do it and it's all just like functions to him are all the same thing. And like to me, it's like, okay, if we can not depend on AI for this, that is better. Like when we need to do some optical character recognition or like analyze this photo, okay, we'll hit a Gemini endpoint or a codex endpoint or whatever. But like if we can do this deterministically with some kind of formula, that is so much better from like a simplicity because like every once in a while, the AI is going to get it wrong.

And then he's just like, well, why is it wrong? How come the AI is wrong? I thought AI supposed to be good. And I'm just like, well, you know, we need to minimize that, like, non-deterministic footprint that exists in our software program. And anyways, that's like, yeah, I feel like this is a thing that, like, has been sitting in the back of my head. And I didn't really, until you've just said it, Sean, and then I heard it on this other podcast,

this, yeah, like, turn things into deterministic scripts. Like, it's so much better. Yes. So I saw this great talk at this, at Goson, Paris. It was actually this presentation of this technical report. and they presented this idea that there's these three different software paradigms now. There's traditional software, which has always existed. That's where the reusable form of that is the source code.

And it's deterministic, but not flexible. And then you have what they called instant software, which is where you're just prompting the AI, right? And so that is flexible, but it's not deterministic. And the thing that everybody's now starting to do is what they call elastic software, which is somewhere in the middle where you have a combination of source code and scripts and also skills. And that is the benefit that it is somewhat deterministic because parts of it are deterministic, but it's also flexible.

And the example I keep giving is when I started doing all my auto research work, I created a skill to do an export from a particular VM that I was using. And like 90% of the time, it would do everything I wanted and it would export it in the schema that I wanted. But like, you know, five or 10% of the time, it would do things a little bit different or it would call them, it would give slightly different file names, you know, because just it decided the random noise and the model response decided to do something a little bit different than it did the other, the rest of the time. And so what I did, I could just write a script instead, but if I just wrote a script to do the export, then, well, what if the node was, you know,

out of disk space, or what if it couldn't, you know, reach the place where it was supposed to export the data to? Or what if there was some file that the script didn't back up because it didn't know about, but was actually really important to back up? So if you instead have your deterministic export script, and then you have a skill that wraps it that drives the script and also goes and checks whether there's anything that was missing or something, then you sort of get the best of both the world where you get the flexibility and you get the determinism. It's definitely an unlock to get to the point where you can say, okay, you know, we're doing this repeatedly.

So it seems like we can just write this as a script now. Do that. I was also going to say, you know, around the process standpoint, like I've got this thing. I posted in Slack the other day where I just saw it scrolling by in the in cloud code and the noise. And it was like, oh, I gave this task to A. agent one and agent one claims success but their their description of their of what they did makes no sense so I looked like so so I looked and they you know agent

one didn't commit any code to get and and yet claims they completed what they were doing and it kind of goes on like this and it comes back and goes like like okay agent one is just completely hallucinating I'm killing agent one and assigning this task to a new agent and I like I And, you know, if you're using these AI systems as kind of a one-off shot, any kind of harnessing in place to have the agents reviewing what they're doing, you hit these states where you're like, like, oh, okay, well, you know, 90% of the time it's doing what I want, but 10% of the time is just so far off in the weeds, I'm lost, and it's, I kind of refer to it as,

it's double-check bookkeeping, right, right? Why do accountants keep, you know, everything is a, is a, a credit and a debit and basically you keep, you know, two tables going. And that's because if you make a mistake in one, you will see there's a contradiction. I didn't get the same result. So therefore, there's a mistake somewhere in there. And in what you're building, you want to make sure that your process is set up so that you're not relying on any piece of your system, just getting it right with no double check. Right. I think we're going to, we're going to see this more and more of these kind of like live self-healing systems that are some percentage of like,

you know, 98% deterministic and it's just, it's just running, you know, but every once in a while there's some crash or some thing or some outage. I'm not sure if either of you run anything off of GitHub pages, but GitHub pages has become, or just GitHub in general, has become like so much more unreliable because everyone's doing what I'm doing, which is like it's free compute, right? Like GitHub actions, I never knew how to write GitHub action code or I don't even know the language. I think it's just YAML. But like you can set up cron jobs and a ton of stuff. But like now because everyone's doing it, their servers are falling over.

And so like once every four days, like my cron jobs that are running on GitHub actions like fail. And I'm like, okay, like we push the servers over again. But it's like at some point they're going to set it up so that, you know, the same way that Amazon and AWS like figured out elastic compute and they scaled stuff up. like it's going to be the same thing with these software like whenever there's an issue things automatically get like spun out and they're like what's the issue and they're going to diagnose it instantly and not only are they going to fix it but they're going to like harden the system of like okay well what what went wrong here and like i have this like i know it's a utopic you know maybe it's extropian you know look at like the future of software but like even

and i i swear to god i must be projecting but there was this small bug in youtube where i have the a little speed up and the shortcut is shift, you know, comma and shift period to like slow down and up the speed of your video. But if you ever hit pause and then play again, it resets to one, which is like not the end of the world, but it's kind of irritating because it's like shift and then usually four more hits to get to 2.0 times speed, which is basically how I watch all my YouTube videos. And like somehow, it's been broken for months and it's been fixed. And I would like to think that that is like, you know, people are like, where, where's all the

benefits of AI. I'd like to think that every time something gets slowly fixed now, it's AI that's not humans. It's AI that's like going to slowly make software. That's my utopic view of the future is all these issues where we're turning off, you're trying to get Chromecast work. Oh, God, help us all. You know, Chromecast,

you know, the most finicky piece. I just want the TV to play on the TV, you know? It's just crazy. You know, a world that that would work. And I imagine in the future we're just going to click one button, it's going to work. And people are going to be like, wow, remember the days when like we had to get Connor and he spent 20 minutes and he just said, you know what, screw it.

We're not going to watch Disney Plus. We can't figure it out to get it to work. I guess we'll just watch Netflix again because admittedly, Netflix does seem to work much better than all the other alternatives. Anyways, yeah, that's my utopic vision, is that all the problems that humans have baked in to all these software code bases are slowly going to, like, heal over time. It's just a matter of, you got these.

The agents, as we've been discussing all episode, they're just human. And so thus. Why would we, why should we suspect that we will live in a perfect software world free? It's exactly what Sean just described, though. It's like you get one to monitor the other one, and they're doing it 24-7. It's not, you know, one person's working, and then they asynchronously check a GitHub issue. It's just like. I know I've tried using this.

I've had it end up in in loops where they're monitoring each other, and they never agree on success. And I've seen this fall down, too. What are your thoughts, John? What are your thoughts? Do you think it's possible or is Bryce, is there always going to be a flaw in the system? I think, you know, systems like this at scale, you're always going to find either like the specifications are self-contradictory at some point or the system just becomes chaotic at some point. And, you know, I think LLMs will have a better, at some point we'll do a better job at doing things like detecting those cases. but whether or not they can fix them is an interesting problem.

But I tend to agree with you, which is the system is much more repeatable and scalable with the LLMs, right? I can write down the rules and it can go do the thing and it can analyze way more software and way more quickly than people can. And the challenge there is I think right now most people, the way they're using AI is they're creating the same crappy code that they would have written, like most people in the industry, faster. The problem is expanding right now. But I also share your utopian vision of that can be fixed, that, you know, cloud code or equivalent can ship with a solid engineering process baked in.

It's got a superpower skill, which is probably the closest thing that Anthropics ships to an engineering process. But like I said, I think we're in, we're in 1.0 territory on these kinds of systems. And so I expect at the right things are going in a year from now, we'll be having the conversation about, you know, whatever, cloud code, software engineer app, and how amazing is that? And like I said, I think we'll get to the point as an industry, we'll start to be able to measure the effectiveness of process, which is a very expensive and difficult thing to do with

people. Yeah. Right. And so that will get to the point where the system can iterate much faster and potentially start iterating on its own, right? It can run its own experiments. Yeah. That's, I think, the next frontier, the recursive systems or the auto-rearchive systems or the

auto-rearche, I think that's what everybody's starting to make a big shift to. So, yeah. In my story about the agent hallucinating and the other agent catching it, I just had this thought in the back of my head, which is so how much longer until the LLM, what I see scroll by is, is, you know, I ask this person to go get me, you know, this additional resource that I need to keep running. And they didn't come back.

So when they do come back, I'm going to kill them and assign this task to somebody else. you definitely like that's yeah i definitely probably not the killing part hopefully not you know fingers crossed we get the alignment problem figured out but definitely like i i don't know i i live at the whatever the edge of this tech and so my wife always reminds me you know most people aren't seeing the things you're seeing they're not thinking but just like i think like you know self-driving is better than humans and just like so much of this stuff is going to become like these AIs, they're just better. You know, it's just safer.

You know, they're better at writing code with like, even if maybe on the first shot, it has as much bugs, as many bugs. It's able to, you know, fix them and iron them out a lot faster. And, and yeah, it's a very exciting future. We'll keep our fingers crossed. We know that you have to go in like 10 minutes or so. Maybe on a, because we're probably, definitely,

going to release this might be our longest episode ever as episode 300. might not be as edited as heavily, but, you know, it's a trade-off. You know, the listener gets a almost two-hour podcast with Sean Parent. And maybe as a final question, you know, for the younger listeners out there, of which I know we have quite a few that are potentially either in school or, you know, about to leave school, you know, we talked about Dave and how he's retired and now he's focusing on music. And we talked about how you've still got lots of exciting stuff to work on.

But by the time, you know, the 20 years rolls around. and AI is better than everybody at everything. You're also going to retire. And you mentioned a few other folks. I happen to know a guy at Nvidia that said if this is really what engineering is going to become, he has no interest in it. And I won't call the individual out, but said that he's got multiple children, one of which is in music school.

And he thinks that it's possible that that is the big, what do you call it? Like the best outlook career-wise is for that student, where the other ones are in a more traditional STEM. I don't want to give away too much information. but that listener knows we're talking about them right now. What is the either advice? If you were in school right now watching this, like are you just panicking?

Like what would you now tell a younger version of you that isn't going to be retiring in the next 20 years or at least not forcibly, I hope. I hope there will be still jobs for some people. What would you, what would your advice or wisdom be? Yeah, in the crazy times that we're currently living through and we will continue to live through. Yeah.

You know, my advice is always, the same for any student, which is the primary thing you can do in school is learn how to learn. And so that you can adapt to change. And change right now is coming more rapidly than never before. And so that's the key thing, right? Keep reading, learn the fundamentals, get a broad education. Don't just specialize in one thing, right? I spend a little time in the philosophy department in the school of arts and not just in your STEM career.

So that would be my advice. And then the other thing would be to start thinking about society and policy and get involved on that end. You know, the Industrial Revolution was a huge change from a societal end and a standpoint. And I think AI is going to be a bigger change. And I think just saying, you know, I don't want it, I don't like it, is not a productive stance. So getting involved with what are meaningful policies for how this can be rolled out and start to think about what is the role of people in society where we are no longer the most intelligent thing on the planet, you know, build the future the way you want to see it. You know, right now, I think our politicians and leaders are woefully ill-equipped. I don't think they have any idea of what's coming.

I think we just have a small glimpse of what's coming. Damn. Any books? I mean, now, I love to ask. I've been getting most of my book recommendations from Boris Churney because he's read all the sci-fi of like, you know, that deals with these potential utopic or dystopic futures. or even also too, I would love to read a book that, like, details the industrial revolution from, like, different points of view because there have been these shifts in society, but I'm not a historian. And also, I haven't been a big reader of history other than, like, a few biographies here and there.

So I'm not sure. What would you book recommendations as well for people that want to get started on that journey? Boy, this is horrible because everything I've been reading lately has been papers. I don't know. I don't have a good book recommendation. but I will think about that. So next time he asks me, next time you have me on the podcast

already make a good book recommendation. Okay, we will. Maybe if you, because this will go out on Friday, if you think about them before Friday, we can put them in the show notes. Because I am, there seems to be like a lack of book.

Like there's a ton of sci-fi books, but there's not a ton that, like, deal with what's going on because this has happened so quickly. And like, obviously, AI comes up time and time again, but not, not, probably there are a bunch of books out there. there. I'm less well-read than I should be probably at my age, more well-read than the average

person, but that still doesn't say much because there's a lot of people in society. Bryce, do you have book recommendations? I know you've read the Asimov books. Yeah, that's been the last thing I've read recently. I've been so, I've been so busy with everything that's happening right now. I haven't read any books in a while. Yeah. I could recommend one, but it's like, it's like 5% an erotica novel, which is not great. Well, nice. Now you got to recommend it. But it's monetized if it's accelerando. I'm in the middle of three books, one parenting book, one like, I don't know, self-help,

you know, better mental health book. And then this book that's like AI, it's Accelerondo by Neil Strauss, which is on a lot of these lists. If you ask AI, like, oh, I read, you know, this book, that book, this book, what next? It almost always gets recommended. The AI did not warn me, though, which is not that I'm against reading about erotica or like, you know, what do you call it, like 50 Shades of Great kind of stuff. But I would just, you know, every time it starts during this book.

Because I'm just trying to consume all of the novels about like, you know, AI take off and, you know, like cloud uploading your brain and stuff. Trying to get ready, you know. I feel like that's the closest thing you can get to an education in the future of AI because, you know, there's nothing out there really that is like addressing this exact moment. All right. Should we let you go, Sean? Is there any last things you want to say?

Thank you so much for taking two hours of your day. I mean, this has been fantastic. I mean, there's so many times you were talking. And then I was like, oh, that's the cold open. I was like, oh, my God, that's the cold open. Oh, my, how am I going to get? I'm going to have like four different cold opens for this show. We got to say something about episode 300.

Yeah. So, like, congrats on hitting 300 on the podcast here. This is great. And thanks for having me on for the 300th episode. How many years has this podcast been around? Is it like six? Well, 52 weeks in a year. So it's coming up on six years in November.

We're so old. Yeah, the key is the key of consistency is really chunking these up into smaller things because it's so much easier to record once and get like three episodes than it is to like what CTP guests did for that many years in a row. Every single episode, like recording was a single episode. Although, you know, I have. been thinking about like future plans if coding's not a thing and maybe full time talking to people is not a bad you know i don't make any money from this right now but i could pivot and try to and uh you know like i said connecting with people it's i would love to do this on a more regular

basis and the problem is is i don't like ads so you got to find a different way and i don't want it really to charge people for this stuff so if you don't like ads and you don't like charging people it's kind of hard to make a living off. Yep. All right, well, we will let you go. Thank you so much, Sean. We will do this again. And also, too, are you wearing the CPVCon shirt? Are you going to conferences or speaking? Is there the last, if people want to find you places? Is there places they can do that? No current plans. You know, invite me right somewhere. All right. You heard it conference organizers. Invite, Sean. He'll be there.

And if we hear from you, we'll announce it to make sure that wherever you're going, ADSP listeners can know where to find you. Be sure to check these show notes, either in your podcast app or at ADSP thepodcast.com for links to anything we mentioned in today's episode, as well as a link to a get-up discussion where you can leave thoughts, comments, and questions. Thanks for listening. We hope you enjoyed and have a great day. Low quality, high quantity.

That is the tagline of our podcast. It's not the tagline. Our tagline is chaos with sprinkles of information.