The Pragmatic Engineer - Why performant code matters (but gets widely ignored), with Casey Muratori

Episode Date: August 26, 2026

Brought to You By:• Antithesis – verify your system’s correctness without human review or traditional integration tests – and avoid bugs or outages.• Sentry – application monitoring software considered “not bad” by millions of developers.• turbopuffer – a vector and full-text search engine built on object storage. It’s fast, cheap, and extremely scalable.—There can be few people around who care about software performance more than today’s pod guest, Casey Muratori. He’s a programmer and videogame developer, founder of Molly Rocket, and creator of  Handmade Hero – a long-running series about building a game from scratch. He also evangelizes about performance on his Substack, Computer, Enhance.We got to know each other about three years ago, first via messages, including this one from Casey:“Why does the industry zeitgeist place so little emphasis on software performance when there seems to be overwhelming evidence that performance is critical to their bottom line?Like you, I run a Substack for professional programmers, but I focus exclusively on software performance. Although we are quite large by Substack standards, so a certain subset of programmers must believe performance is important, I nonetheless hear lots of dismissive excuses when I post on social media. This happens so frequently, I devoted an entire article to cataloging the extensive pro-performance evidence we already have from the world's leading software companies: Performance Excuses Debunked.Strangely, nobody has a rebuttal to why performance is important. When I point people to this, they actually tend to agree. But the prevailing attitude nonetheless stays the same.”I’m delighted we finally have Casey on the podcast because it’s overdue! In this episode, we discuss why software performance matters, why it’s overlooked, and how developers can get better at writing performant code. We explore why performance should be considered during design, the value of learning to read assembly & understanding how CPUs work, Casey’s critique of ‘clean code’, and why he believes testing shouldn't drive software design.We touch on how videogame development has changed, and influential game engines. Casey also tells us why he prefers to write code by hand, not with AI, and more.—Timestamps00:00 Intro05:17 Games at Microsoft12:52 Building games16:00 Why performance matters27:12 Why you should learn to read assembly30:36 Designing for optimization42:51 How to get better at writing performant software49:04 Understanding how the CPU works55:53 Building games then and now1:05:56 How game engines changed building games1:10:48 Why new games compete with old games1:13:25 GTA 6: why is it taking so long?1:16:59 Casey’s critique of clean code1:21:48 Casey’s take on TDD1:24:30 What is good code?1:27:32 What makes a good software engineer?1:33:56 Why Casey doesn’t code with AI1:39:01 AI’s impact on the game industry1:44:43 AI and burnout1:50:21 Why you should read papers—The Pragmatic Engineer deepdives relevant for this episode:•Pushing software engineering limits with “napkin math” with Simon Eskildsen •How Games Typically Get Built: prototyping, game engines, and a different type of QA•Game Development Basics: deepdive on how game studios differ from standard software teams•Inside Linear's Engineering Culture: building a performant product with a tiny team•Building a best-selling game with a tiny team – with Jonas Tyroller. A two-person team built a game that sold 1M+ copiesMore on premature optimization: read or watch Casey’s extended take on “premature optimization is the root of all evil”: https://www.computerenhance.com/p/theroot—Production and marketing by ⁠⁠⁠⁠⁠⁠⁠⁠https://penname.co/⁠⁠⁠⁠⁠⁠⁠⁠. For inquiries about sponsoring the podcast, email [e-mail]. Get full access to The Pragmatic Engineer at newsletter.pragmaticengineer.com/subscribe

Why do most devs not care about writing performance software? And should we? Today's guest, Casey Moratorium, arguing that we should. He also says that most software out there runs tens to 100 times slower than it needs to. Today we discuss why the focus on performance took a backseat across the industry and why Casey thinks the tide is finally turning.

Why you'll want to learn reading assembly if you're serious about high performance code and why it's less scary than it sounds? The saying premature optimization is the root of all evil, why Casey says that the majority of people use it to avoid thinking about performance when they really should, and many more. If you want to get better at writing faster software and become a better engineer while doing so, then this episode is for you. And if you're one of the people who hit a like on this post by Ryan asking for this episode

to be more about programming than about AI, this episode is also for you. This episode is presented by anticis. If you work with agents, your job is no longer just writing code. It's also specifying and testing it. An antistesis is the most effective method of verifying agentic code today. This episode is brought to you by Century. You probably already know what Sentry is because you're a developer, if not just ask a dev, and they'll tell you.

I use Cingertr to monitor the backend of the pragmatic engine for any and all events and errors. Of course, Sentry doesn't only do errors. They also have logs, replay, spans, profiles, metrics, and more, because they're all connected to the same trace. One new capability Sentry has I'm really liking is its ability to fix errors. Let me show you. Here's a list of errors on my admin backend. There's a recent error on auth that I want to check out.

Let's have CIR run an Autofix for us. Sear is Sentry's AI debugging tool. First, it generates a root cost analysis. It's finding some problem with HTTP versus HTTP URLs. Cool. Now that we know what's going wrong, Seer can create a plan on how to go about fixing it. I could go and edit this plan, but I'm happy with it,

so let's create the actual code fix. Here's a code fix that Seer generated. Assuming it looks good, and in my case it does, let's draft the pull request. And boom, the PR is created, ready to merge. What I love about Autofix is how since you went from showing a list of errors in some application to offering me a fast way to fix it

and close the loop while I stay in charge of this buckfix the whole time. Debugging got a whole lot faster and a whole lot easier. Check out Century at century.comio slash pragmatic and start detecting errors, diagnosing your root causes and fixing issues and regressions today. All right, Casey, welcome to the podcast. It's so nice to have you here.

Thank you so much. It's great to be here. Thank you for the invitation. Now, I want to go back when we start to the beginning. How did you get into tech? programming computers. Well, I guess computers, it's like very, very early on. My dad was a programmer at Digital Equipment Corporation, which is a company that people will know if they studied computer history, but would not know if you just looked at the landscape today. They're completely gone,

right? They got absorbed partly by Intel, partly by Compaq, I think. There was, you know, they kind of got broken up. At that time, it was kind of a really big computer manufacturer, you know, computers like the PDP 11. That's a digital equipment corporation computer. The Vax, like things that you may have heard of in computer history. Oh, these were these massive mainframes. Yeah, mini computers as well. So like smaller also sometimes than mainframes, like the kind of next step down, right?

And so in general, that era, my dad was a programmer there. He would later end up at Intel because, you know, like I said, parts got acquired. He never actually left his job. He just ended up at Intel through kind of digital's eventual demise. But as a result, we always had computers at home, even though at that time, you know, that might have been a little bit odd. You know, I learned a program when I was seven, which would have been in like, you know, 1982 or something like that. And so at that time, you know, maybe you might have an Apple or Commodore kind of computer at home, like some kind of early computer. I don't remember the exact dates of those computers, but most people didn't.

And it was only until a little bit later that you would. And you probably wouldn't have had a programmer in your household to teach you, more importantly, right? So I learned really early on, and that's when I got into computers. How I got into games was I ended up randomly interning at Microsoft, and I met people there. And like I went and, you know, sort of went off into games through them. That was how that happened, if that makes sense. But wait, how is the Microsoft slash games relationship? That's not kind of a given, right?

Microsoft is not. They have one game, right? It's flight simulator. Yes, at that time, it would have been very weird. The reason that happened was a guy called Chris Hecker, who a lot of people don't really know his history because there was sort of two waves of bringing games to Microsoft Windows. Because, I mean, it's funny to think about now because people think of like, what platform are you going to play games on if you have a PC? Windows is like the default. Linux is now an insurgent to that.

but Windows is the default. And so you think about how did that happen? You know, because that wasn't the case if you were back in the early days, Microsoft Windows, not a gaming platform. It's almost nothing on it. It's like Solitaire and Minesweeper and a few other sort of games. How didn't happen? So what happened, the reason for this,

this kind of gets into a technological reason. The reason for this is that it was very hard to actually produce images that could be displayed on the screen quickly. And to understand why this is is like its own kind of topic, but in general, you can just imagine you have this operating system running, which is Microsoft Windows. It's controlling the graphics card. It's often running at a fairly high resolution compared to what a game might want to run at.

You have to negotiate with it in order to display your bitmap in some way that won't destroy all the things that it's trying to display, so and so forth. early versions of Windows up through like Windows Windows for Work Groups, let's say, if anyone remembers that name. Was that after 3.1?

It's 3.51, I think, or maybe just 3-1. There's NT351, no. So it's like, I think you're right, 3.1, 3.1, I don't know, something like this. Yeah. Windows for Workgroups was, you know, around that time. That version of Windows,

which is in the 3-series, didn't really have a way to quickly use the CPU to fill pixels which is what games need to do, right? There's no GPU acceleration really at this point. There's a little bit we could talk about, but it's not mainstream. It's not in consumer. So they need to be able to do this sort of thing.

They need to do it in a double-buffered way so they can draw to a back buffer and then show it to the screen. And that has to happen very quickly. And there just wasn't a way to do this in Windows. In Windows, you had to kind of go through this API where you would produce sort of a bitmap that wasn't necessarily in the right format

for the display you were using and then it had to do a translation from that bitmap to the other one when it displayed. played it, all this sort of stuff. So games on Windows, like, things like Doom, they're not coming to Windows, right? That kind of future was not in the works for Windows. And that was kind of what some people in Microsoft wanted to change. And one of the people who brought this change about was a guy named Chris Hecker.

And he was like, okay, we could actually just make a library that did the fast blitz to the screen so that we could have a way that people could do these draws and get. get them on the screen quick enough to make gaming viable. Would it be 100% as fast as DOS? Probably not, but could you actually run, you know, some of these new games? You know, Wolfenstein 3D, I think, would have been out at this time. Doom was kind of on the horizon and that sort of thing, right? Yeah.

And so he started this project that he was not supposed to do. He did not have the authority to do this called Win G, which I'd at least stand for like Win Graphics or something like that. Total Skunk Works project. He had cover from his manager. I'm assuming I can say all this stuff now because it's ancient history. He had cover from his manager's guy's name was Michael Edwards. And Michael Edwards basically just kind of ran cover for this,

which is a thing that probably wasn't going to fly because they were in a division at that time, which would have been Microsoft Research today. It was called AT Advanced Technology. It was the early version of Microsoft Research. You were not supposed to be shipping core libraries for Windows. Like it had nothing to do with that. Long story long, what ended up happening is that product did ship, WinG, I guess you wouldn't call it a product.

It's an add-on for Windows. It did ship, and it was the first step towards DirectX. People forget that. Wing-G was the first way you did this. And then eventually we had DirectX and Dibsections and Win 95 and all that sort of stuff. When I went to Microsoft, amusingly, the person I was supposed to be reporting to as an intern was Michael Edwards. When I showed up, first day, along with the other interns, a guy named Rudy, a guy named Rajiv, We were all supposed to report to him, right?

Because you get a couple interns under one, you know, sort of manager. We show up and we're just taken aside, you know, after some kind of, you know, stupid HR orientation thing that was, you know, lame as it always is. And we get taken aside by somebody. I don't remember the lyrics. They're like, look, a bit of a problem. The person you interviewed with and were supposed to, like, report to, they aren't here. Like, I don't remember exactly how they put it. So you're going to go talk to this other guy

and they'll find something for you to do. Turns out the wind G thing had blown up the previous week. And Mike Edwards had stormed out of the building and had not been seen since. That is what actually happened. So I arrive at Microsoft, the person I was interested in with. He just flamed out and left. No one knows when he'll be back.

He did end up coming back like a week after I got there. But he was kind of moved over to a different division. You know, there was a bunch of like triage work done there. to throw what's going on. So that was my experience. But suffice to say, it meant that I was right in sort of like ground zero of games on Windows. So I met Chris Hecker.

I got to talk to a bunch of people there. He actually took me out to see one of my childhood heroes, Ron Gilbert, who's the guy who did the N-Sum engine. Yeah. He knew all these guys because working on Wing-G, he had gone out to see a bunch of game developers and, like, work with them and this sort of thing.

So I got to go to Humongous Entertainment, which was Ron's new company. He gave me a secret of mine. Lucky Island mouse pad, I remember. So anyway, it was really cool, and that's how I ended up getting to the game industry, was through Chris Hecker. And he's kind of an unsung hero of getting games on Windows because, you know, he just, he wasn't out there making his story known, but, you know, I am now, I guess.

But it's so interesting to hear these stories. Obviously, now you can share it, I'm sure, you know, like for a while, this would have been like only within the inner circle. But the fact that, you know, of course, Direxx was a huge success. And it did, like, as far as, you know, from my vantage point, huge reason why games were big on Windows. But now here's someone who just ignored, wasn't asking anything, just was doing something, got into conflict, fights,

and just, like, pushed an idea. It's more than you think, because there were three people who really were the core people who pushed direct X, meaning institutionally pushed it. There are tons of programmers like Todd Laney who did, you know, really important core work. It never would have shipped without people like, him. So not on the progress side. I'm talking about institutional side. It's Isler, Engstrom,

and Alex St. John. I don't remember which one is either Isler or Engstrom was the tester on Wingy. So he came from that team. So the start of DirectX, one of the core members of DirectX was on the Wingy team. So it's a direct lineage. It's not even like an unrelated push. So Wingy really was the start of it. And then DirectX was kind of the actual. full blossoming into an org with Microsoft's blessing at that point that actually became powerful internally. And then after

this Microsoft Pinterest, I've been getting exposed to all these folks, you actually went and build games tooling. You then started your own a studio as well, right? So like, how did that sequence? I guess there's a couple of steps in there. I worked at a startup with Chris Hecker that didn't end up doing

anything interesting. Then I went to a company called Gas Powered Games, which was actually a Microsoft I guess they're not a Microsoft studio, but they're publishing deal with this Microsoft. And they did a game called Dungeon Siege, which is kind of a, you know, I don't know, it's not

a particularly well-known title. From there, I went to Rad Game Tools, and that's where I stayed for quite some time. I did their character animation system. That was a very popular product that ended up getting used in lots, lots of games. It's still used to this day, much to my surprise, because that's a very, very, I haven't worked on it

since 2004. But I guess other people had main it and some studios just kind of integrated into their you could get source licenses so I guess some studios just integrated it into their pipelines that have never removed it and they just maybe keep updating it to to keep it working the way that they want so anyway that's what I did there and then afterwards I've been independent since then I just have a company called Malay Rocket where we do various stuff I've done you know contract work for for people through that we now do like the substack through that where we do educational materials so I've kind of just done

stuff since then. Although I have done some work on games, I noticed your checklist. You're talking about The Witness. Obviously, that one was an actual specific title that I worked on. But that was, that was mostly just because it was a very big project. And I was, you know, I'm friends with John, so I was just trying to do some helpful programming on the side. I did some stuff on how the movement system worked. I thought there were some interesting problems that we could solve there for games. And so that was a really fun project to work on. Yeah. And then today, you're, you're doing educational stuff on a bunch of sub programming performance on your substack. And what else are you busy with?

So we do actually have an unannounced project that we've been working on. Uses up sort of the rest of the time that I have, if that makes sense, which is not always so much. And that we will, we're hoping to announce it sometime soon. But it's not quite out yet. Believe me, I will, you will hear, but I will send you as soon as we have at actual announcement. But given the fact that it is kind of like a split time. sort of thing for us because we're pretty focused on making sure the substack is good and all that

sort of stuff. We're trying to keep it fairly tight-lipped until we actually know we're mostly done because we don't know how much time we can always devote to it if that makes sense. Yeah, no, it's pretty typical games related things, right? Like tight-lipped until you have something for good reasons. Well, sometimes people play the other game. They go like, look, we're going to day one, we're going to be very loud about this and try to build a community around the development of the game and all that sort of stuff. And that's great. So, you know, that's another route you can go. But if it's not your full-time thing, if you have, you know, other responsibilities, that doesn't seem great, right? Because you don't have any insight

into how much time you will actually be able to devote to it, right? Yeah. And on substack, it's called computer enhance. And you started it with performance related topics. And that's how we started to talk, I think, about three years ago, when you already had a substack. And we had a direct message conversation. I remember you messaged me saying, hey, Gerge, why do you think in the industry people, software developers, just don't really focus on performance? I think you specifically wrote...

I did. You were saying how there's little emphasis on performance, even though there seems to be overwhelming evidence that performance is critical to the bottom line of most software. And I wanted to ask you, we've had a good back and forth on this. And actually, I think initially I told you, like, oh, here's why you don't need to care about performance when you're building. I don't distribute the systems or like, but since then, what have you learned?

Why do most developers don't care, or even most companies, teams, engineering team, not care about performance all that much? It's a really good question. And I think, I do think your answer at the time, if I remember it correctly, is certainly an accurate one for some subset of industries, which, you know, you said something along the lines of, look, a lot of these pieces of software that you're seeing, the user isn't the purchaser, right? Like, you were, you were like, this is some kind of thing where, you know, somebody very high up is going to look and say, we need software for managing HR. They're going to

look at the cost of the software. They're going to look at the compliance terms of the software, the legal liability, whatever, right? And then they're just going to make a purchase decision on that sheet. They're not in there looking to see whether it takes like, you know, whether there's a 30 second pause every time you want to try and access somebody's record, right? And I think that's very true. Like, unfortunately, the situation for a lot of enterprise software probably is that way. So maybe an individual might well be upset about the performance of that software. And I certainly hear from people all the time who are upset about the software that they use. They might not be any position to change it.

I think that's one thing. There's thing, too, which is that in a lot of cases, you simply have monopoly effects. You know, people aren't right now realistically going to challenge the social networks. that currently exist, for example. People have tried. It's very hard. You know, Blue Sky and threads have tried to assail X. And, you know, you've got Facebook and Instagram and TikTok and they kind of just own those spaces, right?

And it's very hard to push into those because of these like network effects. And maybe performance could be part of a package where you try to take on one of those players. Like, hey, look at how much more responsive our thing is than theirs. might be a nice plus, but that's not going to be sufficient. If you just show up with no plan for how you get adoption, no plan how you get big influences over there, all that sort of stuff, it can't sell a product on its own into a monopoly space, right?

If you're just talking about apps that someone can choose to download, maybe you've got a shot there, but those are just, they're forming a smaller and smaller subset of what software is, right? And this bigger and bigger subset is like these monopoly platforms you go on to sort of work with. So I'd say that's another thing. Thing number three is I think now people sort of are caring about performance more.

I think over the past decade, the people including myself, but many, many other people who have been saying that this is a problem, have actually had some effect. Like, I don't think that it was a waste of time. I'm seeing a lot of new emphasis on performance, people talking about performance, people posting benchmarks on things. And so I actually think that the third thing is, well, actually, it kind of does seem that pointing at this issue, in saying this is something we should be doing better has not been completely a waste of time. I do see things as sort of starting to turn around a little bit. I also see people attacking major product categories now with performance-based pitches, things like file pilot or the blick video editor, like things like this that have been coming

out lately where it's like, oh, really performance software to try to take on incumbents in a space and they've been getting traction. So I think that's also a really good sign. Yeah, and I guess on this last category, a really good example, in the developer community is Bunn versus NPM, where Bun just said, like, okay, we're like 10x or 20x or 50x faster. And devs are like, what is possible? And then it was.

So there was this outrageous claim. I wonder if, like, you need to have these outrageous claims because dev starts to pay attention because it was 10x faster in many categories. You know, linear versus Jira is also a good example, where linear has. have this benchmark of, okay, they have 300 milliseconds for any action. And Jara, of course, we know is just slow because they have a bunch of complexity. We can explain why, but it's slow. It was never built for that.

Yeah. And, you know, if you think about something like a 300 millisecond budget for an operation, 300 milliseconds is like an eternity in computing, right? And so if you're talking about like our pitches that were not more than 300 milliseconds, that just shows you how the bar was so far. past where it probably should have been for something. And you see it everywhere. You know, you go on to programs and you're waiting sometimes seconds for an operation.

And I don't think people realize just what an eternity a second is in modern computing, especially when you're sitting on networking like that has, you know, sub 10 millisecond ping times sometimes. You're talking about this, you know, the actual packets had to travel physical distance to get to this data center. And that was being done far faster than this very simple operation that you were. failing to do in a reasonable amount of time. It's just like, we are massively underperforming.

People don't believe it when you say 10x, 100x, but it's actually true, and we've seen a lot of proof of it, as you point out. I do wonder if one part of the not really much focus on performances that a lot of developers don't know the baseline thing. And I'm reminded by Simon Erickson, the founder of TurboPuffer, has this project called Napkin Math, where he did a list of mostly networking operations. How long does it take to transfer one byte? between like two ADW's data centers.

How much does a gigabyte take? How much does a terabyte take? How much does it take to write an SSD to an NVME and so on? And so he had these numbers. And he said that what he found is whenever in Shise Shopify, they were deciding, do we choose vendor A or vendor B as a database? They would just run a benchmark that they would write themselves, and they would get like, okay, like, I don't know, storing this and this.

it takes two seconds on this one, 10 seconds on that one, we will choose a two second one. And he looked at it and said, like, hey, like, this doesn't make sense. Like, the amount to store in a foul system, like, here's a theoretical limit, which is, I don't know, 100 milliseconds. Like, there's no way that's going to be 10 seconds. And it often turns out that he found that the benchmark was just wrong. They were benchmarking the wrong thing and they were making decisions. So I wonder if there's a thing where many engineers, developers are maybe just not aware of how truly devastatingly slow this.

thing is versus the resources you have? That is the entire point of like my substack, right? So what you just said is exactly true. And it is the thing that I hammer home on the substack through all the parts of like the courses on there, which is that in general, there's a misconception about the way that you approach optimization in like in computer science or in whatever software engineering, let's say. And that misconception is that what you do is you run a profile, you identify.

where the like, you know, big parts of the profile are, you make some changes to those and you measure like statistics, the better the statistics, you know, the more statistics you can get the better and you look to see if those statistics improved, if they have that was a good change, and you proceed as such. And this is completely not correct. That is not how anyone has ever, you know, I've worked with many extremely good optimization people and that is not how it is done. the correct way to do optimization is very much like what you just said. You first go, what are the operations that this system has to perform? What is the underlying hardware capable of doing at its theoretical peak?

And then you measure the delta between that theoretical maximum and what you have achieved. And then your goal during optimization is to shrink that gap to something that you think could be plausibly explained and hopefully come up with explanations of why you aren't at theoretical. Because oftentimes you can't hit theoretical. That's why we call it theoretical, right? And it's crucial that you do this because otherwise all you're doing with that other method is finding a, you know, with the, I'm just going to, you know, make something I think might be an optimization and look if my statistics improved. All you're doing there is finding a local minima. That's all you're doing.

You're just, you know, you've got this, this shape of your performance and you're finding some little spot and you're sitting in it. That's not optimization. That's improvement. But optimization means to make optimal, right? It means we're going to find what we actually should be able to get this machine to do. And so, you know, that's why I emphasize that approach, because it's the one that I've always seen great optimizers take. That is how they get good performances by knowing what the maximum could be. In addition to that, it also is what lets you become better at optimization.

Because no matter who you are and no matter how much you already know, when you go to tackle an optimization problem, there may well be some things in the, new way that the system is laid out that you don't know about. New things that people have not figured out about modern CPUs, new things that are different about the network back plane, new things that are different about the GPU drivers, who knows, right? And if you

don't have some theoretical maximum to look at and to measure your delta from, you don't know if there's some serious anomaly there. And you would be surprised at how many times we find anomalies like this. Things in CPUs that no one knew about and

we, you know, like, I've literally had them in the course of making the substack. I've been like, what is this thing? And I look into it's like, oh, there's this new renaming, this new rat table thing that Intel chips seem to be able to do. We didn't know about that. And that's like a new thing we have to model when we talk about how to do performance. And so that's the other crucial part of, I guess, what you were calling napkin math. I also call it back of the envelope.

That's the term I've heard used for it. Oftentimes, they're kind of interchangeable, right? Knowing what the theoretical is, is how you learn as well, how you're, how you're you learn about new hardware and new performance options. Interesting. Plus, by doing this, you're just learning. You're becoming better professional. You understand more about given hardware or the inner workings of your computer or software

or software stack or kernel, you know, all the stuff that, I guess, goes way beyond the vanilla programming language. Like, because you can say, I'm an engineer. I'm a software engineer because I know how to use this programming language. But I'd argue you're probably an engineer if you can go down the stack and you have that ability. And you have a good, good understanding of some of it, at least, right? And you can learn the rest. And I would also say that one of the other things that we do in the class is teach how to read assembly language. And people often ask what, like what, like, you know, assembly language,

what would I ever need that for? There's a very good reason for it. And that is that everything else that you might use doesn't tell you anything about what the CPU is actually receiving. You know, if I look at a Java program, if I look at a C program, if I look at Haskell, O'Camel, whatever, right, Rust. All I'm seeing is input to a compiler. I have no idea what the CPU is actually going to be asked to do. If I look at the assembly language output from that compiler, I know exactly what the CPU is being asked to do.

And it's not that hard to be able to learn to read assembly language so that you can see very quickly, is the CPU being asked to do the things that I think it should be asked to do them and in that way, right? You don't have to write it, hardly ever. It's very rare that you have to write assembly language to do anything other than sometimes for test purposes, it's easier to do that. So you don't have to try and convince a compiler to output something. So if you're just testing something, sometimes it helps to be able to write some assembly language.

But if you're just talking about the vast majority of tasks you might do an optimization, writing it, no, reading it essential. And it also unlocks this sort of huge world of possibilities to you because once you know assembly language, you can now do things like read those. CPU diagrams. Like, you know, when they announce a new processor, they put up a little diagram. That diagram tells you stuff like the fastest this thing could do multiplication and stuff like that. It tells you that if you know a semi-language, you can read it right off the chart, right? If you don't know a semi-language, you can that chart and like, I have no way to do what I'm looking at, right? Like, it's just this weird flow chart that doesn't really tell me anything, right? And so one of the really great things about assemblies just unlocks all of this knowledge for you

because it's the, it's the actual input language to the machine and it allows you to figure out how it's operating. Plus, I guess we should add that assembly language is not all that complicated, right? Just by nature. It's a far simple language. Okay, it's harder to read if you've never seen it, but it's in terms of the number of operations, it's so bare bones because that's what assembly is. Like every single higher level language will have way more keywords, structure, whatever, you name it, right, than assembly. Massively more. And especially when you consider the subset that are actually used. If you look at the subset of constructs that you would need to understand to be able to understand, say, just a website from today.

All of the JavaScript libraries, all of the JavaScript syntax, all of the DOM, you know, all of the behavior that's going to go on their CSS. React. CSS, right? All of that. Assembly language, you know, maybe there's 20, 30 instructions you might have to learn total because most assembly, most things in legacy assembly like X-64, most of them are hardly ever output by the compiler.

So you only need to learn a very small subsets as to the ones that you're actually going to be seeing in 90% of the cases. It's so much simpler. And also, when you're looking at performance, you're typically only looking at a very small part, right? You've kind of understood roughly what's going on. You've seen the basics layouts of your program. You've identified what's supposed to be happening. And you're just looking to see like, wait, why is this part, which I don't think should be running this slowly? Why is it running this slowly?

It's just a very small piece you typically end up having to look at as well. So it's really much easier if you can understand. how to center a div as they say. If you can vertically center a div in HTML, then you can probably learn assembly language, I would say. Okay. You're super passionate about performance optimization. You also have really good educational materials,

both free videos, your paid substack, the free parts of it, et cetera. But let me just play devil's advocate. There's this saying that premature optimization is the root of all evil, and we typically use it or I typically use it so many times we're like, oh, should we make this performance? Should we optimize this thing? like, nah, let's not do that. Let's first build it. Let's see if it's good enough for our customers, for ourselves. And if we need to, we can always optimize it. I mean, you know, like it's not

the hardest thing in the world. Okay, maybe not as good as how you mentioned, because maybe I don't really assembly. But, and that's, that's kind of thinking of building, you know, like, SaaS software, building software at big tech. What is your reply to that? Because I, I feel really good that I made a really good argument here. So I guess what I would say is the important part about that. And I guess I'll divorce it a little bit from the saying. I have an entire lecture on that saying, by the way. It's like two hours long.

And I gave it at Better Software Conference this year. I believe the VOD will be coming up. We're linking that with Joan those below. Okay. It'll be like a week or two, I think, till it's up. So it may be right at the same time as this. But so if you want to find out the history of that phrase, you can go look at that.

But I wanted to talk about the idea behind it because I think there's, I don't want to dismiss it entire. because it's not entirely false. And the idea is that, well, I'm just going to delay optimization work. I'm not going to think about that. And then I'm just going to make whatever I'm going to make. And then either myself or maybe I'll just hire some performance person to come in and clean up the mess later. Right.

So here's the positive side of that first. The positive side of that first is for some types of code that will work. If you happen to have written some operation poorly where the optimized version of that operation just looks like someone taking a loop and changing the loop from your really, like, you know, naive version to a really well-optimized version? The typical of, like, I wrote a bubble store, we can later optimize that.

Who knows, right? Anything of that form, okay, maybe we can just do that. So there are certain times where you do in your head want to be doing this, where you want to say, okay, I could go spend a week researching the fastest hash table implementation here. but part of software engineering is being smart enough to know

it won't matter if I do that now or later the architecture around this piece won't have to change I'm quite certain because I understand the problem well enough so it's okay I can defer that to later maybe it's never too slow with the naive one I put in there

and then we don't have to do any work maybe it's too slow later that's okay I just target this one hash table implementation and we'll get as fast as we need right if you're doing that if you're applying that true engineering mentality to it,

you don't have a problem. The problem comes when you don't know if the choice that you're making produces that kind of optimizable hotspot. And I'll give you a very simple example that usually people have had experience with. A very simple example would be we write our entire software thing, like we just whatever this massive thing

that we're imagining doing where we're going to ignore optimization, we sit down and we write it and we use a paradigm where we ask the server for something. We have like some API, you know, that we've built for asking servers for things. We ask the server and it returns to us what the server's response was. And that's like kind of how we architect's thing. So everyone writes, you know, hundreds of thousands or millions of lines of code

and they all look like, ask the server something, do some calculations, ask the server for the next thing, do some calculations, right? Then at the end you find this is way too slow, but that's okay. You weren't worried about that because you're like whenappetize at the end. you call in some performance experts, they look at and they go, there's nothing we can do for you, sorry. Why?

Well, the reason is because you created a serial dependency chain. All of your code looks like, wait for a network request to come back, do something. Wait for a network request to come back, do something, wait for a network. And that serial dependency chain

can't really be shortened without just rewriting it. If instead you had made the paradigm and told your programmers, look, here's what you need to do. At the top of every operation, you need to figure out all the things you might want to ask the server for. You ask them for all of those things, right?

And then you do all of your processing there. And you only create a chain of dependencies if you absolutely couldn't have determined what it was you needed to ask a server for. Now you're just in this situation because you didn't tell them to do that. You have to rewrite all your code. Everyone is now going out, rewriting all the code, if they even can. If it's even possible to really do that in a way that's not slower than just rewriting the thing, right?

So what happened there? Well, again, we talk about this a lot on the substack, but there's this idea of a serial dependency chain. It's when you stack things in order, right? And the performance of your software is generally determined by the longest serial dependency chain because it's something that cannot be paralyzed. If I have thing A, that then B depends on, that then C depends on that. We cannot shorten that because it has to go in order.

Everything waits for it. We can't multi-thread it because it's dependent. We can't, you know, make it run wide. We can't, you know, amortize the network request, whatever. That kind of thing can be pervasive in the programming, and we can't cheaply remove it because it's not a hotspot. It's a way that you did things. That's the part where that kind of thinking breaks down. If every software engineer knew to watch out for false serial dependency chains, things where they were creating series of dependent operations that could not be optimized away, or other sorts of architectural problems like that that cannot be easily fixed,

then the world would look more like just wait and optimize the hotspot, right? Yeah. So this is the architecture, the planning, right? Like if in that phase you're like, okay, like as this thing grows, like what would get in the way of performance, what would slow it down? Or you can ask all these questions or like different flavors of this questions, you know, or from the other side and so on. Yeah, another way to think of it is because hotspots is the way that people talk about

that. Like, oh, it's going to be hotspot optimization. We just got a few spikes. Someone will come in and clean up those spikes and we're done, right? The way to think about it is your code base will not end up that way by accident in most cases anymore. You have to engineer up front for a hotspot code base that people can then optimize, right? And so that's the crucial takeaway is everybody on your team who is making architectural decisions, those people must know performance. And they must make decisions that will allow the other people downstream of them to use an architectural

which can be optimized later. If you don't do that, you're just rolling the dice. Casey just talks about how engineers making architectural decisions should know about performance. This is also true when choosing your dependencies like which database to use. And this is where I want to mention our season sponsor, Turbo Buffer.

You already know how TurboPuffer is a vector in Fultex search engine. But here's an interesting story from Linear on what happens when you stop thinking of Turbo Buffer as a search engine and start using it as a primitive to reduce latency. As context, Linear is a local first app. So each client keeps a local database, and when that client goes back online, it needs to catch up with what happened and do it fast.

Their biggest work spaces generate around a million sync actions per day. Doing catch-ups by reading from Postgres was getting slow for large reads, so the tail latency got too large. And adding more replicas did not help either. Linear solved the problem cleverly. They started using turbopuffer as a serving index for each client. This is because turbopuffer itself is built on top of inverted indexes. So for every index value, it stores the documents that that value can be found in. The lookup cost for such an index is constant.

So Linear took this structure and had each client's index point to the changes that they needed to sync. As a result, not only did they reduce latency, but they kept it constantly low, no matter how long the change list is sync to the client is. Linear published a blog post about this refactors titled Rebuilding Linear's Delta Sync Readpath. Check it out. I love this story because it shows how important it is to choose the right primitives and how good primitives can improve your system. If you're building systems where you store a lot of data or serve a lot of data,

TurboPuffer can probably speed things up or save on your costs. Learn more at turbopuffer.com slash pragmatic. I'd also like to talk about a presenting sponsor, Anticis. While Casey deliberately does not use AI coding agents for his work, most of us do. And when you work with coding agents, your job is no longer writing code, it's specifying and testing it. Antisyses is the most effective method for verifying a judging code today. Let me explain how it works.

Anticistuses runs your whole system in a hostile simulation. By doing so, it finds every bug before your users do. And because the simulation is fully deterministic, Anticistuses doesn't only find bugs, it gives you a perfect reproduction of every issue. To create such a tool, the Anticistice's team needed to invent new kinds of debugging tools as well. For example, here's what's called the bug probability graph.

The X axis is virtual time and the y-axis probability. As antithesis runs a hostile simulations, it plots timeframes when the bug probability increases, which greatly helps us find in the root-com. cause of the bugs. And Anticisys also has a log visualizer. Vertical lines going down represent events branching off from the same state. And the purple dots are where the debug happens. Antisysis is as good as it gets in being able to ship agent-written code. It's what seems that Jane Street, Fly.I.O. and the SEDC. community used to ship with full confidence.

Head to antithesis.com slash pragmatic to learn more. And with this, let's get back to Casey and how, if you don't design an architecture that can be performance optimized later, you're just rolling the dice. and we've seen so many projects I have an entire video where I go through like look at all these blog posts of people who like say it's Facebook, it's Uber, it's everybody they've got blog posts of we had to rewrite

this whole thing because the performance was bad if it was always hot spots that made your performance bad you'd never have to rewrite the whole thing so we know that that doesn't work anymore why because of the things I just said I was at Uber where I was not making the decision but the teams next to me were and I saw or I kind of understood

why they were making but typically, and right now it's happening with AI companies, oftentimes it's like we chose this technology, which is Python, and it's single-threaded, and it made sense at the time on the web server. But now we're big, and this happened at Uber. It was Python and Node.js, and then they went to go and Java on the back end.

And now with AI companies, it was Python, OpenAI, and Tropic are both going through this right now. They're both either public about it or I've written with Anthropic. They share with me, but I put it out there. they used Python because data scientists or AI or machine learning engineers new Python. They put on a bunch of web servers. They had their API run on it. Initially, they just scales horizontally.

But now they're like, well, if we move over to Rust, right now they're choosing Rust or Go, but I think it's Rust. Well, we can actually have multi-threaded and the same machine can actually handle more connections. So cool. I came across a lot of that because I think that's easy and safe to communicate because it doesn't look bad on you. But you're right. A lot of times, I don't think on enduring blockposts, you'll get the real reason that these companies put out there.

Like, when it's kind of a very kind of, you know, easy to own mistake, or not mistake, but just the decision, which made sense, they'll tell you. But if it's something that was an oversight, you're not really going to get that on a public-facing engineering blog post, except for maybe some startups who are really there. But don't forget, like, a lot of those block posts are going to help someone get promoted or get recognized. and they will always be way more positive, especially when there's a content writer team, which large companies do have. So it's not quite PR, but it's somewhere midway in between. And I mean, yes, and also I would just point out the fact that, like,

the fact that these things are happening, though, is all we really need to know for the signal, right? Because in general, this should not be happening. If the ideas about occupation were true, you'd never have to rewrite something in a language, in a different language unless you just preferred that language. It would just, the story would just be, we rewrote it in this language because we wanted to use this new language.

It would never be, or for Rust, it might be just memory safety. We see those blog posts, right? It's like, why do we write into Rust? It wasn't performance. It was just we wanted the memory safety or something like that. If I'm a software engineer programmer, and I'd like to just get better at writing performance code, I'm interested, you know, maybe after this podcast or,

or looking to some of the things that you did. What is a learning path you would follow outside of your substack where you cover a lot of these things? But what are areas that you think are kind of like, you need to understand these things to get better at writing Performer and Code? I think it's actually very simple and perhaps a little bit counterintuitive. So I'll start with the very good news about learning to write performance software. The good news is that optimization of the kind that we sort of talked about, the hot spot kind,

where it's like somebody's going to go in here, maybe they're going to even rewrite this routine in hand-coded assembly or something crazy like this, right? That's very rarely necessary these days. One of the reasons that you don't see hotspot optimization as a thing that really matters that much anymore. And one of the reasons I advise that architecture and, and not making bad decisions is much more important, is because a lot of libraries already have been optimized for you

that you might use. CPUs are incredibly good at taking bad code and running it quickly and so on. So typically when we're talking about the causes of performance, negative performance, that aren't squeezing every last little thing out of the hardware, but rather just making sure this thing isn't running like a hundred times slower than it should be. Usually it's more just about having an awareness, again,

of what the computer should be able to, do and making sure you're making software architecture choices that allow it to do that. And if you do those things, you will generally be within, you know, 2x or something, which is 50x better than the people who are 100x away, right? So the good news is, in order to write software that's much better than a lot of the software you use today, you don't have to know that much. So what do you have to know? What I argue and what we focus on the substack is I think you just have to go through the

experience once of learning, reading the assembly language, seeing how the CPU works, seeing the difference, seeing why Python is slow, which we show on the subs. One of the first things I show is I walk you through the assembly language necessary to execute A plus B in Python. And it's so vast that, you know, I have to skip most of it. It's massive, right? It's like this huge. And whereas, you know, if you have the equivalent function in C, it's one instruction, add.

Right? So, you know, understanding basic things like that, if you go through, learn to read a summary language, learn to look at some code, learn to do some CPU timings, and you just have that experience, just spend, you know, a month or two of nights or whatever you want, just understanding some performance stuff and going through a few examples where you play with it and you see the difference. And just so I understand, you're saying do this not because, let's say you're doing iOS development or web development to read what React. Like you will not look at the assembly that the React does. But if you do this on a project, you will be able to conceptualize what is likely happening, what the layers are. And you might be able to decide like, do I want this layer or do I want to use, let's say, WebGL, which if you're a React engineer, you probably. haven't touched it, but again, you can skip a bunch of those things and it comes to tradeoffs with maintainability, yada, yada, but that now you will know, like, kind of what you say by

keeping this layer or not keeping it and so on. Do I get that right? Essentially, yes. And like, you know, the simplest example is the Python example. Most people have never internalized the fact that it takes, you know, maybe on the order of 100 more CPU instructions to do an ad in Python than it does to do it in an equivalent language like C for the same piece of text, just A plus B compiled in two different, you know, in two different languages, right? And so just understanding even just that is enough for you to kind to kind of go like, oh, okay, A, now I kind of understand why if I'm using Python, I kind of have to use libraries to do things and those libraries were written in C because it's like if I'm

ever going to do any operations on a large number of things, I can't do it in this language because the amplification factor is so high on each operation that it just kills the performance immediately, whereas these other languages don't have that, right? And so understanding those orders of magnitude and what's actually going on, I think that allows the programmer to know, okay, if I think through what I'm doing right now,

can I afford the super slowdown that I'm going to take? And usually, I don't think you have to be a performance expert to make that decision. You could usually know, like, okay, is this a part of the code that can afford to be a hundred times slower than it should be or not, right? And, you know, most people can, I think, make that decision fairly logically. And if it's not, then now you know, like, oh, okay, if I'm in Python, then what I got to do is either I got to go find a library call that will do these sorts of things and structure around how that library works, or I should maybe get something like Scython or something where I can do compiled stuff inside my Python and make my, uh, code work around calls out to that kind of code. You know, you now have the tools you need up front to make sure that when you write the

program, you've put the parts that needed this and you've structured the code in such a way that you are only paying the 100x on things that you know are very infrequent or happen like only, you know, once per every so often, things like that, right? That's, I think, the biggest thing is just the knowledge. And once you know, you can start to make much better decisions in any language. because it doesn't take you very long. A simple search or asking an AI or whatever is the common practice that you're going to do,

a simple bit of that, once you know what you're asking for, will get you this information back very quickly, right? You just have to know that you should have been thinking about it. Now, as a software, you mentioned it's good to understand how the CPU works. As a software engineer who is not a game developer, I'm not doing low-level stuff, what does that give me? Because for the most part, even in academia or in computer science, you know, there is some level of some basic CPU theory taught. But usually we just kind of, we stop at the code, okay, maybe you look at the assembly, but you really go further than that.

The folks who, you know, you've taught and they learn these things, what do you see them get out of this that they wouldn't otherwise? So you're talking about specifically the knowing the CPU part? Knowing about the CPU, knowing about the details about the CPU, because you mentioned that that's also part of it, right? It's not necessarily just stopping at assembly. So the reason for that is more the other way around. The reason to learn the assembly language is so that you know what the CPU is doing. So it's the CPU part that's actually important.

And the part that's important about it is that the CPU is basically, you can think of it as a little machine whose internal gearings we are not privy to, because for the CPUs that we care about, so, you know, an M-Series CPU in a Mac, a Zencore CPU in a server or in a laptop or an Intel, you know, core series, those sorts of things. These CPUs are not documented at the level

where you're going to be thinking about how each little individual part works. And to that end, it's unclear that you would have time to do so anyway because these are massive, very complicated machines that we're talking about, right? But from a high level, from a more black box perspective, they are machines that we can think of in relatively straightforward ways

once we know kind of what their core instructions are that they tend to execute. And they break down into a couple different categories. There's how does data move into and out of a core. And this is basically how like load store units work, how the cash levels work L1, L2, L3, we have L0 now sometimes, things like this. How does that work?

why? What is the granularity of it? What is the policy? How does the CPU go about actually working with those things? Understanding that part of the machine is crucial because when you're working with a lot of data, the difference can be massive if you structure it in one way versus structuring in another way, right? Again, architectural decisions that have nothing to do with hotspots. They're how all the data is laid out and what the access pattern is, right? Things that are very hard to change sometimes. So that's one part of the machine you want to understand. Another part of the machine you want to understand is how the instructions flow through it. And a lot of people have heard about branch misprediction or things like this. I cache misses. There's words that you might hear,

but you're not really sure what they mean. They're all actually pretty simple to conceptualize. Sometimes they're harder to pin down exactly how they work because branch predictors are, you know, getting more and more complicated and so on. But you can still categorize the behavior of them and understand how code flows through it and when you might care and when you won't. And then finally, there's the execution unit scheduling part which is about knowing what's the raw sort of throughput for any particular type of operation?

Floating point multiplies, integer additions, division, whatever it is that you might want to know, right? Once you learn a little bit of assembly language you understand what it's reading, you understand how it turns those assembly language instructions into micro-operations which it actually does

and how they get distributed through that machine. That flow chart that they put like basically up on the, we've announced the new Zencore, that flow chart, You can look at it and go, I know the performance of this machine roughly, right? Not exactly, because like I said, there's all these little edge cases that if you really want to be a crazy optimizer, which again, I don't really advocate people do.

I don't think it's important that they be like crazy hyper optimizers. If you like to do it great, it's a lot of fun, but it's not the important part. Just look at that CPU. And like, okay, I see what the CPU should be getting in terms of like what I could do with this size data load, that size data load. This is what I could probably get out of it for if I was doing a bunch of like, how to do a bunch of, like math ops on it, you know. And I think that's, that's just something that should be kind of par for the car to start for engineering. You go to school for four years to learn this, there's no reason you can't learn this in a few months. It's not that hard. Yeah, plus I guess just from a

craft management perspective, like, we should know our tools. We should know the machines that we're programming. Obviously, we know that our code will be, if you're doing web, it'll be running on all these different things or if it's a, if it's mobile on all these different phones. But from a conceptual point of view, like, we should be able to know what's going on. So I feel there's a bit of a pride as well. Like, if nothing else, you would learn a bunch of stuff. Like, I know some of it, but I'm now getting motivation to learn more about it. I do think there's a craft and shit angle. I think there's a large number of people who maybe don't feel fulfilled when they, I've certainly heard from lots of people who, when they write something and it's just kind of this amorphous,

high-level thing, they don't get as much satisfaction of it. And then when they learn how they can look more deeply at what's going on, they feel much more satisfied. Even if they didn't change what level they were programming at, they feel much more satisfied that now they know what they're doing, right? And it's like, oh, I see. And I understand why this thing was happening this way and this thing was happening that way. That's very satisfying, right? So there's an aspect of that.

I also want to emphasize another part, which is that it's a percentages game. If we convinced enough library maintainers that this stuff was important and the libraries all get a lot faster, all of us. sudden all the people using the library's code gets a lot faster and so on and so forth. If the API is starting to make it easier to optimize the libraries, because people now thought that through, right? Like, it's infectious. The more people are doing performance, the less people need to do performance, if that makes it. It's kind of paradoxical, right? Well, plus, plus I do think right now, there's still an edge in inches being performant.

Again, you mentioned, but there are categories of software that is just winning by being much faster. And to do that, you need to do this. And if you know, know how to do this, maybe you're going to spot opportunities a software engineer. Maybe right now you're not as happy in your position, but maybe start something or do a side project that turned into a full-time thing and so on and so forth. So like I feel there's like, and worst case, you just learn like net new knowledge, which will probably not be as outdated with AI, which we'll get into later. But this stuff, it feels, it just feels very interesting, right?

Like kind of, it moves your brain. It does. And thankfully, like, it's also not that hard to update your knowledge because, again, you get these presentations that the CPU companies give and they're like, here's the changes we made. So you kind of are aware every time a new thing gets a, you know, there's little tiny things that creep in that you don't that, you know. But again, there's people out there who are running lots of micro benchmarks that you

will find out about. And they often uncover these for you as well, right? So I want to talk about games. You've been, you've built games for like decades at this point. Can you give an overview for those of us who are not. in the games industry, how is a game typically built from the games that you know of, that you've observed or you worked on, especially trying to compare for, you know, like in typical SaaS or a distributed system or something we're building a website or services.

You plan this stuff. You know, we'll do an estimate. We'll build it in a few months or a few weeks. We deploy it. And then we monitor it. And then we keep tweaking it. And then, you know, fast forward five years later, it's now this, like, gigantic thing with microservices. But it keeps evolving, right?

Like, we do this, like, a lot of prototyping thing. For games, it's pretty obvious, right? From the Givian, as we're talking, like, there will be a launch. But can you, when you're inside or when you join Game Studio, like, what would you observe there in terms of what the process is like and how it's different or how it feels weird compared to, like, this, I guess, I don't, traditional software or SaaS software, whatever development that is. So I guess what I would say is, unfortunately, I'm probably the wrong one to ask, because

my knowledge is outdated at this point, because one of the things that has happened to games recently is I feel like they've moved closer. I don't want to necessarily say entirely in development practices, but at least in terms of the nature of the product has changed somewhat dramatically to be more like something like SaaS, where, you know, if you take some of the most popular things that are being in terms of dollars, let's say. So I guess maybe popular is kind of might be hard to say specifically,

but let's just say revenue generating. So if we were to measure the total games industry revenue and you look at what are the largest slices of that, you're seeing things like Fortnite, like Roblox, like Grand Theft Auto 5 online,

etc., etc. Minecraft. These things are starting to look a lot more like an always-on live service kind of we ship incremental features to our customers, kinds of things. And so I would actually say that, you know, I'm the wrong one to ask about what that actually looks like from the inside

because I haven't actually gone and worked at one of those companies. I have friends there so I hear things, but I'm not the right one to like give it an accurate picture of it. But I just would point out from my perspective, the game industry practices look different today than they did when I have a more intimate sort of experience with what we were actually building. But can we talk about it when you were building games, which was, you know, 10 plus years ago, when I understand these were the games where they were built, they were launched, you know,

maybe they got a patch or two, and then they were kind of, you know, the team moved on, they were disbanded. It was a time box thing that was a lot of development, a big launch, and either it was, it went big, huge hit or, you know, a huge failure, right? And the studio goes bankrupt. So how did that work? Because I feel that's a world where, okay, today a bunch of games don't have those constraints,

but it has a bunch of constraints and I'm interested in what worked in those constraints. So in the early days, you didn't have licensable engines. So up until sort of the point, like nowadays, like this is why I say it's a lot different now than it used to be. You know, nowadays you think of web development, like I'm going to go grab like a thing like React

and I'm going to make this thing or whatever. I'm going to go grab an off-the-shelf data. database thing, Postgres or Oracle, or I don't know, like, what would be the thing of choice, right? But by the way, just definitely postgres and not Oracle for most people. Okay, sorry. Postgres. I didn't want to slight anybody there, so I apologize. So, okay, definitely postgres.

Sorry, Oracle. Yeah. So you're going to go use some kind of a variant of one of these off-shelf databases and so on. That's more, like I said, what people might be doing nowadays, too. They'll grab the Unreal engine. They're not going to develop an engine on their own. They'll grab backend server stuff from people. It might even be some Postgres in there, right?

Like, who knows? In the earlier days, none of this stuff existed for games. I actually worked, like I said, in middleware at the time. So I was actually sort of one of the people who was working at the time on maybe changing that a little,

like producing code that would get reused throughout games, which was actually fairly rare. And so back in a day, every game built their own rendering engine, for example. Correct. No way. Yeah, yeah, yeah.

And so really early on, right, if you rewind the clock far enough, then yeah, the degree to which people were reusing code for their thing for like their rendering engine it'd be like because I got some code from like Dave or whatever who was or we were both at Atari and somebody wrote this good routine so we used it right there was that kind of thing

but there wasn't like this sort of set engine and the time when that sort of maybe you could say first started happening a little more widespread was with things like what id software did where they sort of started having like oh, you know, like someone's going to build something with the Doom engine or someone's going to build something with the Quake engine. There was also the build engine at the time made by Ken Silverman and some things like that.

So there were some early cases where a few people would make a game, but they were making a game very much like that. Like if we did the Doom engine, we're going to make a game very much like Doom. So it really was the case that for most games, people were rebuilding most of the things from scratch, at least for their studio. And studios often, their existing code base was kind of, part of the value of the studio, too.

Like if you think Blizzard and we just built Warcraft 1, while rolling all of that knowledge and code into Warcraft 2 is a huge advantage for us. Because everybody else who wants to build a competitor to Warcraft 1 has to do all of that from scratch. You have to make the pathline. You have to make the level editing tools. They have to make the rendering. They have to make whatever.

And so, you know, that was how things were traditionally done. No wonder the games industry is so secretive compared to the the rest of the software engineering industry. Like, seriously. Used to be. Used to be, at least. Yeah. Maybe now it's changing.

And so there were two really big risks, typically, when you started a game project in those days. One was the engine risk. Would we be able to make something that would be technically able to do what we need to do for this game? And that risk comes in a lot of flavors. One, will it happen at all? two, will it happen fast enough for us to actually reliably build the game on it, right?

One of the things, I mean, I don't know how detailed answer you're looking for for this question, so stop me if I'm going down too many tangents. But one of the things you also have to remember is that at that time, and this is sort of still true today, but at that time was very important, there was no way to really buy something all that much faster than what you had. there was not a huge strata of PCs that you could, you know, buy or anything like that. So the rendering engine, there wasn't like a way your level designers could like be playing on a faster thing than the consumer would have really.

You can only have the machine that you have now. And if this game comes out in a year, that's sort of roughly what the consumers might have or a little bit, you know. So there are people who started doing things like buying SGI workstations because those were actually faster enough, right? And things like that is what, you know, you kind of had to do. And so on. Anyway, so there was a huge engine risk, and some games just failed because they couldn't produce a thing that could technologically do what they needed.

You saw houses who survived on technological prowess. You had id software that was kind of unrivaled at making those kind of first person engines. You had Bullfrog who had this engine that those two or three engine that they did for like they'd racing games magic carpet, dungeon keeper. Like they were all based on this, you know, one core attack and all this sorts of things. There's that engine risk. That was huge. And how do you mitigate it? You didn't.

you were just gritted, right? Because there wasn't a way to buy one off the shelf, so you were just kind of gritting your teeth. The other big risk, and this one is still somewhat true today, but it's just much less because you can start, you could do prototyping early. The risk is, is the game any good? Like, what are we building?

Is it interesting? Is it fun? And when you think about this problem of, we can't even really run the game as it will be because we're just building this engine and we don't even really have a way to test the game super well, and we can't really build much of a final level because we don't have level editing tools,

yet. Those are just coming online. Trying to guess what you are actually going to be shipping in terms of gameplay is incredibly hard. And there are games, famous games. I want to say like Thief the Dark Project. A very famous game, looking glass game, was formative in the stealth genre, launched a franchise, which was thief. You know, I want to say everything I heard from people on that team was that like the final core gameplay only sort of came together like right at the end, right?

And so the game just could have been a lot more, could have been polished a lot more. more, but it's just, the timing of these things coming together was so hard. And so it really was an incredibly different thing and nobody really had a way around it. Eventually, there was sort of this push towards something, well, it was basically early vertical slice prototyping, where as games started getting bigger and people were like, we can't keep doing this, especially if we're going to be putting millions of dollars on like, this is not like an option, right? They started to move towards this thing about, look, what we're going to do is we're going to

focus the entire studio on building one vertical gameplay slice as fast as we can, as hacky as we can. Whatever we have to do, do that. Prove that that is engaging to play. And then start building out everything else. Because we

simply can't afford to not know what that thing is. And then we can start building like spreadsheets that will schedule, what are the assets we need. Because that we actually believe in the thing. We can see it running and it fills in all those details, right? And that, I believe, you know, I'm not a game

historian, so take what I'm saying with a huge grain of salt. That, I believe, was a pretty big paradigm ship for the industry when they started going like, okay, we got to actually know, and it became much less seat of the pants after that, if you will. Now, I'm interested in your observation. I know you're not a game historian, but

you were in the industry and you still remain connected to it. What happened when game engines became widespread? They became not own licensable. Like, you know, we're talking about, like Unreal Engine for larger studios, but ones like Unity or Goldot, which now amateurs could also afford.

I mean, amateurs in the sense that you're a college kid or you do some side project, you can already afford the license and you can build stuff because now that risk has gone for clear the studio, so that risk is eliminated. And it also, I guess, it now opened up so much more people who can now have a shot at creating a game because you no longer have to either have this massive amount to license, this super expensive, the game engine, you no longer have that risk. The only risk is a fun. What have you observed happen in terms of both for the industry, for development, pace, those kind of things? And the reason I'm asking, because I wonder if there's going to be a parallel with AI where, okay,

you know, you needed, you needed to have an engineer who is you nailed it. Right? I feel games might give us a bit of a hint of what we might expect at the broader industry. So that is actually, I would say that's a brilliant analysis of the situation for not having lived through games and for noticing that, that's impressive. I'll say that first. And I totally agree with that. I've said to people in the past

who have asked about sort of AI impact on games in that sense. And I've sort of said as much, I've said like, the licensable engine thing kind of was our AI transition already, unfortunately. And I regret to inform you that the news is not, probably that positive. So there are some definite positive things that happen early on, because

as you say, it opens up the ability to make games to people who could not have marshaled the technical sort of the technical staff necessary to produce a competitive engine. And so giving them the ability to make games is a pretty important thing. And it allows a bunch of people to make sort of some artistic expressions that made they just, wouldn't have been able to do. Early on, that tends to be a net positive because you just have some more games coming out. Maybe some of them aren't that good,

but some of the existing games aren't that good. That's not that different. But then you get some really cool games coming from some sources that just simply wouldn't have been able to do it. Thumbs up. Problem is, it rapidly kind of accelerates into this kind of a nasty scenario

where you just have massive numbers of releases. And I think at this point, we're at the point where I want to say Steam games are in the like tens of thousands or 100,000 per year or something like that. It's so massive

that there is no way that your game will be organically noticed anymore pretty much period. So essentially it's this really nasty problem where you just have the market flooded with products. And there, you know, it used to be that if you made a quality game,

if it was fun, people would find it because there were so few games that someone would play the fun one and tell people about it and it would get purchased, right? Like word of mouth or just exposure on a storefront would be all you really needed to get, you know, sort of the word out about a game. You didn't need a huge marketing budget or anything like that. Fast forward to today where we have this sort of massive influx of games again pre-AI. It's just because now the barrier to entry is very low. and you really need a strategy to make sure your game gets found.

Is it possible that sometimes, you know, a small indie game with no marketing plan or nothing will get discovered and become a huge hit? Absolutely. It does still happen once in a while. The chances that you will be that game are like zero. So you kind of now need a marketing strategy, a real marketing strategy,

and going into the market for games without one and expecting to sell any significant number of copies above maybe a few thousand at best is really unwise. If you want to hit reasonable numbers of sales of a game, you have to have an idea of how people will find out about this game. So if I'm getting this right, it sounds like the game itself being good as table stakes,

but not enough of its own, right? that distribution, marketing, getting people to hear about the game is much more of the differentiator because there's just too many good games out there and now they're easier to create. I think that's exactly right. And that's just the unfortunate reality of it now. Was that a good trade? I don't know. But that's what happened.

And so that's where we are in the industry now. There's this other thing that I heard about, which is how new games not only compete with other new games, but with old games as well, right? Like the other day, I spent a few hours playing Death Rally, which is a game from the 90s. And every year, there's more and more good games to play. They all take away from the time that the new games have. Yes, and that problem will only get worse, because one of the things that the game industry could rely on in the past, that is much harder to rely on now, is that older games would look dated technologically in ways that consumers cared about. and we have now kind of also crossed the threshold where there is a segment of the market

where people really do care about the latest like ray-traced lighting and all these sorts of things and more photorealistic rendering or whatever it is. But a large portion of the gaming market by revenue doesn't really care what the game looked like all that much in a sense that whatever we're doing today is good enough. So 10 years from now, the games look much better for some reason, no one will really think of that as a huge

differentiator in terms of sales. You go back to 1995 and technological advances were a huge differentiator in terms of sales. You come out with something that looks good, that takes advantage of the hardware of that day, and boy, did it look cooler and feel more responsive and all these are the things as compared to earlier titles, right? And so that's also going to increase the degree to which the thing that you're talking about will happen. I can go play an older game because it doesn't feel obviously dated in an audiovisual way. I don't have to be an appreciator of retro gaming to go play something from 2017. It just looks fine probably, right?

So though there's that. The other thing that I'll just mention, which we kind of already touched on, but that ties directly into your point, is that also live service is such a prominent thing now. People are just logging on and playing Fortnite for several hours or something. that's also taking away from the possible revenue that might be spent on buying some indie game or some new AAA game even. So you have these sort of incumbents people playing Minecraft, spending their time playing Minecraft, spending their time playing League of Legends or Dota.

And that's taking up a huge amount of their time. That's, it's zero sum, right? They can only spend their hours in certain places, just like Netflix or anywhere else. They have to start thinking about, you know, they're competing with everyone else for entertainment hours. Okay. So I need to ask you. this. GTA6. How is that in 2026 at a time when we have better tools than we have ever before and we can build software in games faster than before? Like, how do games take 10 plus years to

develop? Is this some kind of outlier or has AAA game development taking many, many years just not changed at all? What do you think is going on here? So from a player's perspective, I can understand why someone would look at it and go, wow, Grant Thatth Auto 6 has been in development a long time. how does that make sense or, you know, something like this. From a business perspective, you have to understand that Grand Theft Auto 6 is not a game that they are selling to players who are going to play the game. That's not what it is from a product standpoint, right?

What Grand Theft Auto 6 is from a product standpoint is a replacement of Grand Theft Auto 5. Grand Theft Auto 5 at the time was, if I'm not mistaken, by far the most revenue-generating entertainment product in existence. The online part of that game was generating like billions of dollars. And like I said,

not a game industry historian, so take what I have to say with a huge grain of salt. But Grand Theft Auto 5 was kind of like Fortnite before Fortnite, if you will. They were a huge, huge, live service revenue generating product. So from Rockstar or Take 2's perspective, right,

grand theft auto six is not just let's try to get out the next grand theft auto as soon as we can because we'll make money selling that title it's a we are going to replace the most profitable thing we have ever built which is still generating a ton of money for us with a new thing and you can better believe that they want to make sure that they are going to do that right because the last thing you want to do is ship a new product that cannibalizes something from your old product and then is less revenue generating, right? So I'm sure that they're planning around Grand Theft Auto 6 is not just about trying to produce a Grand Theft Auto that their fans will love and will buy as the original single player gaming experience that it was. I'm sure they care very deeply about that just from a reputational and from an artistic standpoint. I'm sure there's a lot of people on those teams who care about that. But from a business standpoint, I am sure there's also been a tremendous amount of thought and work put in. into what does the live part look like. And that's a huge undertaking, you know, that I'm sure that they've been planning for quite

some time as well. So it's a massive, massive thing that they're doing here. How well will it succeed? I have no idea. But it is not just a new Grand Theft Auto is, I guess, the way that I would look at it. Grand Theft Auto 5, I think was somewhat of a surprise to them. I don't think they knew it was going to generate that kind of online revenue. I mean, maybe they had hopes, but I don't think they knew that it would be that kind of

of a massive moneymaker that it was. And so this is the first product, really, where they know they will have the audience. For Red Dead Red Dead Red Dead Redemption 2, they did a similar thing where they tried to have the online thing. It didn't, I don't think it hit nearly as big as Grand Theft Auto. Grand Theft Auto 6 is the first time they're shipping a true update to what is their flagship.

And so, you know, it's equivalent to like a relaunch of Google search or something. Like, that is what they are doing here. And so I, you know, I'm sure if I was in charge of that project, I would be sweating bullets. So I'm sure that they are putting a lot of thought into it. And it's a very massive undertaking, I'm sure. I'd let you switch gears to software craft. You made this video titled Clean Code Horrible Performance, an essay slash video, showing how Uncle Bob Martin's Polomorphism-based refactoring pattern runs about 1.5 to 50 times lower than a plain table switch version.

Can we talk about the responses to this piece? Well, I guess I can put that in context. So that is sort of from that course on the substack. So it kind of goes with a bunch of other videos that are part of like the substack thing. I guess the first thing I'd say is I feel like the response to it was very positive. I was kind of surprised. There are plenty of people who didn't like it. Don't get me wrong.

It's controversial to be sure. But I was surprised at just how many people were enthusiastic about it as well. But what I would say is it's really, I don't really think it should be so controversial because there's one thing where people want to just use the term clean code to mean code that they like or think is written properly. And that's not something you can argue against, right? Because that's just, you know, I probably have a version of what I think is clean code. And obviously I don't think that's bad, right? Like it's my idea of what good code looks like. So if your idea of clean code is just whatever you want it, you know,

whatever you happen to think are good programming practices, I might agree with those programs, I don't know. So in this particular video, I was talking specifically about the things that were advocated that are like very specific things that are said, like don't have functions over a particular length or these sorts of things, right? Things should not know the type at runtime or whatever, right? There's all these like kind of rules about it, preferring polymorphism always, right? If you look at those things, they're kind of just bad.

programming practices. I don't really know how else to say them. They don't mesh well when you put them together. In isolation, some of them might be fine. So, for example, if you really prefer lots of small functions, that's actually fine if the compiler can see all those functions and know that it can safely inline them and collapse them as necessary. This is a part a lot of people missed about the video, I guess, because it's a pretty short video, so I didn't explain anything in detail, but a lot of redundant code happens when you have lots of tiny, these little tiny functions and if they're all virtual functions

in C++, let's say, the compiler can't know for sure which ones of them are being called and so on, even if you put things like final in them and there's all, people have a lot of weird beliefs about how the code works. You can just go do this testing when you have lots of these little functions, if they're all like

statically defined and aren't virtual calls if they're just known calls, like or just member functions. When I say static, I kind of mean just known to the translation unit, not external. The compiler can put those together collapse all of the redundant code and actually produce something reasonable that will run pretty fast out of that. It can also do things like widen the code path if it needs to vectorize to like run in SIMD and stuff. The compiler has all these options to take what is fundamentally not particularly great code in terms of

how you would want it to run at runtime, but it might be able to turn it into that because, you know, compiler optimizing, optimizing compilers are pretty heroic these days and the sorts of things they can do. if instead you use all of these things that were recommended, you completely block out the compiler from being able to do those things. Because if it can't tell what it's doing at runtime, if it has to leave open the possibility that you substitute it in a different class here or something like that, then you end up in a situation where the compiler can't do any of that work.

And people mistakenly think that this is just because virtual function calls cost too much or something like that. That's not what it is. It's not the cost of the virtual function call. We could talk about that as a separate thing. Because you can analyze that cost as well. It doesn't have much to do with specifically, whether it's virtual or not. It has to do with a lot of things like branch prediction

and how much stuff was getting pushed on the stack and whatever else, right? But it's the cost of the compiler not being able to do any optimizations. That's the actual cost. And that cost can be severe. I showed only, I think, a pretty mild degradation compared to what you would actually see in production if you really had a huge number of things doing this. And I think it landed pretty well. it's a very widely viewed video

and a lot of people seem to really like it. I thought it would be probably even more controversial than it was so I was pleased with that. But yeah I mean all that stuff remains true today I guess is what I'd say and I think it's good for people to hear because they need to hear opposing viewpoints. I think you can write

code that is maintainable and easy to read that doesn't follow those principles in that way and that doesn't have those problems. I don't think you have to do those things. So I think it's worth exploring other options that are still maintainable

that are good code, but that allow the compiler to do the right thing. What is your take on test-driven development? You know, when you write the test first, then you write the business logic. You've talked a little about this as well because it's a practice that used to be super

popular in the, you know, like especially when you're building services, some of those things, especially in the 2000s, kind of got a little bit out of fashion and now it's unclear if it'll come back or not with with agents or not. I don't have that much of a spicy take on that one. My take is very pragmatic,

which is that if you can identify tests that will save time in total, that's usually what I try to emphasize. In other words, if the amount of time it takes to create and maintain the tests will actually save us total development time because they will identify bugs that would be hard for us to find in production or in, or it would be very costly to get to if they got out,

Then great. And I've used them before, like I talked about working at Rad Game Tools. I had a regression tester that I ran on like the core libraries there that I had written for the, you know, they're not like called libraries, but the core like routines to make sure that, you know, anything that I could be testing for our customers, I sort of was. And so I think there's good times for testing. I would say the part that I don't like about test driven development is the test driven part. I don't think development should ever be driven by tests.

I think tests are a thing that you should be aware of. You should know what your options are for testing, and you should make intelligent engineering decisions about tests. Now, could that decision be that for this particular project, we are going to drive it primarily from the tests? Yes, that could be a decision that you make, but you shouldn't really think of development as something that is primarily test-driven by default, because that might be a very bad decision for some other project

where it just ends up costing you more to have done it that way. So like with most things, I would advocate for a pragmatic approach to testing. You should understand the cost of testing, the cost of developing, maintaining the test, and the cost to your code base if it makes it harder to change your code base because tests have to be rewritten and you therefore don't make changes you should make. All of that stuff should be in your brain. And you should make an intelligent decision about what your testing strategy is. If that decision intelligently made turns out to be we are going to have a lot of testing on this project, that may well be a good decision. I don't think there's an absolute thing you can say about how many tests there should be.

Some projects probably shouldn't have very much. Maybe some projects should have a lot. And I think knowing which of those you are doing is part of being good software engineer, is I guess, what I would say. You mentioned being a good software engineer. But Before we get into what is a good software engineer, what does good code mean to you specifically? So good code to me usually means that you have written a something that is as straightforward to what the machine actually needs to do to solve the problem as it can be. And also hopefully that you have, I guess I'll say, properly identified ways of breaking it into easing digestible pieces and named those pieces in ways that are easy for someone to understand, especially yourself, because you are very likely to be someone who's going to have to modify

it. So that's the way I tend to code. I try to identify what do I actually need the computer to do. I try to write as simple as possible the thing that will do that. And then I try to put that in terms that are, you know, I would say least redundant. So, you know, I don't want to see the, you know, the equation for Euclidean distance scattered throughout my code. I want to have a function that's like compute that distance and I want to use it, right? I want it to then be nicely broken into

the pieces that it represents, and I want those pieces to be reassembled properly by the compiler in a way that will produce code that runs very efficiently, right? And so that's what I'm usually trying to do when I'm trying to program. And for me, I have never really understood

the sort of mentality of there's a difference between code that is like well architected by some principles and code that runs quickly. Because in my experience, usually the code that is architected properly is also the code that runs quickly. And yes, there is a point where if we decide that something absolutely has to get as close to theoretical maximum as it possibly can, yes, we will start to make that code harder to read and modify because we are not. now like really over specializing it for this piece of hardware or whatever. That's true. But that point is like, you know, way out on the curve.

It's not the common case. Most of the time, assuming you just want code that runs pretty darn well on most hardware, the simple, readable version of the code is actually very fast. It's only once you think you need to have 27 factories and 8,000 microservices and all these things running, that it starts to be this thing that's like, good architecture, but also like hard to modify, hard to read, run slowly, right, all these things. So I tend to think of like good code. There's like this nice nexus of runs pretty, pretty darn well, easy to read, easy to maintain, isn't as close to theoretical maximum as it could be,

but it's close enough. And the paths to each theoretical maximum have not been foreclosed. We left the door open with the way that we wrote it. So if someone really needs to come along and boost its performance, It's set up to do that, right? And related to this, what is a good software engineer to you? Is it just someone who writes good code or it goes beyond that? I would say it really depends on the environment a little bit. Because I think I've seen a lot of different kinds of good software engineers. And so I would liken it more to a, you know, if you want a sports analogy,

you'd imagine something more like a baseball team where it's like, what's a good baseball player. Well, it's like, are we talking about a pitcher or a designated hitter, right? And it changes quite dramatically. So there might be some things like, hey, if someone's pleasant to work with and, you know, doesn't, you know, goof off all the time and actually gets their work done, those are obviously things that we would say are true of any software engineer. You know, there are some general personality traits that might be positive. But when you're talking about things that are more specific to just software engineering and not just being a good employee or something like that. I would say I've seen a couple of different kinds.

I've seen people who are like the utility infielder. There are people who just like, they can identify and go and try to fix a problem and succeed. Even if the code base is kind of wacky and out there, they're good at getting the lay of the land very quickly of identifying some that's going on. And they're not afraid to go in and like, okay, this is kind of this code base kind of ugly here. It's okay. I'm going to patch around. I'm going to do what I need to do and get things done. That's a great engineer to have around.

I've also seen great engineers who are these. exact opposite of that. They are just like, I take this one particular problem that we have, and eight months later, I have ground out every last thing there is to know about this. And sometimes to the point of like producing new algorithms that no one's even known before, right, that are like these, you know, breakthrough things, right? And that's a great software engineer to have on a project if you happen, if you're going to be having that kind of thing. And so I've seen a lot of different people that I would consider great. software engineers and they aren't all the same person, right? So I think that it's kind of important

if you're asking it from the standpoint of like, hey, you need to put together a team to go build this project. What's a great software engineer? I would say the best advice you could give someone in that position is think about the roles. Think about what kinds of roles there are going to be here. And don't think great software engineer. Think great that role, right? Who is going to be a great pitcher, who's going to be a great first baseman, who's going to be a great outfielder, who's going to be a great this, that, the other thing, great third base coach, whatever it is, right? And that's what you're trying to put together if you're trying to build a team, to me. Yeah, so like, it's just not one size fits all. But I still want to push you a little bit.

Like, what are things that are that you think are non-negotiable for someone to be a great software engine? I mean, we talked about the things that we talked about, which is a recurring theme with you is just going deeper and deeper and understanding the next, the next layer, you know, like understand if you're doing web development, understand React, once you understand React, understand what's going on in the DOM, go all the way to assembly. Once you've done there,

understand how the CPU is doing operations and branch predictions and some of those things. Like to me, that's a skill of like curiosity, driving deeper, crafts, whatever you call it. There's different ways we could do it. But along these lines, what are those traits that you think, no matter what kind of role we're talking about.

But if you think back of some of the different type of roles that you work with? Like, do you see some overlap that they all had something? I would say that it's pretty unusual, I guess, that I can't think of someone I would think of as a great software engineer who, like, didn't know how to, like, read assembly or something. That is true. It might be that having that curiosity about how things work and knowing at some level what's going on is kind of maybe something that's going to be very common to a great software engineer.

but I would just underscore the point, the degree to which they are employing that knowledge may vary quite a bit. For some of them, that may be their bread and butter, and they're doing that all day. For others, it's just really a thing where,

because they know how a computer works, they're not making those stupid architectural decisions that come back to bite us later, right? And that's great, but they may not really be doing all that much actually at that kind of level or thinking about at that level. They're just going like,

yeah, I know we've got to kind of push, okay, this stuff's going to have to be done in bad, because I just kind of know that that's, you know, how the machine's going to have to handle it. So, you know, I'll make sure I write the code that way or whatever. Yes. But so there's a little bit of that. The other thing that I would say maybe is actually, like,

not being dogmatic about things that they haven't actually themselves proved out is probably a huge one. I find there's a lot of like received programming wisdom that's just nonsense. Like, clearly no one's ever tested it. And if they did, they would have found out that it's, that there's no actual basis for it. It doesn't actually mean it's false. It's just there's no, like, there's no actual tangible way you can demonstrate.

And sometimes it is like you could demonstrate that there are actual concrete downsides to this received wisdom, right? And so in order for it to be received wisdom, you should have to be able to at least demonstrate concrete upsides, which oftentimes cannot be done. So I would say people who actually focus on what works in practice is a huge plus. And you can apply that anywhere into anything, right? not just saying, oh, the flavor of the month is that we're writing everything with classes and virtual functions and hierarchies or whatever. It's like, did you actually determine that that results in less code or that the code actually is murmured? Like, that we do any testing to figure out if this is helping us rather than hurting us. And the answer oftentimes is no or if it was at all, it was extremely shoddily done and you would not take those results as conclusive in any way.

And so it's like being more skeptical about coding practices and actually trying to focus on what is working in practice and what we can, demonstrate and measure in some kind of a repeatable way is, I think, a really great thing for a software engineer to have as well. So people who don't tend to fall prey to that just like, I watch some presentation and someone at Google says, always call MEMSET or never use if statements or whatever it is. Like, if that's the level that you're thinking at, then I probably am not going to put you in that category of really good software engineer because that's not how it works. Plus, it's not that hard to try these things out or setup or run an experiment. Now, the final topic I wanted to touch on, which I deliberately didn't get into until now, is AI and how it's changing your work.

And I'd like to start with that. Like, in the work that you're doing at Molly Rocket with this project that is yet unreleased, how are you using AI tools if you're using them at all? We are not using them at all. So you're doing it just like before. You're writing your code. What made you decide to take this path? Well, we're a little bit different, obviously, in the sense, for two reasons. One is that we, like I said, we have like sort of two projects here, and the substack

is our primary focus. And this other one is a thing that we're doing because we want to do it. And when you think about that perspective, like, well, why did you want to do it? Well, the reason that I want to program like things in a game is because I want to program them. If I just wanted an AI to program them, I probably, you know, first of all, we'd probably just go use a licensed engine, right? Like, I wouldn't even bother asking AI to do it. I'd just go get the Unreal. Get the engine, yeah, yeah.

Right? Or something like that and so on. So I think a little bit of that decision is probably not that relevant to your audience because it's more about what do you want to do. Like what, why are you spending this time, right? It's a philosophical question, not a product. activity question. So it's not like I evaluated it and said, I don't think this will save us time or I or, you know, or I have questions about the copyrightability of it or the ethics of it or all the sorts of things that you could rightfully evaluate AI on. It wasn't necessarily that. It's more just like, this does not further the goals of the project to use it. So it kind of was a non-issue at that point, right? Stepping out a little bit more to a broader philosophical framework about AI, I guess what I would also say is I think, that if you, regardless of what you think will happen with AI in the future, because obviously we don't really have any way to predict what it will look like 10 years from now,

it's anyone's guess, really. I think there will probably also be at some point a notion of like traditional handcrafting that will come into play because we've seen this in most other times when you automate something. So if you automate making furniture and you have like IKEA or whatever, that doesn't mean that there isn't some weird guy down in the industrial district of your city making crazy wood tables with iron and welding and something. And that that's just a thing that people are still doing. And some people want that table. I don't necessarily have an explanation for it.

And I'm not trying to argue that it has more or less value. But it's just something that happens, right? And so if I imagine what I love about computers and what I want to do with computers, and you asked me, move that into some other context, which of these people would you be? My answer is always, I'd be the organic farming guy. I'd be the guy who's making the weird table in the industrial district. I have no interest in managing a division at IKEA.

I literally couldn't care less about that, right? And so I think for me, another reason why I'm not that interested in pursuing AI is because I would like to be part of whatever the set of, people are who are going to keep this traditional craft alive just because that's something humans do, not because we're trying to say that that's the right business case, right? If that makes sense. And at this point, there is a bit of a tradition, if you will, even if we assume that these machines will do as good or better than humans, it for like, what, 60 plus years.

We've only exclusively handwritten software because that's how it got done, right? Like a lot of us, anyone who started coding before 2023 or at the end of 2022 or probably honestly, 24, when these things have gotten like decently good, you just wrote it by hand, a lot of it. Or tap complete still counts. Yeah. And I guess I would say like, again, it's just, you know, why do that, right? Is the question that's like, I don't know why humans do that. Humans do that because it's something humans do, right? Humans like to do things themselves sometimes.

you know, people can buy a hat, they can buy a wool hat trivially or they can buy whatever. And then someone's out there knitting a hat right now. That's just, it's just something humans do. They like to make things by hand sometimes and at varying levels of handmadeness. You know, there's some people just buy the wool or whatever or buy the pre-made yard. Some people raise the sheep or whatever and shear it, right? Like, you can go arbitrarily far out. You can find somebody who's going all the way, right?

right, even further than probably I would ever go if I was in that thing. So, you know, you could imagine someone making their own hardware these days, right? I'm not doing that. And so that's kind of my take on it. So I'm kind of the last one to ask about, you know, AI coding or what you might want to do with it. I really have no, nothing of value to add. Yeah, but I am interested in asking you through the lens of the games industry. And we touched on on games engines arriving and now so many more.

more people can make games. Not everyone, but it's a lot easier to enter. What are you observing in terms of most people outside of who are still handcrafting code because they want to are using these AI coding agents for two reasons. Either it just makes sense and they realize, well, this thing can now generate code as good as I did, which was a turning point in January. I had the turning point actually myself. Or some are actually just pushed with corporate mandates of like you need to use these tools and eventually they kind of get. get on board, whether willingly or unwillingly, but so many folks are having AI write the code for them.

They're prompting it, but they're doing it. What do you observe of the effect having, you know, from your vantage point, may that be on quality, craftsmanship, on just output, speed, et cetera. What are you seeing? I think it's a little too early to assess, to be honest, because kind of, as you pointed out, obviously there's been people who maybe, you know, we might derogatorily call AI shills, who have been saying that it was producing as good a code as humans for, you know, two years now or something like that, right? Yeah.

But in reality, the people whose opinion I would trust more, none of them thought it was really all that usable until much more recently, right? And so we really haven't, they haven't had very many months to actually be figuring out how to use this thing or to determine to what extent they can use it and how what it's best at, what the workflow looks like that makes it produce the best results. It seems like at the moment I would say probably need to give it at least another six months, if not another year or something, to let everyone kind of shake out like what are actually the best ways to use this thing? I know tons of people in the game industry are using it. So I know that they are doing various things with it. Whether those things are the same sorts of things they will eventually think are the way they like. You know, like the things that they're doing right now may be like, oh, that was kind of dumb. Like you shouldn't have used it that way.

You should do this other thing with it and it's way more productive or something. So I feel like it's probably too early to assess. we haven't seen any real, like, obvious, like, oh, wow, like, you know, the, you know, Fortnite ships once a week now and it's bug-free, like nothing particularly interesting has happened in terms of output there. But again, it's been what, like five months or something? So it's just, it's way, it's way too early to see how it actually gets integrated into a reliable process, right? Yeah, and there are, I know there are some companies who are now tying up, let's say, agents fixing bugs, but that's only a few months.

The oldest software that's widespread that is written close to 100% by agents is from the labs, Open AIS codex and Anthropics Cloud Code, but even there it's been since November or some parts of it, December, so like maybe six months. And it's different, right? That is the product they're selling. So there's, I'm not sure we'll know for sure, like, is it truly 100% how much, you know, there's a marketing angle or not, does it? but there's a self-bias there.

So, like, I would put those aside in terms of trustworthiness. And you're right that the rest, we just don't really have the information. It'll be, I'm sure there's so much experimentation, so it'll be. But to your point, it takes time to bake, right? To see the impact. Most of these things are currently presented as tools, meaning a human has to operate them at least in some way, like at least setting it up to do what it's going to do. and therefore you have to give it some time.

You know, nobody currently is selling a product where it's just like, oh, just turn this thing on and it will just ship Fortnite by itself forever and you can just get rid of all your engineers. Like no one's actually selling that product yet, right? We could evaluate that product because we'd be like, did anyone do it? Did it start shooting Fortnite on its own, right?

So if it's still something where humans have to kind of figure out how they want to, like, slot it into what they're doing, then it's entirely possible that the reason that we haven't seen some big uptick and productivity that would be obvious to an external observer is because it's going to take a while for people to like shake that out or maybe the AIs need to get a little bit better. Maybe like we've got to go through some more update steps

or whatever, I'm not sure. So there's all that's on the table. Then there's another possibility which is that it actually already has worked but just the productivity boost isn't as big as would be obvious. If people got 10% more productive, that would still be pretty impressive because it's hard to get a 10% across the board uplift.

I've said this before on podcast. I'm like, if you have a tool that can give everyone 10% up, but that's great, almost no one would know, right? It's like you can't, it's not really externally observable that clearly if that's what you got,

but it may have happened, right? So it's really hard for all of those reasons. At some point, if the AIs are really fantastic and people figure out how to use them really well, it should be obvious. It should be obvious. be like five people are now shipping for it

night instead of $5,000 or whatever, right? But until that point, it's really hard to know because it's just like, especially if it was small, it'd be hard for us to see. Well, this is anecdotal, but I'm getting a lot of data points and messages from software engineers and managers. One impact is having is there's this

kind of like AI fatigue slash burnout from software developers who are like, look, I am good at coding. I've always been good at it. I enjoyed the work to various extents, but But since this AI thing happened since the end of the year, beginning of the year, since it's actually I'm now prompting and now all my code is generated, whether that's corporate

mandates or is just faster, I'm starting to lose my drive. Like, why am I here? Like, anyone could do this. And I think there's a sense of like I'm using a lot less of what I'm capable of. There's all this pressure from above to be more productive with it. And it's, I think we should, like, I'm seeing more and more signs that it's, what do we call burnout, AI fatigue, et cetera, loss of motivation. I haven't seen a technology.

I don't remember technology having this widespread impact everywhere. I'm hearing from folks at some of the leading, like kind of not AI companies per se, but like big enough like database providers who are now hugely into AI and they're powering a lot of the things, traditional companies, modern companies, everywhere. Have you observed some of this thing? and would you have any advisor and any pointers to folks who are feeling like this right now?

I guess I would say observed no, heard about yes, I guess is what I would say. Like I have talked to people who have been like, such and such has been having a really hard time with this or such and such has been having, like, there's,

I've definitely heard that interacted directly with someone, not currently. know. And part of that is probably largely because most of the people I talk to have a fair amount of latitude with what they do

and how they do it. A lot of the people that I talk to on a daily basis are able to make their own decisions about what they want to do with AI and so on. And so I don't necessarily hear from as many people who are going to be in a position where some manager told them this is just what

you have to do. This is very interesting because one thing that keeps coming back in Armin, when Roneitur was telling me the same thing on the podcast is he's observed that autonomy, like at your work, how autonomous you are at your work, like how many decisions you can make on, what you work on, how you do your work. The people who have a lot of that are typically like, oh, great, I can use this for this. So I can use this tool. But the people who are told, you know, like beforehand you're given a ticket or the PM tells

you this, they don't have much Google Room. And now those folks are seeing it way more as a threat because, of course, subconscious or consciousness, they're thinking, well, this thing could automate my, my job. It's now, or it made, it made from that little effort I had to do that it took it away as well. So I wonder if there's a connection here. I mean, that sounds totally logical, right? If you're somebody with the high degree of autonomy, then when are you going to reach for an AI? Well, whenever there's something that you didn't want to do, right?

So kind of by definition, I think at that point, you're going to have a much more positive experience with it because worst cases just doesn't work, in which case, I guess that's not great. You're going to be like, ah, this thing was kind of crappy. But assuming that it's able to accelerate some part of that, that was great. It's like, hey, I didn't want to do this thing already. I had this AI do it for me. And now I have the thing. That's just a positive experience for them, right?

Whereas, yeah, if you're just told, like, you had this thing that you wanted to do and you were told, you can't just do it yourself. You have to do it with the AI, you know, and by the way, we just had layoffs or whatever. You know, a lot of that stuff obviously could totally change your mental reaction. thing because now it's not you deciding to use an AI because there's something you didn't want to do that you thought the AI could do for you. Now it's you just being told that you're supposed to be using this AI to automate whatever your job used to be. You can see pretty obviously why that would have different psychological effects on people, right? So I guess it might be just an idea for folks in this situation that now you might want to evaluate your current position or if you're

interviewing your next position based on how much autonomy will you have. Because the more autonomy you'll have the more likely you're going to have control over how you're using this stuff, how much you can experiment versus being given a mandate that I don't know we're expecting you to have this output increase or output change, whatever that is. I wonder if this will reevaluate some of, you know, like what is considered an attractive position. Because like, for example, big tech was considered a great place to work because high compensation, pretty clear expectations, like easy to understand career advancement.

But now they're the ones who are starting to measure. your AI usage, which is, again, like, giving a kind of a bit of a handcuff of what we're expecting you to do, or there's where you might have, there might be mass layoffs, which again, you have no control over, right? Like, it's, again, one more, or inside of metaphors, reassignments of, like, you will now do labeling for X months. I mean, you could sort of think of, you know, could we coined the phrase, are you using an AI to do your job or is an AI using you to do your job, right? Like, because at some point, it definitely, it definitely, definitely felt like meta, for example, from your reports on it, and I've seen the same thing

said by other people. So it does not sound like a one source kind of thing. It sounds like this was kind of just accepted as fact that they kind of just were using you as AI training, right? Like that's what you were kind of, you know, you're just there to train the eye to do it so that we don't need you anymore, right? And so thinking about that from a, from a perspective of choosing your job, it does make some sense if you do it, if you have any latitude, right? But But, yeah. As closing, what are one or two books that you would recommend that had an impact on you? I'm going to have a hot take here if I might because it's sort of a push I've been on recently. I don't think people should necessarily take a book recommendation for me.

I want to recommend that people read a paper. I'm trying to get more people to just to just read papers because I realized I read a paper. because I realized I read a ton of papers. Like, I am constantly reading papers on things that I am interested in. Like, if I'm going to go do some programming in an area that I haven't done before, I will read a ton of papers. I'll crawl the references on papers. I'll read a survey and go gather all those references and read those references and crawl

them back. And I find that I learn a ton that way. And I feel like a lot of programmers just don't do that. And so my recommendation would, you don't have to read a specific paper. I'm not going to give you a specific paper. this one. Just think about the domain you're programming in. Do a search on Google Scholar for some part of that that you're interested in. Try reading a paper following the references. See what you think. I think it's a great thing to do. And I get a tremendous amount of

not just enjoyment from the education of it, but also just like more knowledge about what I'm doing. Pretty much every time I do this, even if it's just to learn a little bit more about the historical record of how things got discovered. But a lot of times it's just like, Like, I learn about whole new techniques I just was not aware of because there's way too much out there for any one person to know. And I don't know to what, again, since I don't currently use AI in my workflow, I couldn't say. But my assumption would be that AIs would also be very good at helping you find some papers to read if you were interested as well. Because that's, you know, shooting through a lot of the technical record is something that they do. and so maybe you could even ask your favorite AI

to suggest a paper that you might like based on some things that you tell it. I don't know if they're good at that, but I'm guessing that's something they could do. Casey, thanks a bunch for this conversation. This was great. Thanks so much for having me. It's been a pleasure.

I've been wanted to talk about performance with Casey for such a long time, and I'm glad that we finally made it happen. I kind of wish the industry had more people as excited and interested in high-performance code as Casey is. If you made it to the end of this episode, you might just be one of them. I appreciate that the Casey did not beat around the bush.

If you care about performance, you want to be able to read assembly and spend some time reading it. Reading assembly is several times easier than writing it. If you can read assembly, you can see what's happening at the machine level. And it's a lot easier to understand, for example, why programming language like Python is much slower than something like Rust or C, when you see the assembly code for simple operations. I was chuckling when Casey talked about these blog posts about how we rewrote our services in a new language and got 10x performance improvement, and how those rewrites are usually not about the new language with fixing the architecture that caused the performance issues to start with.

And although we did not talk much about AI, I found it amusing for Casey to say that the games industry had its AI moment years ago, when game engines became accessible to pretty much anyone wanting to build a game. Before, large teams were needed to build both a game engine and a game, and now teams of one or two can create full-blown games. After a brief spike of positive effects, with lots of new good games released, games have flooded the market in such great number that is now impossible for a new game to become hit organically. So marketing and distribution becomes mandatory even for great games.

For more deep dives related to game development and performance software, check out the link to Pragmatic Engineer deep dives on these topics. If you've enjoyed this podcast, please do subscribe on your favorite podcast platform and on YouTube. And a big thank you if you also leave a rating on the show. Appreciate it and see you the next one.