The Standup with ThePrimeagen - Why Making a Debugger is So Hard! (ft Ryan Fleury)

Episode Date: August 22, 2025

Ship Faster with Neon Postgres #ad Neon comes with Branching, Schema Diff, and Instant Point-In-Time Recovery to help you ship fast and recover faster. Get the free plan at https://twitch.tv/ThePrimeagen - I Stream 5 days a Week https://twitter.com/terminaldotshop - Want to order coffee over SSH? ssh terminal.shop Become Backend Dev: https://boot.dev/prime (plus i make courses for them) This is also the best way to support me is to support yourself becoming a better backend engineer. Great News? Want me to research and create video????: https://www.reddit.com/r/ThePrimeagen Kinesis Advantage 360: https://bit.ly/Prime-Kinesis 📍 Chapters: 00:00:00 Intro 00:02:10 Printf Debugging 00:10:20 RAD Debugger Origins 00:25:00 PDB (Program Database) Limit 00:27:27 RAD Linker 00:28:40 Working for Epic 00:32:13 What the hell is MUMPS 00:33:28 D Sucks 00:34:45 Oodle Adjacent 00:35:40 How do steps work? 00:42:50 INT 3 Instruction/Trap 00:46:20 Step count 00:47:50 Overwrite and replace 00:49:42 The desire for user level INT 3 handling 01:00:39 Alternative to User level INT 3 01:03:10 Chrome Debugger complaints 01:07:51 RAD Debug features 01:16:52 Compatible languages 01:18:41 Gabe Newell annoucement strategy 01:19:37 Outro 1 01:20:01 Cleverer than Stack pointers 01:22:55 Prime pees on stream 01:24:20 Faking addresses and the shadow stack 01:27:50 Data breakpoints 01:28:38 Prime shoots his shot 01:32:46 TLDR 01:34:28 Debugging in VM langs 01:35:00 PUBE G 01:36:28 Outro 2

All right, you know what I think we should do? An intro. An intro to the stand-up, Prime. All right. You got this. Are we doing a podcast? We're doing a podcast today every Wednesday and Friday, live on Twitch, 11 a.m., the stand-up happens. And today, we have a special assortment of guests.

We have Teage, as always. Say, hi, Teach. I like when you say that the stand-up happens every day, but then you say it's Wednesdays and Fridays. That one's... You didn't say that this time, but you've said it the last few intros. Anyways, hi. Actually, this is the second time I said what day it happened, so guess what? I know.

You're wrong with every few. Oh, we're being technical. Hi, I'm technical teaching. Bring it back from the last episode. Stand up, which is streamed every day, Wednesday at 11, and Friday at 11 a.m. You literally said it happens every day. Every day. Wednesdays and Fridays, which correct.

So what we've learned today, this is insane. What we've learned today is that when Prime does actually do the intro correctly, will then immediately derailed. Torpedoes it. He torpedoes it. What the hell? This was our one chance. The plane was like leaving the runway,

and you were like, I'm going to be the chicken that flies into the engine. Here's my question. Is this called the stand-up, or is it called the well-run meeting? Good point. Yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah. Anyway, sorry. All right. We also have with us, Casey, as usual, say hi, Casey.

Hi, Casey. I knew one of you, I knew one of you sons of bitches. do that. And a very special guest today, which is Ryan, is it Fleuré? That's perfect, actually. I was leading into it, yeah. I mean, I knew it was French. I didn't know how, like, what you got to do with that, you know? Okay, well, anyways, say hello, Ryan Fleuré. Hello, Ryan Fleurie. Nice. Oh, that was wonderful.

Thank you. And today, we have a very special topic. It's one of those evergreen topics. We're going to be talking about debuggers. And since I'm a print-eff debugger, TJ, I don't know what he is. Casey, definitely a debugger, debugger. Ryan Fleuray is the one who writes the debugger, so we figured he'd be the best single person to bring on and help us understand all the buggers. Ryan, I have a very important question to kick this off.

Is print F debugging an abomination towards man, or is it a perfectly viable strategy? Well, yeah, I think so the first thing I would probably say to introduce someone to the topic, if they were just, if they had just started programming, and they're like, they figured out how to insert log statements under the program to sort of see what happened. The first thing I would say is that there's no dichotomy between that and what a debugger is offering.

Like, there's no dichotomy between logging and what a debug, like the information that a debugger's actually getting. The only variable is what work you actually had to do in order to get a log of information out of your program. And so the first thing I would describe about a debugger is that it's collecting a log of information from your program in a sense. It's just another program on your computer that's running, and it's running alongside your actual program, the program you care about that you're working on. And it's saying they're collecting information from your program, and it's really in a log-like fashion.

So now there are technical constraints about that, because obviously the debugger has to collect information without modifying the actual program, because it's running with it live and collecting information. So there are limited ways in which it can modify the program. some of that, you know, varies depending on how well the kernel facilitates that process and stuff like that.

But the point is it's sort of, it's collecting the same kind of information. At the limit, when debuggers can't go too far, you do have to go back into your program and say, okay, now I'm going to print F here and here and here, or like, generate a log or, you know, plenty of other versions of that exist. But so it's not that like team debugger

versus team logging. It's very much like there are, know, people should kind of remove that boundary from their heads because the degree to which debuggers could theoretically do logging is, I think, more extreme than people, maybe if they're just getting used to the idea of a debugger, they might not realize. So it's not an abomination, although I think it's a lot slower than the tools that debuggers do offer you in a lot of cases. That's how I would put it. This is hard for my American mind, because I only know red versus blue. So this is like really,

this is really difficult for me to have some sort of nuanced answer. Well, I mean, if you think... Oh, sorry, go ahead, Ron. I was going to say, you can reslice it so that red is the people who think that there's a red versus blue, or vice versa. So you're on Team Red, which knows that there's no red versus blue. But blue is all the people who do, and that it's just another red versus blue problem. Because I'm American, too, so that's how you...

Beautiful. That's how you do that. This makes you want to light up a barbecue and eat apple pie. Yeah, exactly. Don't break the Kerr's life. Oh, well, I would never. I would never. Only the finest microbrewer.

All right, someone has to have a real question here to kick this bad boy off. Well, I would go. I would keep going on that front, actually. Hey, Deach, just finished the front end. I'm going to get the back end in JS. Could you get me the database? That's node problem. You got a boss.

That was a mess without types safety. I rewrote our backend and TypeScript. Could you fix up the database? I'm going to fix the ESLint errors. Not any problem for me. I got this. Tej! AI is so hot right now,

and my professor's helping us rewrite our backend in Python. Could you get a database up for me? Certainly, I would love to assist you. You're correct. Let's delve into this. Hey, Teage. Python scaled worse than TypeScript. We're going to need go.

Could you give me a DB? Fast, predictable, boring? I'm ready to go. 18. We're shipping a lot of features lately. I know, maybe too many features. I'm worried they don't need both of us. We need to slow down, but we need to look smart. Are you thinking what I'm thinking? You don't want your tools to pick your sack.

Neon does serverless Postgres. You choose the rest. Get your free plan at neon.com. So there are some issues. Like, one of the reasons why I think people like print-f debugging and say, I don't want to use it. debugger or I don't see the point of it. There are some valid, like, I think those come from a valid place, but they're not actually describing the problem correctly. So the valid place they're coming from is the fact that the

debuggers often suck. Okay, so like, yes. Like, I have complained about Linux debuggers many times, and that is in comparison to Windows debuggers, which also usually kind of suck. Like, Linux is like ultra-suck debugging, but Windows debuggers historically

have also been pretty darn crappy. And so, you know, one of the things that's important to recognize is that debuggers could give you your print-f debugging way faster than you actually writing your print-f debugging if they weren't sucking so bad, but oftentimes they are sucking. So when I am on Linux, for example, I use print-f debugging a lot more, right? A lot more. All I hurt, all we need is Casey saying, I use print-up debugging. Don't give the context. Okay, sorry, sorry. I use print-f debugging. I need it. It used to be a clean thing. So, and the reason for that is because the debuggers are so bad.

If I'm on Windows and I have something where I can do my data collection really efficiently because there's good debuggers, largely thanks to Ryan now, but also thanks to George Menhorn, the guy who wrote Remedy BG, which is a bugger that I've been using previously, when you have a debugger whose user interface and design is set up to get you the information that you need very quickly, it will beat the speed of printf debugging because the amount of time it takes you to go add those printf lines, recompile the program, rerun the program, see where it printout, oh, then then go, oh, wait, I should have a few more printfs

and then you go back in there. That speed, you can be way faster than that to collect the exact same information with the debugger if the debugger is actually good. Now, if the debugger isn't good, it won't beat that speed, and then I am right there with everyone who's like, why would I use a debar?

Yes, like, you're right, because the debugger's too fricking slow. And so part of like what Ryan's project is with the rad debugger, for example, is to stop having that choice around. It should just be a given that the debugger will collect the data for you way faster than you could have collected it by instrumentation. But the sad reality is that a lot of times currently it isn't, especially on Linux, but also for certain circumstances on Windows as well. Thank you, Ryan, for actually tackling this problem for us. So when I say I use printout debugging, that's the exact same thing as saying I use art. by the way. Oh, um...

It's like very... No, I wouldn't necessarily go that far. I wouldn't necessarily go that far. I wouldn't go that far. All right, so, Ryan, are you actually trying to provide us a solution? That's Linux sites? Linnecites? Yeah.

Yeah, that's like, so... Linusations. Linusations. Sorry. Linusations. Right. So, yeah, I mean, it's actually like... And if you go back and look at...

I think it's on web archive at this point, but if you go back and look at it, the rad debugger's like original like pitch page where they actually described the project that was one of like the primary ambitions of the project all the way back then um i think this would be like 2013 or something and i was nowhere near the project i was in high school so i was like not even working on the project but all the way back then if you go back and look at what the original ambitions of the project were it's like let's get a viable debugging solution on linux and so that's still part of the ambition and that's actually like i'm starting to work on that not quite full-time because there's a lot of stuff to fix and everything.

I mean, with a debugger in general, there's like hundreds of possible problems you could work on at any point in time. But I'm starting at this point, it's gotten stable enough on Windows that I'm starting to do more of the Linux port work. And so I have like the UI running and I'm starting to do process control and we have dwarf debug info parsing and stuff like that. So yeah, it's actually going to happen. Like it'll show up at some point soon. Before we go further, I think there's a lot of people who don't know anything about Rad Debugger or Rod. or other stuff. I know we jumped right into the print D.F controversy. We wanted the hot take, but maybe we can just take a sec. You can walk us back a little bit. You can talk about maybe

rad debugger and give people some info of what's going on. Casey's mentioned it like a thousand times on the channel, so I think we got to, we got to tell people what it is, you know. And is it Epic Games debugger now, or is it still Rad Debugger? It's called Rad Debugger, but that's sort of just historical. Epic bought Rad Game Tools back. Actually, shortly before I joined, so I was never technically a part of, like, RAD proper. But, but Epic owns it now, but it's still, it's still called the Rad Debugger, because I don't know, that's just, that's what they just, I don't know, nobody's told me that it should be renamed, so I haven't renamed it. But, but, yeah, I mean, basically, like, the story of the project is, um, it started with what I said,

like, it, they wanted a viable, to my knowledge, I wasn't even near the company at this point, so. I was there if you want, but, yeah. actual story. Yeah, maybe you should do the early and then I can say what happened after. Yeah. So what happened specifically was, and this will just show you, remember the idea of the Steam, like, the Steam PC, like Steam Linux? Do you remember this? Way, way, way back? Like, way way back in, like, the early 2010s. They were like, going to ship this Steam Linux set-top box thing. And it never really materialized. Like, some IHVs kind of did, but it never became. came a thing. And then like fast forward to today and they redid like sort of a similar plan to that.

And with similar people, like I think Pierre Liu and those guys were same. I think it's like similar team did do the steam deck and that has hit. Like that worked and they are going with it. But way back there was this sort of idea that they were going to do a Steam Linux distribution. This is where it sort of actually started. Because obviously RAD is a, does a lot of cross-platform stuff. They have to ship on Linux as well as Mac as well as as X, X, X, X, X, X, X, X. Like, in fact, when I worked at Rad, it was dizzying the platforms we support. Like, I'd build, you know, the character animation system, and it's just a laundry list of platforms that's, like, 14 long or something like that. So that was where I learned cross-platform

programming, like, the hardest way, right? I had a little bit experience before, but that was the big one. And so Rad and Valve were always historically very close. Like, Jeff and Gabe, so the two heads of the companies, used to have lunch pretty regularly. So they were never, like, at each other's throats at all. They were not competitors really or never thought of themselves that way because they didn't really operate in similar business spaces. So they were actually fairly close and

oftentimes they would even share employees sometimes. So there would be times like for example on the VR projects where some of the rad people were still technically rad employees but like Valve would pay their salaries and they were at Valve. There was all kinds of stuff like that. So it was kind of closer than people know about

because it's, you know, there's not like this is secret or anything but just like you know it doesn't get written about or no one cares or whatever. So because that was a very, like, closely, you know, they talked a lot, there was this idea because Valve really wanted to do Linux stuff and there isn't great debugging tools that game developers are used to. And again, we also need a lot of specialized debugging tools for debugging things like 3D geometry and this sort of stuff.

So we need things like GPU intervention stuff and we need things like to be able to visualize vectors and all this other stuff that you might not need. So we need ways to add that too. So how are we going to get this stuff? And so what Gabe and Jeff decide is they're going to do a debugger project to get a good debugger going on Linux, and Valve's going to fund some of it, and Rad's going to fund some of it, and it's going to happen at the Rad offices. So they expanded the Rad offices, they added, there's like a new, like, area that got added on. They, like, grew and took over a little bit more of the floor.

I assume people seem to like when I go on Ridiculous Storytime, so I'm just telling the whole story. If this sucks, you can stop me at any point. You edit it out. It's great. We love it. Leave a comment on YouTube if you love this. I love the BC, yeah, I love the BC and I can't wait for the AD coming up. Yeah, well, Ryan will take the AD.

We're going to hand it off, and then Ryan's going to do the AD. And so they had some people there, but it was always hard to get a team of people who actually would, like, get it to the point, like, where it needed to be. And, like, largely that was because debuggers are kind of, I guess if I had to say, have I described them. they straddle a really uncomfortable boundary. These nuts jokes aside, the straddle is very uncomfortable for a lot of programmers if they have to tackle it

sort of like holistically, it might be the way or head on, let's say. If they have to come at it from the front, then this straddle is very difficult. I just have to say, Casey, you've been hanging out with us a lot and I love this. I feel like I'm allowed to say these sorts of things here, so I do.

Yes, it's a safe space. It's a safe space. So what I mean by that is it's a project that crosses between UI sort of like, I need to know how to create visualizations and interactive things that show a lot of information in highly structured ways. If I had to rank programs by what is the hardest, kind to write the front end for, it's actually debugger. It is literally the hardest problem that I know of. If you wanted to see if a UI programmer is actually great or just kind of mediocre, have them try to do a debugger and see what you get out of it, right?

Because it is literally, I do not know of a harder problem that you would have to solve other than debugging really, right? Ryan agrees wholeheartedly. He's like, you're so right, Casey. Yeah, totally. 100%. I mean, maybe you could argue that, like, like 3D editor, like blender because it integrates the 3D part.

Like those are the two that come to mind and they each have kind of different stress points. Ryan decided to do both by adding 3D visualization to the debugger. So that's, you know, where he's at. But you have the hardest UI problem or close to the hardest UI problem on one side. And then you have like system level process control, register inspection, module loading and disassembly on the other side, which is all ultra low level, right? So when you put those two together, it's really hard to find

a group of people who can actually go from zero to debugger. It's really, really difficult. And the project never really got off the ground because it just didn't have the right mix of people to actually make that happen. I'll spare the gory details of that.

But can I jump in for one quick second? As many seconds as you want, it's your show, Prime. No, it's our show. But one reason... I'm feeling the line. love over here. Thank you. One reason why I've historically keep

using VIM and then integrating DAPD, Debugger Adapter Protocol, and then not using it is because the VIM UI just does not feel currently kind of set up for a debugger, because I do believe a debugger UI is a very visual experience. It is not like a, it's not just

separate buffers and text. There's more to it and every time it just feels very clunky. So I agree with you. YouIs are like super duper hard to be able to build. Yes. and debugger UIs because they have to marshal so much information and the entire value of the debugger is can it allow you to edit that information display

and update and present that information display as smoothly and as quickly as possible because there are plenty of debuggers. When DBG would be a classic example of a debugger that has all the features under the hood and is largely completely unusable 99% of the time. You will only fire it up if you need a feature that only it has like time-tobble debugging.

Otherwise, it's completely useless because the amount of time it will take you to actually set it up to do the debugging you want, you could have just done your printout, right? Like it's the canonical example of what we were talking about at the beginning where it's like anyone who was given only when DBG and was like, I pretty much only use printoutabody. I'm like, yep, me too.

Yep, like absolutely, right? So anyway, the project kind of languished as a result of this. So really not much actual usable work happened between, you know, that when it's actually It started in the early 2010s. And later. Now, I tried to hire Ryan, full disclosure, out of college.

Like, not when he finished college, but while I was in college, I was like, hey, do you want to come work on this? And he turned us down. So that didn't work out. So eventually, eventually. You just wanted him to come on the pod to air the dirty laundry case? Yes. And suck it, Ryan. We hate your guts.

Yeah. I came later. I don't know what to say. I love it. No, I was so delighted. I was super happy, but I was so unhappy then. I'm like, damn it. All right. I should have been more aggressive. But anyway, so we tried to hire Ryan and that didn't work out. But eventually it does work out. And not only do we get Ryan, but we got Nick and we also got Alan Webster. So like three people. And also, I mean, I guess Martin's never really went on that project. But he was another person I tried to hire. succeeded. So out of those four people that I tried to hire, we ended up getting all four, and through some combination of them, then they were basically able to reboot and successfully complete this project. And now it's a total usable debugger already, even though technically it's only an alpha, and they really haven't been working on it that long, because even the project was very long, most of it was not actually usable. So anyway, Ryan can now pick it up from that sort of modern era, how did it happen?

Yeah, so I came on board in 2021, and I think Alan, I don't remember when Nick joined, but I think maybe Alan was slightly before Nick, but Alan and Nick showed up, and they were working on, Alan had inherited the project from Juan Chun, I believe, and he, so he had a bunch of stuff, and it's, the thing about debuggers is that there's like a thousand problems, like you have to know how to parse debug info formats, everything that Casey was talking about, you need to know how to do process control. Reading registers even is like more work than it might seem like it would be. And and so they have all this.

Binary too. Like you have to be like patch the binary for breaks and data break points and all this crap. It's huge. Yep. Yeah. Like even how to implement like line stepping is already like that's a deep rabbit hole already. So there was a ton of work done in that direction. And then once one had left Alan sort of Alan, Alan and Nick worked together on a bunch of the problems around.

I think Alan was focusing mostly on the debugger stuff. Nick was working on, Nick is really good at like reverse engineering, like how the PDB format stores certain things, or how do you do on winding, you know, in this particular case, or all that sort of family of problems. Nick is really good at kind of drilling deep and figuring out how that actually works. So he was kind of working on that stuff. Alan was trying to think, like work through the problem of like,

how do you actually build, like, how do you actually build an architecture for the actual debugger sort of engine, if you will? And then I joined, and my role initially was sort of like, I was helping with a bunch of the debug info parsing stuff, but I was going to build the UI. Like that was the end goal. Like I would be on the UI, like the front end side. Allen would be doing the engine. Nick would be doing like debug info format parsing, unwinding, like figuring that stuff out. And for a while we were just working on debug info parsing.

So that was 2021. And maybe six months in or eight months in, I built like, I switched over to building like a little prototype debugger just so I could like familiarize myself with the problem space. That has been like thrown away. So that's not, that's no longer in the code base.

But it was just like a really crappy little like, let me figure out how you even do things like stepping. Like how do you actually specify how a step is even supposed to happen? And and then, um, Alan worked, the last thing he worked on before, uh, he left was the,

we also have a custom debug info format. So both PDB and Dorf, one of the, so one of the problems you have to deal with with debuggers, especially one that tries to ship to both Windows and Linux, is you have to deal with like the situation of the different tool chains. One of the realities of that situation is that these two different tool chains use completely different debug info formats, both of which are extremely complicated, and they're very different in how they actually organize and slice information. And so we had kind of always talked, at least for, I'm sure they talked before I even got there, but the, while I was there also, we were talking about, like, this debug info format problem is really difficult to deal with, actually. And so if we could actually move to our own format instead, and if you start with PDB and dwarf, you can convert to our format,

and then build the debugger on that, and then it works everywhere. You don't have to deal with, like, an abstraction layer, for example, over incrementally parsing dwarf or PDB. It gets really nasty, really quick. We tried to do that, and then it was like, it turned out to be really complicated. And so the last thing Alan did was he designed the first version of that debug, info format.

We should mention Alan's not there anymore, so that's why it's like the last thing he did, because he left shortly after that yet. Yeah, yeah, yeah. So, and then at that point, once he left, I was building the UI for the debugger, and then I just sort of took over the pieces that he had written, and I either took them over, rewrote them, or they kind of like, you know, dissipated because, you know, it's just, as the, as the system evolves, if there was a layer that he wrote that I wasn't familiar with, I was like, I don't, I'm not really sure.

to do with this, so I'll rewrite that part or whatever. So, so, um, understanding to replacement. Yes, exactly. So I kind of, at that point, I took over most of the debugger stuff. Nick, Nick actually works on a linker. This is another very deep rabbit hole

we could get into. So Nick works mostly on the linker, but he still does a bunch of like the, like he built the dwarf format to our debug info format called RDI. He built that converter recently. It's still, it's still early, but he did like a first pass on it. So he's working on problems that overlap with the debugger,

but he spent most of his time over the past like year or two, maybe a year and a half on the linker. And then I've just been on the debugger. So, and I guess that's the story of the project. What it actually is is it's a graphical debugger that you can debug your programs with. So one thing, one thing that I think people, so one of the things that I was literally complaining about

on Twitter just recently was people are like, well, game programmers just don't understand all, the real problems that banking people faced or something like this, which is one of the most absurd things I've ever heard, and I'm really sick of it. But just to underscore how much that kind of thing is ridiculous, I have not ever actually heard of any other project besides Fortnite that hit the limit of PDB.

So the literal output of debug information on Windows, Fortnite got so large as a project, it could no longer be built because the debug info could not be stored I just want to There's a file limit size right that exists on Windows so if you

No it's not a file limit size on Windows It's a file limit size in PDB The way that off sections Yeah one of the sections They use 32 bit yeah they use 32 bit offsets Actually they use signed 32 bit offsets Into like it's like one of the type Information sections and so once you get enough types

And especially once you're doing like heavy CPL Plus template metaprogramming or lots of lambda functions, those generate types. Every single one of those generates types. And so you get this just massive expansion of type information. And then you run out of that space, like pretty quick once you're a... So JP Morgan, call me up and let me know when you had to write your own linker because you exceeded the PDB file size. I'll wait. I was wondering, though, like, Ryan, which collab was it that pushed them over the limit was like Star Wars?

Or was it, like, which one was the... That's a good question. You know. Oh, my goodness. It might have been... All the Star Wars characters. Oops, it's the Pokemon collabs. You know, we gotta make sure it's got all the Pokemon.

You know, I think it might have been the Snoop Dogg Skins. Like, when they added the Snoop Dogg, you know, doggy dog skins, that was like, this is too much. It was when they stored all the Darth Vader voice lines that got them in trouble. Yeah. Yeah, yeah. Well, Sabrina Carpenter lyrics, they don't get compressed very well into PDB, so the problem is like that just... And it was a problem, and it's fine. But most of the songs are...

like six different words. They just don't compress very well because they're so creative, Teage. Right. Oh, God, they're practically white noise. They're so creative. They wanted to compute them at compile time too, so each weird word was a template

instantiation. And so that's why there's so many types. That makes sense. That makes sense. So, long story short, there are some pretty cool things about this that if you read between the lines are there that's really awesome, and I think will be very, very interesting to people going forwards.

One is, Rad did release the linker, which means that now if you want to bypass the entire crappy, like, Microsoft and or Klang linking scenario on Windows at least, and eventually, presumably Linux, you can. You can have a much faster linker that produces RDI. Is it RDI is the name of the file type or something? Yeah. Directly. So you can have that debug info, which is more efficient to use and much more, like, well documented and, like, this is easy. you know, it's easy to kind of get into. That's pretty awesome.

But two, those RDIs also now, there's like this convenient sort of thing you can look at where it's like, here's a code base. If you would like to access the debug information for your own programs, to write your own instrumentation tools, your own inspection tools,

your own metaprogramming tools, whatever you want, now RAD has done that for you, or Epic has done that for you, right? And so this project has a lot more cool stuff in it than just the debugger because they chose to tackle a bunch of other hard problems at the same time and succeeded.

I got a question for you, Ryan. As someone who also worked at Epic, but a different one, how often, like, on a weekly basis, after you tell someone you work at Epic, do you get, whoa, bro, that's epic. Every time, like, because I feel like, it's, you know, like, guys, come on. Everyone said that joke already.

You can relax. It definitely happened. It's funny, because I was wearing an Epic shirt to a hospital at some point. Yes. And they're like, Can you help us with this quick? Just like fix my chart problems really fast.

Yeah. Yeah. And like it was like, it was one particular shirt design where you could see Epic very clearly from a distance, but you couldn't see the games part very clearly. And so they were like, oh, epic. Yeah, we use Epic here. I was like, don't tell me they're putting on Real Engine like in the MRI machines or whatever.

It's a brand new therapy. We put them on Fortnite. It's like the days go so fast. Before you know it, you're healed. You know, it took six months. All the surgeons are like flossing in the room. Like they're like doing stuff. And you're like, what is happening?

And they're like, ah, it's an emote. Sorry. Don't worry about it. They've always emoet after a surgery. Yeah. Crushed it, bro. Yeah. Yeah.

New heart. New heart. New heart. New heart. Closed up that valve. All right. So I actually have a pretty serious question. Since you do work at Epic, do you get free Fortnite?

Isn't Fortnite already free? Ha! I don't know. The Sabrina Carpenter skin. Can you get that for free is what he's asking? Just asking for a friend, okay? I see, I see. I haven't tried, but I can check for you.

I'll let you know. At my epic, they had my favorite phrase, which was, consider your salary a stipend for that. Like, whenever people are like, are you kidding me? Why isn't gym membership included in our paycheck? Right?

Because, like, that's the big tech thing to do. It was like, we're going to micromanage your life and give you 100 extra dollars a month to pay for like a gym. Right. And I loved Judy. Judy on stage, like 12,000 people answered one time, like, consider your salary a stipend for that. And I love that so much. I'm like, yes. Yes.

What do you think we are some kind of health care company here? Well, they did give really good health insurance. I missed that after I left. They were, you know, but yes. Well, yeah, they probably just hack into like, It's running all the systems. It's just they set all their employees to zero dollars every time, right? It's like, we're providing health care.

Zero. Yeah. Oh, yeah. The other thing about the epic misunderstanding, TJ, is not, so you probably don't have this side of the problem. So they're like, oh, you work at Epic. And it's like, no, no, no, not that epic. The one who makes, like, unreal. And they're like, oh, that's unreal.

So there's like two words they can say. Yeah, exactly. So it gets, that gets rough. I just had the look of like intense sadness when they found out I didn't make Fortnite. Like the only thing they thought they were going to be able to connect with me on. Like I'm a software developer. I work at Epic and they're like, oh, cool. Something interesting.

And I'm like, no electronic medical health records. And I work in an insurance and billing section. Yes. They're like, well, that's kind of cool too. almost like making the world's biggest video game. It's almost there. What language do you write in? Oh, mumps.

The conversation's over. I did a little bit of programming in college. What is it? C++. C? Oh, mumps. What the hell is mumps? That's the language that they use at Epic. Oh, that's right.

Casey, you weren't there for when we talked about. The hell is mumps. That's a disease. Why is a healthcare company programming in a language just named after it is? What the hell is happening? Casey, but this is like, why do people, why do programmers think of dumb names? Because they're stupid.

What is like? Tell me, no, I want to know what Mumps is, though, now. It stays for like Massachusetts University, multi-purpose system or like something like, it's like some, they made it at some hospital in like the 70s or something like this. I don't remember exactly. All right. And so they don't call it Mumps anymore for exactly. They call it M. But it's, it's not better.

They just call it M. That's very confusing. well yes is James Bond now like it's like oh that's Q and that's M they also skip like D E FG like they started to see

and other I am yes but guys there's tradeoffs the option is presenting the doctors with this was built with mumps or this was built with MAM okay they're gonna say something fine sometimes they say intersystems cache because that's one of the people that provided I would just like to point out that basically

all of the D language stands out there are now furious at prime for saying they skipped D Wow. That's going to hurt some people. All one of them. Wow. Seven of you in the comments.

Let us know that you've written a line of D. Oh, my God. Oh, we have garbage collection and manual memory management. Oh, but you don't have users, okay? Hey, what's one thing that both C++ and Rust users can agree on? D sucks. Right? D sucks.

Wow. Oh, my God, you guys. All right. I'm not involved in this. This is nothing. I don't want to have a fight with D, people. We're cutting that out. We're putting the whole thing in here, and we're putting in a U laughing.

We're just going to splice you out of context. Oh, yes. I should never have let you guys have edited. You should never have been able to edit these. Do you actually know anything about D? Me? I do, yeah. Yeah, yeah.

A very little bit. A very little bit. You can just say that if you need to. What? Casey, but explain this then. If you're such a big fan of D, then why is it not D moratory? I'm not a great question. I'm just saying like I don't...

Oh, we got him! He's not a fan of D. Cut it. He's not. Boys, put it in. All right, well, thank you, D. Moratory for that. You know, I'm not going to lie to you.

Ryan, you've been suspiciously quiet about Oudle. Could you please explain why you haven't talked about Oudle once yet? What? He doesn't work on Ouddle. Yeah, I don't work on it. Obviously, you'd use the compression in all of your debugging, so I would assume so, at least. I don't actually... Like, the only thing, I use, like, one header file that Jeff gave me for compressing,

but I don't even think it's any of the Udle stuff. It's just, so I'm very, yeah, I'm very Udle adjacent, if you will. Do you want to talk about Udle Prime? No, I don't actually. Okay, I'm like, they would come on probably. One of them might, I don't know, if you want to talk about Udle. I just wanted to ask an Udle question at some point, okay? It was very important to me because it's such a funny name.

Okay, we'll do an Udle stream next time. This time, though, can we talk about debugger information stuff? Because I'd like to talk about that. I actually will, can I do a real follow-up question? Please. You said something, Ryan, inside your talk, which is how do steps work? And I thought the same thing. I'm like, how do magnets work?

How do steps work? And so can you explain why that's such a, like, a tremendous problem? Yeah, so I don't know what a magnet is exactly. Yeah, no one does. You don't have to answer that one. It's not possible. We don't want this to become a, Christian stream right now, so please don't answer that question.

They don't know the reference, Brian. They don't know the reference. No, he does know the reference. That's why you said that. No, I don't think he does the reference. Take the look at this fine creation. And enjoy you better with appreciation.

I don't think I know. He doesn't know. That's fine. Neither does chat. Move on. Okay, got it. Okay. That was the perfect response if you did know. though that was like so good because we don't know how magnets work. I mean no one does right. That's the whole point. Yes. Thank you, Casey. I see. All right. Okay. So I can explain how to get back to this.

How does steps work? Okay. So stepping, so line stepping is okay. Um, I guess the first thing to understand about the problem of stepping is sort of how how a debugger interacts with like another running program in like another native running program. In like another native running program on an operating system. So basically the debugger has the opportunity to make changes to memory and make changes to registers and then it can suspend or resume certain threads. Like you can basically set up certain threads to run or others to not. And then it just says go basically. You can just say to the operating system, okay, now you can schedule this program's threads again. And then like the debugger loses all control at that point until that program

does something that would that would cause the kernel to transfer control back to the debuger. And so if you think about what it actually means to, like, step over a line of source code. First you have to, first you have the problem of, like, locating where the, like, where a thread's instruction pointer, because you're stepping a thread over some line of source code that it's on. First, you have to, like, extract the register value that stores the instruction pointer of whatever thread you're stepping. You have to then understand where that actually map, like, what source code that actually maps to. once you know that, you know the range of addresses that are associated with that particular line of source code, that that thread happens to be stopped on.

And then you have to say, like, you know, let me, basically I have to produce the effect that this thread will stop execution, like the program will stop execution when this thread leaves this line of source code. Now, a very naive algorithm you could do to do that would be to say there's something on CPUs where you can single step a thread, which at that, at that, level, it's just stepping over one single instruction. And then it immediately interrupt. So you can basically set a bit in one register on the CPU and you can say, okay, step over one instruction and then stop. Can I interject one thing real quick here for the folks at home? So just to be clear for anyone who hasn't looked at this kind of thing before, so when you have a line of source code, a compiler is turning that into potentially many assembly language instructions that will actually be executed by a CPU. So there's this mapping. And not only is it a like one to potentially

many mapping, meaning one line of source code could produce many assembly language instructions, but also in more sophisticated builds, like when you're running an optimizer, those assembly language instructions do not occur in a chunk. So it's not like going from one to like three instructions in a row. It could be going from one of your source lines to several instructions peppered throughout other instructions it has been mixed with that come from other lines of source code, right? So the mapping is very messy, and one of the reasons that debug information is not a super easy problem is because the whole point of that thing that Ryan was talking about before with PDBs and dwarf, like this debug info that gets produced when you build your program.

One of the reasons that's so messy is because that is trying to store in some way that a debugger can use, among many other things, how do you go from a line of text that the user typed into VIM, right, to which assembly language instructions the CPU is going to execute which ones it caused? because when you're going through an optimizer, it's combining instructions, it's like removing instructions, it's reordering instructions. And so even just that one part is messy.

So everything's Ryan's talking about now when he's saying, how do you go from one of the other? Like, just keep in your head, it's a very messy mapping between those two. That's all I wanted to throw in there. I had to also throw in just a quick question

to answer along the way, which is when you break point or you hit some sort of stopping condition and you're in the middle of a line, does that ever happen as well, like where you're just like, I'm halfway through a line? What line do you appear on?

Does that screw up debugging information that causes confusion. I'm very curious about that. Yeah, so that definitely does happen. And it can even happen, like, for example, if you step out of a function call, like you're inside of a function you want to return to whoever called you.

Generally, like, you might still be some instructions to execute after you've actually popped from the call. So if you do a step-out operation, you'll end up in the middle of the line. You're not just at the first address of the line's machine code. You might be at some instruction in the middle.

And so basically, that's just an extra wrinkle in the problem. It doesn't really screw anything up, but you just basically have to say, like, this is a range of instructions, and if the thread ends up anywhere in here, I have to associate that with this particular line of source code. So it's one detail in the mapping,

but it's not like a showstopper by any means. So, yeah, if you wanted to step over a line of source code, if you wanted to basically proceed this thread until it has exited this particular line of source code, you, one simple thing would be to single step that thread until you see it has left the line. Like you just keep mapping the instruction pointer every time it comes back.

And then if it leaves, if it's changed the source code line that it's on, then the step is done. So that's a very simple algorithm. It also is enormously slow because remember that each one of these interrupt instructions that comes back, or sorry, each one of the single step, after a single step has completed,

basically what's going to happen is the kernel is going to schedule that thread, and it's going to say, okay, execute one instruction. It's going to immediately trigger and interrupt. The kernel will then switch to the kernel, and the kernel will say, okay, now this is, I need to report this debug event to the debugger, so it has to transition, it has to context switch over to the actual debugger itself.

And so every single time you do a single step, it's one of these round trips you can think of it as between the debug, the debugge, or the target process, and the debugger itself. And so if you've got a line that has a hundred or a thousand instruction, I mean, you know, you can make it arbitrary, arbitrarily long, really. You know, you're causing a round trip every single time someone, every single instruction within that line.

So that gets very slow. So instead, what you kind of want to do is say, like, I'm going to set up a bunch of, there's another thing, another concept that I should introduce, which is the int three instruction on X64, but there's equivalents elsewhere. where you can basically put a, it's a one-byte instruction that you can use to overwrite any instruction basically, because it's just one byte.

So it's the minimum instruction size. And you can go ahead and say, I'm going to put this here, and then when the thread executes this instruction, it's called an int three, and when it, there's other kinds of interrupt codes, but int three specifically is this one-by instruction that you can put anywhere in the code stream. And when a thread executes that, it'll immediately interrupt.

So it has the same effect sort of as the single-sense. step except it's caused by an instruction you put in the code. And so then you, so in order to avoid all of these round trips, what you can start doing is saying when the thread is in this line and I say step over this line of code, I'm going to basically set up, um, it's called an int three, it's also called a trap is what people call it. So what you can basically do is start putting these traps anywhere you think the thread might go within this line. Now if it's pure linear code, like it's going to do a bunch of mobs and then an ad and then, you know, whatever, very simple linear code, there's only one trap you actually have to place, which is at the beginning of the

next line. But it starts to get more complicated if you, for example, if inside of that line of source code you've got a jump instruction or a call, then you have to start thinking about like, where are all the places this thread might actually go to? And so then you might have to start doing things like, okay, let's disassemble the line, let's figure out like where the control flow will lead to in this line. And let's go place traps in all of those places. And then it gets it gets a little bit more complicated too because for example if you're in a recursive function call

so two variants of stepping exist for people who haven't used to buggers. There's step over and step into. There's a lot of other steps you might imagine doing but those are the two like common ones and step out. Step over will basically it won't follow call instruction so if you do a step

into you're sort of stopped at a function call and you want to go into the function call and see what's actually happening inside of the function call. With a step over you don't care about what's going on inside the function, so you just want to step to the next line and have whatever happens inside the function happen. Now, if you wanted to do a step over,

you could imagine if you are in a function that is recursive. And remember, a debugger has to deal with any code that someone could write and feed into it. It has to kind of handle all these cases gracefully. There's lots of recursive code that exists. So if you're stopped at one particular line, and it's a recursive call into the same function. It gets more tricky because if you just place a trap at the next line

and then you try to step over, you're going to hit that same trap address, but in one layer deeper in the call stack. So you basically have descended to the next call, hit the same trap, but it wasn't actually a step over. Because the step over, it didn't step over the line. It didn't step over all of the call work that happened. Now you're one layer deeper in the call stack.

So there's no way anymore to actually step over that line of code. So you have to get tricky about like, okay, have I, has my stack pointer changed? When do I update which value of the stack pointer that I want to check against? It becomes kind of basically like a little program you have to send down and say like, if it hits this trap, do this. And if it hits this trap, do this. And if it hits this trap, then single step here. And like, it becomes a very complicated sort of state machine that you actually have to implement.

Okay, so, hold on. Can I get one more? Oh, no, TJ wants it. I just got just a quick question. I was just wondering, like, on average, what your step count is every day. You know what I'm saying? Like, because for me personally, like, I don't know if this makes me a debugger, like a step debugger or not. But, like, I've got my own, like, walking treadmill now at my desk.

Yeah. But I didn't know if this is kind of like what, like, what scenario this kind of goes on. So I just didn't know how this related to debugger. Does this make me a step-driven debugger? Like, if I'm reading- You're stepping into, you're not stepping over. Over. So we're talking about stepping over right now, TJ.

Okay. I'm trying now. There you go. See, like, I'm trying to stuff all in there. That's the harder one to do. Which one is hard? How do I do recursive treadmills?

You're on an inch three and it's inside the staff. You put that treadmill on a treadmill. Oh, that makes so much more sense. And then it's running too. Yes, exactly. Right. I got it. Okay.

Anyways, you can continue now. Yeah. Okay. Okay. Now we have a sub question. Now our stack has been pushed. TJ, how long have you been waiting for that? And also you have just a black square in your screen.

So, I saw you raise it like 10 minutes ago. Immediately after we started talking about step stuff, I put my, I put it into standing mode and I was getting ready. I had to set up that scene. I did not have the video thing ready before it is. So I probably heard about 5% of what Ryan said, but I was so excited for this show. Wow. Okay.

I was so excited for this one. That is some serious professional level podcasting. All right. All right. So when it comes to stepping and putting in all these in threes, do you? You also have, like, because I don't precisely understand how jump instructions work. I believe there are relative and absolute ones. Does that mean you also have to update all the relative instructions around all these places that you put these in threes?

Because theoretically you shifted down some amount of the, like, assembly. So you're not inserting. Okay, I thought you were dynamically at like runtime being like, okay, like they're stepping next. Let's actually insert some stuff in the program. You're overriding it. You're over. That's why it's crucial that in three has a one-byte version because you're romping it over the instruction. One of the things Ryan didn't mention because he had to mention like 8,000 things because this is incredibly complicated,

is that actually when you put that int three in there, you have to remember which bite you overwrote because once you are done stepping, you have to go replace all of those with the bites you overwrote. So that happened as part of this process. Oh my gosh. Okay, so does that mean when you literally override assembly and then when it hits that int, you then have to put back in the correct assembly so

that when it executes the line, it correctly executes the line? Correct. That's crazy, dog. I don't know. It sounds like probably debuggers, it's as easy as making a regular crud app, guys. Dude, have you got a banking app, Ryan. Do you have any idea about a banking app?

Yeah. I can't even imagine. and it's just that's like magnet level stuff right there how do banks work actually we don't know if you would have been paying attention to Twitter Lori Wire just tweeted

how AHC works which is literally FTP so yes all right we do cutting edge we do that's why it anyways so underlying all of this right is like

this is also why you may have noticed there was a little snippet of Ryan at the Better Software Conference doing a little thing with Wukash, where he asked him, like, what was the one OS feature that you would like to have, that you don't have right now? And Ryan said, I would like to have user level int 3. And it's like, what is that, right? And that underlies this entire thing.

So what happens is in the CPU, right, there's this thing called the interrupt table, or there are things called interrupt tables in CPUs, right? And the reason for this is right at the beginning, this is in like 8086. Like the original 8086 chip has this. This is not like a new thing. They've been upgraded over time to have different capabilities, but generally speaking, this exists for a very long time. So they have these tables because what you often want to do

is you wanted to have some kind of like higher level supervisor, higher levels wrong term, supervisory system level, should I say lower level code, operating system code, whatever. You want to have some code that you can trigger from anywhere instantaneously, meaning you don't have to like know in advance that you were going to use this code at this time. A call is when you know you were going to use it.

At this point in the code, I call this function, right? This is like we have no idea what we're going to do. We just suddenly have to do it. And one of the big reasons for this originally is like servicing hardware. The CPU has to read something from some bug. on some hardware, you know, like the original keyboard input or whoever knows what's happening, right? And so what would happen is an interrupt, literally the reason it's called an interrupt three, interrupt three, an interrupt would happen that stops the normal execution, what the CPU was doing,

it looks in a table for what it should start doing, it goes and does that, and then it resumes back to wherever it was in the program. It's literally what it sounds like, it's an interrupt. And this was used for doing things, like I said, like reading hardware and stuff like that, because bytes would like come into the CPU that it had to like look at right now. So it would stop the program, you know, this is back when you didn't have multiple cores or anything like that, right? Stop the program as is executing right in the middle of whatever it's doing. Go service this interrupt. It's called servicing the interrupt, right?

Go jump to that code, do it, right? So what an interrupt really is, is it's basically the same kind of thing as a function call, but it's one where you didn't know what was going to happen. It's a function call that happens out of nowhere. And the reason that this is very useful for something like debugging is because one of the things that a function call will do, right, is it's going to save what the instruction pointer was before it does the call. So we will put the instruction pointer onto the stack, right? This is a call instruction. We'll put the instruction pointer onto your program stack.

It then transfers control over to whatever the location is you're going. So it changes the IP to point to whatever the function is that you wanted to call. does that and then when it goes to do a return, it will pop off the stack whatever that instruction pointer was. So it's basically a thing that uses your program stack, your standard program stack, to track where you are in the program. That's a normal function call.

An int can't do that because it doesn't even know if there is a stack. It's happening in the middle of any code. There may not even be a stack frame. You may not even be using the stack pointer as a stack frame at that time, technically in 8086. So what it has to do

is do some special work to preserve, right? Like the IP in this way that we know we can preserve just what the instruction pointer is. Then we're going to go into whatever. We're going to look in this special table that the CPU literally has a hard-coded register that sells it where it is.

So it's separately stored, right? We're going to go there, look up where we should go. And by the way, nowadays, we also look up a bunch of information like whether we should be transferring control to like a different, level like should we be dropping down to Ring Zero from Ring 3 and all this other stuff. There's all a bunch of crap in that table

and off you go. So the way debuggers are written currently because they have to be is on something like Windows they are written as a user level application which is fine because that's roughly what you want. You don't want your debugger to crash to take down your whole system right? You don't want CrowdStrike happening here right? So you want the debugger to be running at Ring 3 on Windows so

it's running up like a normal program and if it screws up, like if Ryan on like the one bug he has, right? If that bug, if you hit that bug and it crashes your system, you just want the debugger to crash. You don't want your whole system to blue screen. So it's good that it's doing this. But the problem there is it needs to do all the things that Ryan just says. It needs access to the memory. It needs to be able to put in like these things like int threes that will happen. And the problem is when you are at user level, right, the kernel doesn't want you doing stuff like

messing with the interrupt table or, you know, saying that you're going to be the person who gets whenever an int 3 happens on this core that you're the one who's going to get it or whatever, right? It didn't want any of those things for security and stability reasons. So as a result, the debuggers have to do all of the work that Ryan described through an API that the kernel will then use, you know, it takes your API calls and goes, oh, I see. You wanted me to tell you when this guy hits an int 3. So you go, you know, I'll let you access his memory through my memory mapping APIs. You go, you poke it in there.

You poke your int 3 is wherever you want. Great. When the N3 gets hit, the kernel's the one who's going to get called, not the debugger, and then it's going to go through the API to let the debugger know that it happened. What this means is that now all of these things Ryan's talking about, every one of these traps that gets hit, is a full round trip. Debuggy is running at Ring 3. The trap gets hit.

The int three gets hit. It's got a drop to ring zero. Wakes up the kernel. The colonel goes, wait, what is happening? Oh, right. This is a debugged executable. I know I should tell the debugger about it. Okay, let me re-switch the task over to the debugger's context. Let me wake up the debugger in ring three and resume execution of a debugger thread to handle this. The debugger then goes, what just happened? Oh, the colonel told me I had a trap. All right, let me go fix up the trap things. Let me set any new traps that I need. And then let me retell the back at ring zero to resume the debuggy. Back we go to ring zero. It then switches in the context of the debuggy, back up to ring three, and we resume where we were going, right? This is wicked, freaking slow for so many reasons, right? And so what Ryan was talking about is, hey, man, what if we could just set the int three to just jump to something in the actual, the person who's being debugged, that program? The debugger could, like, jit compile some code that just does the trap. operation that it wants to do and resets whatever it needs or anything else, right?

Checks conditions, anything it wants to. We just set the int three to not do any of that. Don't tell the kernel, don't take your ring transition, just jump right to that thing, do the stuff and then resume, it would be so much faster. Getting rid of that like first ring tangism, back up, ring transition, back down ring transition, back up ring transition, right? It gets rid of all of that. So that's what he was asking for, and this would be like a huge feature.

If there are any kernel devs out there watching, please add this. User level interrupt handling like this would be so useful. And then that thing, if it actually wanted to do that huge transition thing, can just have a syscall that it does called actually transfer control to the debugger, right? So anyway, whole big thing of coolness there, we can't do it. All I got from that is horse and debuggy is slow. That's what I got, Casey. I saw him.

I tried so hard, Ryan. I tried so hard. It actually also tries hard. That's the problem. Sorry. Anyway. Yeah. Like what example of, so at a high level, what you might imagine doing in a debugger that uses, that's sort of where you want that kernel level feature would be if you set a breakpoint on a particular line of source code, if you hit, if you like run your program within the debugger, you say stop here.

That's perfectly, that's like reasonably fast because those, like, two transition or. three or four transitions actually, you pay that cost once, and at the scale of like, oh, run to this line, it's not really noticeable. And you wanted to come back to the debugger anyway. So it's like doing what you wanted roughly, right? Yeah.

So when that gets really, when you actually really have to pay that cost to a sort of prohibitive degree in many cases, is when you wanted to say, stop at this line of source code, but only if this condition is true. So conditional breakpoints are like super common debugger feature. And it's like kind of the first,

version of fancier stuff that a debugger would want to do. But this is where it gets really slow because then basically the debugger can't do this installation of code that checks the condition inside of the debuggee. And so what has to happen is it hits the int 3, controls transferred back to the debugger, the debugger checks the condition, it evaluates some expression, it does whatever evaluation stuff it needs to do to check the condition. And if that condition is false, then it goes back to the debuggee. And so if you imagine you put this in a loop that's supposed to run a million times. Maybe it happens instantly, normally, but because you're paying this round trip cost, every single time the thread hits that in three, it's, uh, it just becomes like

prohibitively slow. Now, if you're doing it once per frame, like, it's still worth for, it's still worth it for debuggers to have conditional breakpoints today, but they're like dramatically pessimized just from this problem. But if that problem were addressed, you could start doing lots of other things. Like, another thing, something like printf debugging is profiling, like, marking, uh, inserting profiling markup into your program. Like if you wanted to say, oh, I want to sample the time here, sample the time here. Let me measure that, build me a flame graph of this kind of, of this kind of thing. That is the same problem as print debugging where it's like, okay, go back to the source code, insert the things that I was looking for, build the program, run the program, get back into the

state that I wanted to be looking at, and then inspect the log. And then it's like, oh, I need to drill one layer deeper into this, into this like flame graph here, but I didn't add the markup there. So let me go back to the program, add the markup there. And you can't do it too much. You can't do it at every function, for example, because then, you know, that profiling cost will start adding up dramatically and then you won't even be able to see what the actual performance costs were. So what the debugger could do is like let's just do that dynamically. In the debugger, let's say add a line of markup here, like a sampling line here, add a sampling line here, and then it could gather that data completely dynamically. It'd be way faster to like drill down

into problems when it's really uncertain what you need to be looking at basically. So, so yeah, that's like the high level what you would expect as the user if you had this. kernel level feature basically. And so what we have to do now, basically, right? If you want to do this kind of thing, you can still do it. But what you have to do now is you have to handle a crap ton more edge cases, basically. So what happens is, if you want to, you can still do effectively this by not using int3, right? Because remember, the important part about it three is it's a one-byte instruction.

So even if it's overwriting another one-byte instruction, it's okay. all the code will still work exactly as it should because you only have the int three just exactly where it needs to be. And anyone who's like jumping to the bite immediately after that doing something weird or who knows what, it's fine. Right. However, if we wanted to implement a feature like this without the assistance of int three or halt or any of the other instructions that go through a table and are one bite, right? There aren't very many. I think there might only be those two. I don't know if there's a third one.

but anyway, on x86 anyway. So if you want to do that without this, and Ryan has looked at this sort of stuff, and this kind of thing people are currently doing, so this is how you have to do it, you have to input some kind of a jump or a call actual instruction there, which takes more than one bite.

So you can still do everything we just said, but you have to overwrite the thing you were going to overwrite with several bytes potentially. And when you do that, now you run the risk of, oh, if those several bytes overflow and go into the next, next instruction. That could have been instruction that someone was jumping to.

So now we have to do all this checking to figure out, like, is someone jumping into the place that we had to overwrite to and blah, blah, blah. So now what should have been a very straightforward extension of just patching in an int three and everything works perfectly and guaranteed no problems, what should have been a perfect extension of that is now this huge, like, oh, do we catch all the corner cases, oh crap, it doesn't work in this one place, and now we got these bug reports and these guys who were doing this thing and like you know and so if the colonel could just add this feature all of that complexity goes away and there's like I said so this is a thing that people have done it's not like you can't do it it's just it turns what would have been a

very simple problem into a very complicated problem and so that's why it would be great if the colonel could do it kernels are great at that yes that's why we call them colonels because they're like a colonel in a military thing where they I don't actually know I like it case keep going keep it out in post where does the colonel keep his armies shoot for it again, Casey. It's like the kernels are great at that. Yeah, yeah, just like, we're going to take it May Night, beep. Yeah.

So are all these things you're describing effectively gone in things like JavaScript or in any interpreted language which has a VM because then the VM can go, well, I can just insert and interrupt right into the bytecode. I don't even have to worry about it since I'm processing these things. Do a lot of these problems effectively because the Chrome debugger is actually shockingly a great debugger. Well, okay. Well, I mean, you've got to remember you're debugging JavaScript. There's not, like, a lot.

I've had a lot of, I've been not that happy with its accuracy in catching breakpoints and things like that, actually. So I would not categorize it as a great debugger, personally. Really? I would say it's a pretty mediocre debugger, in my opinion. You don't even program JavaScript. I do. I programmed a lot of JavaScript.

I already know that this is a trap. This is an int three. I am not, I am not getting. bamboozled here, Casey. I have used that debugger a bunch. I had to use that debugger on electron even Prime.

So I used it locally local electron fun. And that was a huge ball. Like that was great. That might be different. I'm not. It's the same. But I've used both of them. It has a, I feel like

it is not reliable. It's not reliable at a couple of things. Restarting and catching breakpoints when a web page is re-lure. I have found it to be fairly unreliable, actually. And that's a very important thing because if you're trying to bug stuff that happens at startup, right? Like at load time. It's just, and so, you know, it's one of those things where I didn't, when you use a debugger, you get this feeling of, is it solid or is it not solid?

Right? Not the SOLD, like acronym SLOG's thing, not LISCOF substitution principle. But like, is this thing reliable so that if I set a breakpoint, I know for sure that this code only got executed when the breakpoint got hit and no other time. And I definitely can guarantee you that was not true of Chrome. I think I can actually explain that one. That one I think I can explain because if you use the keyword debugger inside JavaScript, you'll actually get it every single time. But if you use the actual Chrome debugger to do that,

it uses the CDP, the Chrome Debugger protocol. And so it has to do some sort of acing communication with your process, which if it loads fast enough, you're going to miss that entire plane. I'm like 99% sure that's what's happening because I've ran into a bunch of these issues working on that protocol. Also, now you run into the issues, Prime. at first. Yeah.

Okay. It is a great one if you use it correctly, right? I mean, come on. But that you shouldn't have to, right? The point of a debugger to save you time. If I have to go research why this thing can't set a breakpoint, and it's like, oh, the thing that's supposed to set the break point, you don't set breakpoints that way.

You have to use this other thing that's not how you set a break point to set a break point. Like, then it's, then that's worthless. Because, again, I could have just unprinted, which, by the way, is what I do. I do console. Dot log when that kind of thing happens. So this is why I say, I totally understand people. who like go I don't use the debugger. It's like yeah because when the debugger is unreliable, it's not useful. Yeah, for me I just try to not write bugs in JavaScript and that seems like

all right. Yeah, I don't have any bugs. So, you know, that's why I don't use deborer. I don't ever have any bugs. So I'm not really sure. Like it's cool. If you have bugs, I mean, you could use one, I mean, I don't know. I get it. Yeah. Why Prime might need one. Yeah. Yeah. That's because I always forget no mistakes. It's just like if I could just add that to the end of my AI prompt, everything would be great. See, that's the problem. You got to put it in the agent instructions, so no mistakes isn't every one. See, make it secure, make it good, don't put it in a Firebase bucket with no authentication. But also, if we could expound on why I don't think the Chrome debugger is very good. So going further along those lines. So again, in order for a

debugger to actually be valuable, it has to do a lot more things than just stopping where I tell it to stop, which already it didn't do. So the other things it needs to be able to do is gather and, like, Marshall and display information to me cleanly and quickly, right? And it also doesn't do that very well. Like, yeah, it kind of has a watch window that's not absolutely horrible or anything. But it can't do any of the kind of stuff.

Like, I love like all the rad debugger features where you can kind of go like, hey, here's my data structures. I want to turn that into tabular data. So like walk this link list for me and show it to me like it was a table and like all that stuff. Like, you can't do any of that stuff as far as I know. And if you want to do that stuff, you'd basically be like programming JavaScript.

You know, you'd have to be like writing your own, like, JavaScript stuff to do that, right in there, which, again, is taking me more time. I just do a console log. And so I think, like, yeah, like, Chrome debugger is pretty mediocre. I'm glad they built it, but it's not, it's not in the top tier. Like, it's not an S-tiered debugger by and it's right. Oh, we got to come back with the debuger tier list.

Casey, we're going to do that offline and we'll publish a video. So, Ryan, actually, could we end with some of that? Can you talk a little bit about, like, the cool rad debug features? because like some of that stuff is really cool. Most people don't know you can view like 3D models in Raddebug, right? Yeah, yeah. So the 3D models, like some of it's early. So the 3D model, it's solid, but it's, but it's, it's, I need to work on each, each

visualizer is like a complicated problem that I have to dig into later. And so, but I, but I do have a 3D visualizer. So basically the way you can, the way it works is basically what, um, the debugger's got a watch window. And in the watch window, you can, you can, you can enter expressions and they, show you, you can expand them and all that kind of thing. You can also extend these expressions with visualizer rules. So if you wanted to view like some, like some, the very common example is viewing like pixel data or bitmap data. If you wanted to say, for example, like,

I have this pointer to some giant blob of data that's just a sprite sheet or some, some, you know, font, Atlas or something like that, what you can basically say is like, okay, I want to say that that this is a bitmap and then I want to specify some extra information about like what kind of bitmap it is, like how wide is it, how tall is it, what pixel format is it in. And the expression language like in the debugger watch window just allows you to express that kind of thing. So you can say like bitmap and then the first argument, it's like a call to bitmap and you can say here's my pointer, here's my width, here's my height. Those are just expressions. So they can come from variables in your program. And then you can say here's my format. And then you can expand it and it's

going to show you an actual preview of the image itself. You can also like pull it out to your own tab and like see like, oh, like here's a fully featured at the limit. Like once I'm done with it, it'll be fully featured. But it's like, pull it out to a tab and then it's like, oh, here's a bitmap view. I can pan around. I can zoom in, check the pixel values and all that kind of thing. Now every visualization feature in the debugger is sort of implemented in this way. So another common one that debuggers will often support is like a memory viewer, like a hexer. view of some memory. It's exactly the same feature in the Radd debugger where you can say, okay, memory, let me call memory, here's my pointer, here's my size, you can say like number

of columns, number, all kinds of parameters that you want to do for building this hex viewer, basically. You expand it, it's just a memory view of whatever parameters you specify to it. As Casey said, there's also a geometry one. So if you, if you, uh, you have to specify some pieces of information, um, where you can say like, here's an index buffer, here's a vertex buffer. Here's how many indices I have. Here's how many vertices I have. I don't know if you need to specify the number of vertices. I have to go look at it.

But you can basically say this kind of information. And if you expand it, you just get the 3D representation of like, here's all the geometry data that you actually wanted. And because this is in a watch window, and it's not just like parsing a format that you dump out of your program, you see it update live as you step. So you actually like step over as these vertices are being built and you can see the vertices change in real time.

So it's way more useful than, like, dumping it and then, like, dumping it to an OBJ and then looking at, like, some OBJ viewer, like this 3D model or whatever. Same thing with texture data. And so there's also a bunch of other visualization features, like more basic ones, like viewing multi-line text, for example. It's something that every debugger has to do anyways to show you source code, but it's like you should just be able to apply that same exact path to, like, just viewing any arbitrary textual data.

The disassembly viewer is also implemented this way. It's just a visualizer. And then there's a bunch of different rules that you can introduce to change how the watch tree. So when you put in like a watch expression, if you expand it, you know, you get rows generally, they're formed by looking at the members

of whatever type you expand or something like that. There's a bunch of ways you can customize exactly how that looks like as well. And that's sort of more like sort of what Visual Studio did with NatViz. It's a very similar kind of, although it's not an X, you don't specify that in X, in the red fucker butt, but you can do like the same kind of thing there.

And then I guess the final layer, I could dig into more if of any of that, but the final layer on top is you can also automatically associate certain types with certain visualizer expressions. So you can go ahead and say, I've got a struct, it's called bitmap, it's got a pointer, a width and a height, and a format. And I'm going to say, if you evaluate this type anywhere in the program or anywhere, like, if you're in the debugger and you're evaluating anything of this type, just automatically say it's a bitmap visualizer.

So you're sort of extending type information. You can put this directly in your program. You can put it in the debugger configuration, like really wherever you want. And then any time you evaluate that in the debugger by default, it'll show you that visualization automatically. So that includes hover evaluation. So if you hover the bitmap variable in your program, you'll see the bitmap just pop up. If you put it in the watch window, it'll just be the bitmap. Like, it's, you know, that's sort of the final layer to it where you can basically extend your program's type info to be automatically associated with this.

visualization information. Now you can always go back. Like if you're like, I don't want to look at it as a bitmap. I want to look at it as a struct because that's what it is. You can always reverse it. There's a, there's a way you can do that inside of the visualization, like, uh, or within the expression syntax as well. So, um, but yeah, there's lots and lots of stuff you can do. Um, and it's, that's, that's like the, that's a good first run through, I think. It also loads fast. Yeah. Yeah. I mean, it's, not it's not hard to beat visual studio can't beat visual studio

so it's more difficult than maybe they think you know so just throwing it out there I feel like it's a cool feature is that it actually loads yeah well also like some of the things that Ryan was talking about just then are important in the way that I was trying to describe before in other words

it's really important to remember that a debugger having a feature doesn't matter if it takes you longer to use that feature than printf debacle would have been to find it, right? Or we can extend, like, the phrase printf debugging, we're using that as shorthand because, like, for bitmaps, you wouldn't be printf debugging,

but you would be doing a similar thing, which is, like, dumping the bitmap multiple times. I've literally written this code before I had Rad Debugger, where, like, in image processing code, I just insert, like, dump image here, dump image here, dump image here, dump image here, dump image here, and I look to see, right, like, where did the problem happen? And I'm like, oh, okay, right?

And so it has to be faster than that, Because if it's not faster than that, there was no point to the debugger having this feature. And so some of the things that Ryan's talking about are way more important than they sound. The ability just to view something as an image in your debugger isn't enough. It's really crucial to have the ability to also say, look, when I built this program, and this is a really cool feature for a debugger, when I built this program, I just included this little macro that I put that sticks some stuff in the type information automatically when I build the program

that just says any time I'm looking at my struct-bit map structure, it should automatically set it up so that I can view it as the bitmap. That is like one little line of code I put in and then for the rest of anyone ever using this program inside Rad Debugger, they'll always get it shown as the bitmap automatically with the right width and height and all that stuff, right?

And that is a massive time savings for everyone. Whereas if it just has this way of like, I load a plug in and then I instantiate the plugin and I say that I want it bound to, like, if I'm doing that every time I have to view my bitmap, it's not saving me the time that I actually need the debugger for. So all of these things are wicked important and that's why it's so cool

that Raddebugger's taking that approach of like we actually care about saving you time, not just we cared about writing on the thing, it can view an image, right? Features can view image. They got the thing on the front page. That is not enough, right? It needs to be I can view

and debug image processing faster than if I was printed up. That is the actual feature you want in the debugger because otherwise why are you using it. Yeah, you'll see demos of that exact feature to not to not I don't want to pick on anyone but you can go look at like demo videos if they're like look we can view like images inside of our debugger and in the demo video it takes like 10 seconds to load or like 30 seconds even to load I can't even remember what it was but it was just

like it's not it's not just 30 seconds and then you can see it it's 30 seconds every time you step like it's not updating like right away so it's it's just like how is that helping me like um yeah so and also another good example of printf that's not print-f exactly, it would be graph-vis. Like, if you log a bunch of, like, if you print off a bunch of graph-fiz, like, textual data, you get, like, a pointer graph view. You can, like, print out a pointer graph, and then you get this graph structure that you can visualize and see, like, oh, is this tree wrong here, or is it here or wherever? I haven't implemented this visualizer yet, so it's not in the Radibugger yet, but it's like, that's the kind of thing

where I'd like to push it, where it's like, no, you can just, you can just see your pointer graphs update in real time as you step, and, and so. So yeah, that kind of thing would be really nice. We have to write in it. We have to write a real language, though. I know. I'm sitting here thinking the whole time like, man, TJ made me do elixir and now I can't change.

Like, now I want to, now I want, I can't change languages just to use a debugger. We have observer in elixir. We have observer. It is actually good. Like it actually is... Elixir is not a compiled language. It compiled...

Compiled to a VM stuff. Yeah. And then it runs. But the observation tools are very cool. cool because it actually is set up to observe everything across your distributed system, which is like sometimes kind of difficult. They can be running on other computers and everything, right? Like it's not so, not so simple to do, but Elixir has very nice tooling around this. Well, that's kind of a bummer, but like, yeah, any, any compiled language you could be used around it,

like, you know, you could use Rust or something, right? You don't, it doesn't have to be C++ frame. If it's compiled and produces debug info, then it can do it. So, you know, you could be using it with, Ginger Bill was on, you could use it with Odin, I believe, just fine, right? Ginger Bill, well-known debugger hater, he said it doesn't work for Odin, so. I doubt that. He also hates package managers and LSP's as well. No, that's just a joke from yesterday. Okay. I don't worry about it. I haven't know it's false, too, because he literally put in special features into the Odin compiler to work better with the Raddum. Yeah. He was literally in chat five seconds ago saying, I'm going to go watch from the

beginning so that I can see what's happening. I just wanted to say I use Rad Debugger every day. Yeah, that's what I was going to say. I get to this part of the episode and he's going to be mauled. He cannot wait for the comment. Oh, okay, they were joking. I'm about to get a very angry email and then he's going to watch the rest of the episode.

It was in his BS, it was his better stuff for BSC comp. He literally played. He was like Rad Debugger. He announced that it was coming to Macoss, which it's not. So that was. back off the pressmaker. So I don't know. He basically is forcing, this is a well-known, like,

Gabe Newell tactic. He will just announce something so that then everyone has to ship the thing because it's been said publicly, right? And so, like, he took a page from that playbook as like, well, if I announce it's coming to Mac, it's got to come to Mac. That's TDD, right? Talk-driven development.

Yes, oh, yes, yes. PDD, press-driven development. Yeah. Well, I'm very happy, Ryan, you came on the stand-up today to announce Linux is going to be happening in the next few weeks, so thank you very much for a day-me-mass. That's crazy. You heard it here first. Yeah, you heard it here first. And last.

Ryan said Tim Sweeney personally told me that we would be- Yeah, there you go. Yep. Yep. Yeah. Well, thanks for having me, though. It was a lot of fun. Yeah, this is great.

Thanks for explaining how complicated stuff. stuff is. It's so ridiculous. What you think it's going to be easy, right? It's like, oh, I'm just stepping over this line. It's like, nope. Yep. Yep.

The recursive one really blew my mind because that actually just, I mean, that one I was because now you're counting, you're like ref counting basically, right? Yeah. Yeah. I thought the stack pointer thing was really clever. Do you use the stack pointer effectively, like save the original address at that time? And then until you return to that stack pointer, you're like, I'm somewhere within the recursive realm.

We do have that path, but we actually figured out something cleverer for the recursive stepcase. Because generally like... Nice try, buddy. Nice try, but actually you thought of something not stupid. Okay, first off, own. Second off, correct. All right, so tell my dumbness how you do it then, Brian. I remember when I was naive, I thought that might be an option.

Actually, you know what? I want TJ to explain it really quickly. Yeah, any time a guest tries to politely say how they did something, Teage is just like, this is an opportunity to make fun of Prime. Yeah, that's true. For me, stepping's very easy. I just do left foot. Right.

That's like what I was showing earlier. I literally did a live demo. I was like, here's how I want. I'm stepping right now. Okay, so what is the cleverer way? Yeah. Yes. So the clever way you can do is,

instead of checking the stack pointer on every recursive, like every recursion downstream of wherever you wanted to step over, you would hit that trap and have to check the stack pointer. What you can do, and so recursive calls get very expensive to step over that way. What you can do instead is you can basically say, when you call into the recursive call, or really whenever you call into anything, because it could call into one function,

which calls back into the same function, it could be one of those like ping pong recursions. So anytime you call into anything, I want to catch whenever that call returns back to me, which I can't rely on the code address to do that, because again, it'll hit that trap first. So what I can do instead is basically turn off all the traps that I've set to catch this step,

and then I'm just going to completely fake an instruction pointer on the stack. So it'll be, because when you call, it pushes the return address of the instruction pointer onto the stack, and then when the call completes, it'll jump back to that call address. Yeah, yeah. So it's super nice because then you can say, after this call happens, it's going to have pushed some address onto the stack, which is the real return address. I'll romp it. I'm going to fake that. Nice. And then whenever they return, I'll get an exception, not from an int three, but from, I just executed some rent, like, whatever the bogus address was. We happen to use 911. So we just like, we just say, like, so we put 911 and there is the return address. And then we'll get an exception that's like, oh, like, I don't even, I can't even access that page. Like, there's no code there. I'm trying to access address 911. and to the debugger that means oh you've finished the call

move back to the original return address and then reset all the traps very clever nice question that I have two questions I have actually so the first one is alright hold on I gotta pee I have to pee

oh my god okay hey Prime nice shirt did you guys see a shirt it's pretty good I didn't see the shirt just when he comes back you'll laugh okay Ask your questions. It's okay. Prime can watch it.

Watch it back later. Well, can you go grab, like, when they edit this together, grab video of Prime still sitting there going like, Oh, smart, very smart. Right? So it looks like he's watching the whole time. Well, at the tower, because we used wireless mics, I think one or two times he went pee with the mic still on,

and you could hear it in your ears. Trigger and assert when I click here. That's sick. Which is pretty good. That's courtesy of low level. Hey, can we move at least the line down? Dude, oh my goodness. Prime! Prime! Oh, God, Prime! He's doing it! No! That's literally like the naked gun, right?

Like, that's literally a scene from the first naked gun movie. Yeah. And we, I had in-ear monitoring, you know, for like our mics to make sure levels were fine and stuff. So, like, I'm just sitting here. I'm sitting in the middle of the room and I just hear him. And I'm like, what is that happening? Yeah. Okay. So my first question is, what happens, because I have no idea, what happens in that scenario with like the return address, the special return address stack? Does it actually, is it smart enough? Is the CPU smart enough to go like somebody modified the IP on the actual stack? So I'm going to like go modify the return address stack? Or does the CPU,

always, I've never looked at this before. Does the CPU always check when it pops? You might not know this because maybe it just works. When it actually goes to do a ret, does it just check the return address stack and go, if it didn't match, then I don't do it or something? Or, do you know? To my knowledge, it always will just take the address and jump to it.

So it'll, it'll, it'll, and I don't know for sure because it did just work. Okay. We're basically, we're like, let's just phony this address, and then it just went to it. And we were like, okay. it went to it. Ed says yes. 100% you're right, Ryan. Low-level TV confirms in the chat.

Nice. Confirms what? Exactly what you said that. It doesn't check. Ah, okay. Doesn't check. Then you say it just does a pop. I don't know. That's what it just said.

But it has to check. So, just to be clear, inside a lot of CPUs now, so there's the stack as you think of the stack, meaning the actual program stack that we push like arguments and stuff on or like stack variables or whatever the correct. app, right? Like, things that don't fit in registers when we're passing or any other things like that. And we push the

IPs on there, but that's not where the IPs to return to actually come from. They come from a thing called the Shadow Stack, right? Which is like another thing that's kept as like a separate, right? You know what I'm saying? Anyone? And there's also a thing called... Like, motor nether? There's also a thing called a return address, but there's a bunch

of things in the CPU that are tracking this information. It's not actually from the stack anymore. Like, that's kind of like old news, right? Low-level TV is saying The Shadow Stack is rarely used They go on the real stack But it's there So what happens if that is

What I'm asking is what happens if that is on? We should just get Ed in here And he should just explain it Okay Next episode Next episode That's true Yeah

I want to know what happens if you I want because I don't know Like I assume the CPU Actually has like Some kind of a path for this Or something Meaning like It would look and go like

Oh someone did modify the stack so I can't just assume that it's I don't know like I don't know what they did there and maybe I can probably just go read it like it's probably in the Intel Architecture manual right so I will go do that after but I'm just curious

if you ran into it. It's probably in the 3C. Yeah I didn't I did not run into that okay so it just works basically I did the naive thing yeah I did the naive thing that assumes it is on the stack and then it worked I only know about the shadow DOM so shout of Don.

Well, because there's stuff like that where like if you do code pages, right, if you do self-modifying code, for example, that doesn't really work anymore unless you invalidate the instruction cache, right? Like, that was a thing that actually broke because of having like multiple hours. So I didn't know if there would be something like that with all these additional ways that return addresses get checked because they just assume that no one's patching the return address. And so I didn't know if that was an actual assumption. And so you would be like, oh, yeah, we had to call the like invalidate the return address. address stack thing or whatever, right? And or if it just works. And I guess the answer is it just works.

My second question was, would a data break point work for that? You got to expand on what a data break point is. If you set a break point where you said this location on the stack, I am setting a break point if anyone reads it or something like this. Yeah, I guess you could. I guess the reason why you wouldn't want to is to keep those slots open. for user breakpoints. Yeah.

So I didn't think of doing that, but I think you probably could do that. I don't know if it would catch the, like, will that catch instructions that aren't literal reads, but just mechanisms that the CPU uses to pull the return address? I'm not 100% sure, but it's good idea. Like, with the, like, Wrette maybe bypasses the data break point, possibly, right? Yeah, I'm not 100% sure. I don't know. I know.

I know nothing, but I'd like to give my, I'd like to shoot my shot as well. Could you push, like a fake stack frame on that has? as a single instruction function that is just an int three or whatever the break is, and then continue on and let it go. And then when it comes back, it has to bypass through that function no matter what. In one sense, like, so we actually started this idea. Back when I was prototyping stepping, we started with that idea. We didn't move to 911.

Just like, yeah, we tried that. Dumb. Nice tribe. How could I hear it be in the nicest way since I'm the guest? He literally said they started with it. He's basically saying that Prime's idea was good and it was the same idea as them. And Teague is like, well, that means Prime's an idiot. All right.

I guess he doesn't know anything. I'm collecting L's right now. Yeah. How did you get there from that? He said they started with it. Like, it was their idea too. Casey, you have a great point. You're very smart.

Shut up, T.J. T.J. just does not want these Ls to flow down for you. He can't handle you having to win prime. He can't handle a W in the prime corner, can he? I'm literally wearing it. I love T.J. T-shirt. Oh, my God. Just to, like, help, like, bolster his confidence, and he's still thunking on me. True.

Yeah, so we did start with that. We did push an extra frame onto the stack. But we just, instead of moving to 911, we just allocated a code page where there was an int three and then move and then put it in that we changed the return address to that but you can save that extra code page injection like it it kind of gives you the same effect but when you move to address 911 instead you don't have to inject that extra code page and treat in threes differently like it kind of just gives you it's just a little bit simpler version of the same idea now you wouldn't i don't think you'd want to push further onto the stack because

then you're kind of changing like you know how deep is this how deep is their stack are they close to a stack overflow, are you doing things that are like, you know, you're kind of changing the way the program is running a little bit too much, I think, in that case. I mean, you could do it, though. Well, yeah. You would probably take that approach prime if you actually had some things you needed to do. Like, if you had a really good, like, like, a reason for injecting more code, that would probably be the right thing to do, right?

Yeah, so this is, you can actually test this too. and... Let's go. This is actually how Visual Studio, like, if you're in Visual Studio and you want to evaluate a function, it'll actually do that. So inside the watch window, if you say, like, I want to evaluate this function, or in the immediates window, probably. You want to evaluate a function to evaluate it, but it's a function from the program.

What it'll literally do is take your selected thread, and it'll push a new call frame onto it and call whatever function you wanted to run. And you can actually test and verify this. You can put, like, oh, call. Call Sleep 1000 or whatever, and then if you pause the thread, it's in sleep on top of whatever code you were actually executing. So it doesn't inject a thread. It just uses whatever thread you had injected or you had selected. So that actually does. You may not realize this, but you just insulted me gravely by saying that I have the same level of ideas as Visual Studio, which is a little bit...

That's the meanest thing anyone said to him today. I'm so sorry. Visual Studio, same idea, man. I want to be clear, I would never. I would never. Okay, there's a line between friendly banter and proper insulting, and I would never cross that line from my good friend Brian. I'm so sorry.

I did not, yeah, I didn't mean to offend. Can we get the Primogen subtitle changed to Visual Studio developer at the, like, at the... He never shows his, even though I write good ones for him. I know that's because I actually use my real camera as opposed because if I didn't, I'd actually get the leggy version of me seeing myself talking. So I have to use my real camera. All right. So debuggers, everyone, they're awesome if they're good and they're useless if they're not.

Right. That's the TLDR for this stream. Is like if a debugger is awesome, you should use it because it saves you a lot of time. And if it's not, it's just a waste of time and you can keep doing your print-deaf debugging because it would be just as good. There you go. Well, I actually do, there is something inside of me that always comes to the same, like, kind of conclusion, which is I've also not used enough, like, Vim dapping. I haven't dapped hard enough with the boys yet.

Is that when they do the thing like this? Oh, Bill Gates does this, which is my favorite version of dapping. But that sometimes I think maybe my speed at a debugger is also skill issues that maybe I need to actually invest more into learning the dapp side of things and the navigation. being fast with it. But I also don't run into that issues very often. So it's like it's been this kind of a difficult thing. But I need to give pry. Pry, by the way, is the elixir version of some level of debugging.

So I'd like to pry it up and drop that print-f debugging after this. Because you guys make debuggers seem so like this exotic place that I got to go. But I also don't do game development or C++ or any of those things. Why don't we do it when you next time you're going to be doing a project in a compiled language? Like whenever that next happens, right, we should do a street. where we're like, we'll do some debugging. Like, we'll show you what we normally do. As you can just see, like, here's how we do it.

Let's do a good C. Let's do, like, I wouldn't do the devil's language C++. Let's do the Lord's language C. Okay. Or Odin. Yeah. Yeah. Oden could be a great. Oden's like, is actually trying to make stuff good for rad debugging.

So let's do that. Both of those will work. Right. I will say, though, the thing, if you're not in like compot language land like this, when your language has a rappel, you can get some of the experience. experience from, like, when you're running Elyxer and you're running, like, you can run stuff from the Ripple, you can send messages, you can go inspect data, you can do other. So there is like, you know, there are some other options, too, that you can explore even in, in some

of these other languages. Well, sure, and some VM languages have full debuggers, right? I mean, Java does, right? Even what's running in the VM, you can do full debugging in Java. Yeah, I think Python has PubeG or whatever it's called. That's also like that. Cube G is the name of the debunker. In my head, I call PubeG. But I actually can't remember what it is, but it's like PUG. It's pronounced PubG. It's a video game.

Yeah. It's completely, unless you're playing something completely different than everyone else has been playing, probably. Pure G underground? Crazy. All I know is when I saw it, I had such severe dyslexia that all I saw was PubG. And so now I can't unsee it. And so I'm just like, oh, it's Pube G.

That's what the part. Python what is. That's all I know. I, for some reason, I played Player Unknown's Battlegrounds for many hours and somehow never thought PubeG. And now I will. I'm pretty sure if you were on the stand-up when you were playing it at that same time frame, it would be top of mind, you know. We should actually do, because I've never played PubeG, we should do like a Casey introduces

us to PubeG call. Oh, God. I mean, it has been so long since I've played. it. I would be the worst. We just all get shot immediately would be what would happen. I'd be like, well, I didn't get a chance to show you anything, but sorry guys. But we can try. I like that. Sure. I'd try. Next, stand-up, let's just play video games for now. We're just playing PungGee. That's actually much closer to a, you know, a fang level startup anyways. Friday stand-up, time for some team building. Let's pick a video game. All right, folks. Team building stream would be

so funny. Okay, well, we'll talk about that offline. I got to go. I got another podcast. Oh, sorry. Our bad. Podcast, T. Sorry. I did have some good ideas. I wanted to be clear. Yeah. I appreciate that. Hey, Ryan, where can everybody find you at?

You can, I'm on Twitter at, or X. I'm linking at right now. It's Ryan J. Fleury, F-L-E-U-R-Y. And then I've got a substack that I have a couple of blog posts on, on like, debuggers and how they work. And then I've got a bunch of other stuff. that's our Flurry F-L-E-U-R-Y.com and then I think that's basically everything

Where can they find the Rad Debugger? Oh yeah You can go to GitHub.com slash It's kind of a complicated URL But it's GitHub.com slash Actually, can I put it in the chat? Yeah, yeah, yeah, yeah.

You can also just search for rad space debugger or Epic space rad debugger I usually in Google it will come up Yep Yeah, I guess I don't have Oh, studio chat, here we go. I just see Yeah, someone already linked it.

It's Rad Games extension, rad debugger. Your fans are in the chat. Yeah, you got a lot of fans. Is a return address from your substack 9-1-1? Ha-ha. No, but it maybe should be. Yeah.

Yeah, that's a good idea. Prime. You did it. I got one. He said it's a good idea. I got one. Take that. Universe.

All right, everybody, thank you very much for joining us. Casey, Computer Enhance.com. want to actually learn how computers work. TJ, he streams, by the way, and the name is the Prime Gen. Thank you for joining us. TJ, that was a great outro. I nailed that. That was pretty damn good.

Thank you. Thank you. Thank you. Thank you.