Embedded - 334: Bag of Heuristics

Episode Date: June 11, 2020

Matt Godbolt (@mattgodbolt) joined us to talk about assembly code, becoming a verb, 6502s, exploring compilers, and application binary interfaces. Compiler Explorer can be found at godbolt.org. The code is on github (compiler-explorer/compiler-explorer).  Matt also has jsbeeb, a BBC Micro (6502) simulator. You can try it out at bbc.godbolt.org. Its code and more information is on github (mattgodbolt/bbc-micro-emulation). Matt recently gave a video presentation about jsbeeb for ABUG.  Some other videos that may be of interest: CppCon 2016: Jason Turner “Rich Code for Tiny Computers: A Simple Commodore 64 Game in C++17” Matt’s YouTube videos on using Compiler Explorer Just enough Assembly for Compiler Explorer - Anders Schau Knatten CppCon 2017: CB Bailey “Enough x86 Assembly to Be Dangerous” CppCon 2017: Carl Cook “When a Microsecond Is an Eternity: High Performance Trading Systems in C++” The best compiler book seems to be The Dragon Book. Hyrum’s Law on writing interfaces. Application Binary Interface (ABI)

Welcome to Embedded. I am Alicia White. I am here with Christopher White. Our guest is the return of Matt Godbolt of Compiler Explorer. Hi, Matt. Thanks for agreeing to come back. Thanks. Thanks for having me again.

For folks who haven't heard your show since it's been two years. What? I know. Could you tell us about yourself? Absolutely. I'm the creator of a website that I, I think I alone in the world, although you just did, call Compiler Explorer. But foolishly, if you create a website that bears your last name, and you have a kind of interesting last name like mine, people don't know it's yours. Everyone else calls it just Godbolt, which is the weirdest thing. So anyway, I've created a website. But as a hobby, in my day job, I am a software engineer for a trading company, and I've been in trading for 10 years. But prior to that, spent a few years at Google

and then just over a decade making video games, which was a lot of fun. As you said, you named the site Godbolt, godbolt.org. No, I named it Compiler Explorer. Everyone else was saying Godbolt. I'm not that hubristic. But now if somebody is arguing and maybe going to use the site, your name is used as a verb. Yes.

Yes, that is something I have come to terms with now, having tried to fight it back. But yes, people talk about God bolting their code or putting it in God bolt. I've been in a committee meeting with the C++ standards people and I say, hey, can we just put it in God bolt and see what happens? I'm like, I'm in the room with you. Stop it. Well, we're going to talk more about what the site does and why you might want to God bolt your code. But before that, I want to do lightning round, which is going to be new for you. It was on sabbatical last time we had you on. Are you ready?

Oh, gosh. I'm ready. Favorite language? English. C++. Favorite compiler? Oh, gosh, that's a difficult one. I'm going to say GCC.

Favorite bolt? Oh, the hexagonal ones that, no, I don't know. Favorite 6502 instruction? Oh, it would probably be SAX, which is one of the undefined opcodes. It was super interesting discovering how that worked. Stores A and did with X, but only through accident of the way that CMOS works. Least favorite 6542 instruction?

There's some undocumented ones at Holt, but yeah, probably BRK, Brook. I don't like tax. When I was a kid and I came across that in disassembly, I always thought, what is it doing? It's taking my bits away. It just takes 10% of everything you do away. Do you have a tip everyone should know?

Alt period is the thing I would recommend. It brings back the last element of the previous command, and in some shells it will cycle through them. It's super useful to be able to make quick edits to command lines. I've never used that. Does it work in Bash? It does work in Bash. Oh. Yeah.

It's one of those things, yeah. And it works in, like, you know, the Python interpreter. It will do the same thing because they all use Readline. And it's super handy. I mean, you do something, and then you want to do a different thing to the same thing, like, you know, ls the file and then rm the file, and you just alt-period it back out.

And you can keep banging it, and it keeps sort of cycling through the previous commands, the last argument. And in other shells like Phish, which is the one I use, it kind of does like a search for things. You can type in a little bit of something and then hit alt-period, and it'll search through your history and kind of go, oh oh yeah, anything that ends in.json, you probably meant bar.json over here. Super useful.

All right, well I've learned enough. Yeah, thank you for being on the show. Actually, we want to talk some about Compiler Explorer, Godbolt. For people who've never heard of it, what is it? So it is very simply, it's a web page that has a code editor on the left-hand half of the screen and the assembly output of what happens if you throw that code through a compiler, assuming it actually compiles okay, and you get an annotated sort of disassembly of what your code turns out to be on the right-hand side.

There's some syntax sort of highlighting on both sides, and there's a little sort of coloration that shows you, hey, this part of your C code became this part of the assembly output. And so it's a kind of useful way of saying, how does my code compile? What does it compile to? What happens if I fiddle with the compiler flags? What if I do my dash O2 for most of the optimizations

versus dash O3, all of the optimizations or other flags? So it grew out of a need that I had to try and prove to people at work that we could use some of the new features of C++. And they were like, oh, no, it's probably slow. It probably makes bad code or whatever. And so I wrote like a shell script that did most of what Compiler Explorer does. And it was so useful to sort of end arguments that I decided to hack something together on a web page. And GCC Explorer, as it was back then, sort of eight years ago, was born. Since then, it now supports a lot more than GCC and a lot more than C and C++. But yeah, it's used to sort of determine how your code turns into assembly,

what the computer sees. Which you could always do with most compilers, but it's sort of a pain in the butt. And you have to find the right flag and then look for the intermediate file and then... The list. Doesn't everybody keep the list file open with their C file? No. I mean, absolutely. I mean, these have been around since the beginning, right? You know, as you say, the list files, you can use dash capital S. I mean, that's all that Compiler Explorer is doing behind scenes it's just running that that those commands for you but it's inconvenient and i mean i think there's a sort of an irony or at least

something which shows that there's a hole in our tool chains that it's so painful to generate the assembly code and link it to the code that you're actually writing that you can't in your ide or your editor do that easier than pasting it into a web page, sending it halfway around the world and having me run and compile it for you and then send back the results. It just says that there's something missing. And it is way better.

I mean, as soon as you put it in a web page, it looks nicer as well. There's that sort of thing going for it. And the highlighting is very helpful. But I don't have to put my whole program into your system. It works on functions at a time. I'm looking at it right now, and it came up with the default int c, int add to, int n. And so it's just going to add what is it? I have no idea what it's going to do. Let's see. For i

equals 0 to n, it's going to add up all the digits from 1 to n. Right. That must be what you must have had before. The default is to return a square is what the, like if you completely clear your history, but we try and remember what you must have had before. The default is to return a square, like if you completely clear your history, but we try and remember what you did before and show it. So maybe you clicked a link where somebody was demonstrating a compiler optimization that they'll do some tricks like that.

So that's a great one, in fact. Is it a function which takes an input of which number to count up to and it sums up to that number? Yes. So if you put it on the high settings and you pick an old-ish version of GCC, so like GCC 8 or whatever, you'll see that it does the obvious thing.

It sits in the loop. You know, we've all written a little bit of, enough of assembly to know that you'd sort of set your count, loop counter, you'd go through, you'd accumulate one, and then two, and then three, and then four, and five, and add them assembly to know that you'd sort of set your count loop counter. You go through, you'd accumulate one and then two and then three and then four and five and add them all together. And you return that and you'd be like, hooray, done. This is brilliant. If you then choose either Clang or the more recent up-to-date GCCs, they're smarter.

They just look at that and say, well, there's a closed form solution for this particular problem. If you want to know the sum of digits from zero through n, it's n, n plus 1 over 2. And that's what the code will generate. It'll be slightly different from that for a couple of boring reasons to do with overflows. But that's the kind of thing your compilers are now doing for you behind the scenes. And obviously, this is a trivial example of it, where the compiler is doing something where you perhaps wouldn't think to do this. But it actually flows naturally out of some of the optimizations that compilers do within all loops.

So compilers are super smart. I think that's been my overriding lesson in working with Compiler Explorer is that there are some super, super smart people writing compilers so that i don't have to be so clever anymore well and as you say looking at different versions of gcc i have one version where at optimization 3 which wouldn't be pleasant to step through i have 46 uh assembly lines and I can't read them. But if I put the optimization down to zero, then my assembly looks very much like my code. And even if I don't know assembly very well, I can start learning it. Look, there's a move. Okay, that must move something to somewhere. Do I use that again? And here's a jump. jump okay i know what a jump is that one's easy oh compare and see even if you don't know assembly you do kind of start to get the feel for it

right and i've i've seen that people have used it as a learning tool like you can take exactly like a rosetta stone you know i know what it's doing on the left hand side so i can probably match these things up and see what's going on as As you say, at O0, that becomes a bit easier because the compiler is literally putting out lumps of assembly that correspond linearly to the code that you've written. O1 is probably a good compromise because there's just enough of the gumpf taken away of storing stuff and then reloading it back in again and then storing it and then reloading it back in again and then storing it and then reloading it back in again

that the O0 generates. But it's a great tool for doing those kinds of things. And looking at the one that came up on my default, which, as you pointed out, I don't remember where it came from, it was x86. It was normal GCC. I would type up at my command line. But I can go and look at ARM GCC, and then I get ARM instructions, which is really kind of cool. make a loop super tight how do i make it really small and how do i change my c code so my assembly code is fast and can i can i keep it in c instead of making it into assembly which nobody will ever

be able to read right and i mean there's there's always a danger with with equating the number of lines of code that are generated with the performance oh i think probably yeah and obviously I'm not saying that that's what you were saying there, but like, you know, you're right in terms of, you know, a small tight loop is certainly smaller. And then obviously in the embedded space where you're perhaps constrained in the code size, that's a super important metric for you to be looking at. And with ARM as well, the instructions are, well, each instruction costs the same amount of bytes.

With x86, it's even more complicated with the variable length instructions and all the strange levels of decoding and weird logic that goes on inside the chip. But it still lets you develop some kind of intuition. But I'm always wary of not making, not representing that the number of instructions should be used as a proxy for the performance.

But certainly for code size, it's useful. And just for learning, you know, like the first time I typed in that from the summation, and I went, wait a second, there's no loop in here. There has to be a loop. I wrote a loop. Where's my loop? It's like, no, I got rid of it. And similarly, if you do like a count the number of bits set type trick,

where you just keep shifting down the input value, and every time the bottom bit's set, you go, you add one to a counter, and you keep doing that until the thing you're shifting down has no one bit left in it, and you say, oh, there are seven bits left in it. Again, that's a loop. Most compilers are smart enough to spot that there's usually an instruction in your isa that can count the number of set bits and it just replaces the whole thing with that you're like this is brilliant i can write code that makes sense to me as a human looking at the code i can

see this is counting the number of bits i haven't got like some crazy inline assembly block with a big comment explaining and apologizing for the fact that it exists and then using some esoteric instruction that I don't really understand and have to look up in a hardware manual. But yeah, I can write that knowing that the compiler will generate the right code. It will do the right thing. And that's just wonderful. But you've got, I mean, I mentioned ARM because that's where I spend most of my time, but you've got Clang and AVR and a bunch of different arms, a bunch of different risks. How are you... I mean, this is a lot of compilers.

What's been added in the last two years? Because this list is super long. I mean, you probably can't remember exactly what's been added. It's about 600 compilers now, which is a lot of compilers. Now, I'm double-counting a little bit. I'll be fair, there's a bit of double-counting, because the C compilers are basically the same as the C++ compilers and they're counted differently.

Well, in the trivial list I can get hold of. But it's a lot of compilers. There's a lot of complexity in managing the infrastructure behind the scenes. I'm a stickler as well for the main compilers being built by me. So I know where they came from. I know what version they are.

I know that, for example, the GCC maintainers and the Clang maintainers use the site as like a source of reference for like, we had the bug, did it get fixed or not? You know, and can I just quickly check? So I like to be able to point to them when they say, well, what version exactly is the GCC 7.4? And I can say, well, I built it from this Git shard

with these settings and this bin utils and this thing, whatever. So there's an awful lot of behind-the-scenes nonsense that goes into managing and maintaining probably around 300 actual compilers, especially when the majority of them are built from source. And the whole site is open source. The whole infrastructure is open source. So you can go, and in theory, you should be able to spin up your own entire copy of Compiler Explorer

should you wish to have around about 600 gigabytes of compilers dumped on your hard disk. I saw that. I saw that you started putting up this privacy policy. And it's dense and yet really useful about you're not keeping my code unless I create a link to it, image case. You're keeping that, but you're not looking at it unless there's an error. Image case, you might look at it, but really probably not. How did that come about? I don't remember that being part of the site initially that was something that happened um oh golly it would have been about two years ago actually

because uh the reason i remember this is for i was traveling in europe at the time you remember when we used to travel oh is it too soon to be talking about this will hopefully this won't make sense if you're listening to this in the recording in the future. But right now everyone's painfully aware of what I'm talking about. But, yeah, so I was traveling. I was actually in Prague at the time, and I was walking around one of the beautiful castles, and somebody tweeted at me saying, hey, I think you're probably in violation of GDPR. And I'm like, oh, please don't say that.

I'm on holiday. And I know that in certain areas of law, willfully breaking the law, like knowingly, sorry, knowingly breaking the law is worse than by mistake, through errors of omission. So the fact that i had publicly been told that i was probably in violation of a law meant that i probably ought to get off my or get out of the uh the castle and go and do something about it and there was a very heated tweet backwards and forwards between uh myself and other sort of people in this sort of position

saying do i do i need to do this now do i just need to turn the site off? And it was pretty fraught. Oh, that would be fraught. It was, yeah. And my wife and kids were like, why is Daddy on his phone? I'm like, well, Daddy's going to wonder whether he's going to try and remotely turn off a website that he may be in trouble about. As it happened, I was okay with what I had, I think. We were still within some grace periods. Fortunately, the re one of the reasons I was traveling in, in, uh, at the time was to, to do a presentation at a company who a friend, friend of mine, um, put me in touch with their legal counsel and them being in Europe, I was able to let me, uh, you know, spend some time

talking with them, the legal team. And they've told me basically what I need to do too. So I was able to make some sensible decisions there. And then when I got back to the U.S. and had some time to think about it, I thought, well, I really ought to have a privacy policy. You know, oftentimes people ask me, what do you do with the code? Ho, ho, ho. I bet you it's fun to look through all the things that are on the site. Oh, no. Look at all the errors. Absolutely not.

People will ask, you know, do you look at my code? I'm like, no, I've got better things to do. That's all I do. I just watch it go by. Right. I mean, you know, but it made me think, right, it made me think that. Obviously, I'm not interested in other people's code. I run the site with two or three other sort of people who have administration access so that I know that if I am incapacitated or, for example, if I am traveling in Europe, I can ring up somebody and say, please turn it off or similar or site updates and things of that nature.

But I thought, well, I'm going to write a privacy policy, but I'm not going to just get a template one. I don't like the idea of starting from some boring written by somebody just to tick boxes. It's like, no, this actually makes sense to me as a document that explains to people what I'm going to do in English. I'm also extremely blessed that my wife is a lawyer. So I was able to write it in my words and then have another pair of eyes look over it but from a sort of very uh forgiving point of view in terms of well that you can't say things like that matt please please change them and it was a very interesting experience but i'm i'm pleased with it and and i think probably i'm one of the only people in the world that's had their privacy policy um

uh sorry people have emailed me with sort of fan mail about my privacy policy, which is a good thing. Yeah, because it's incredibly readable, and it's the sort of thing that I want to hear. And then if I do something like this, I would totally want to steal it. By all means. That's the wonder of open source. So as you say, it is open source. People can run their own. Do you think people are?

I know people are. I'm very pleased to know that. So I used to work at Google. I think I may have mentioned in the intro, I can't remember what I said. I spent a couple of years at Google. And some of my friends who are still there have let me know that they're running their own instance. They use it on their compiler team to check, tweak and fiddle with the compiler settings, the output of their compilers. They build their own compilers internally. I mean, they're based off of Clang and GCC, but they

have their own modifications to them. I'm pretty sure I can say that there are teams at Apple who similarly use the Compiler Explorer internally with their own builds of the compiler or just for their own edification. I know that there are other trading firms that have got instances because despite my protestation to the contrary and me not looking at the code and all that kind of stuff, quite reasonably, you don't want to send your code to some random server if it's proprietary and it's important. So it's useful to be able to run it locally and know that you're not sending your information across the Internet

and you're not sending it to some person who's promising not to do anything bad. So they'll run their own locally as well. And I'm fairly sure, you know, there are people I don't know who are doing it. I really have tried to make it as easy as possible to just be able to git clone the repo and then pretty much just type make, and it should just work. And it's not building anything.

The site's written in JavaScript and HTML and all the horrible things that go with it, but the make file is designed to sort of be like the thing doer that will just bring together all the bits you need and then start the site up. You haven't put all of the compilers on anymore. When did you stop being the person who did all of it and start getting open source help? Good question. So obviously being on GitHub means that folks can find the code and pull requests can be made and issues can be tracked and similar. I think it probably really took off when we started, when I made a channel

on the C++ Slack that was just for Compiler Explorer. And then suddenly I had a way of communicating in real time with people who were helping. And over time, picked up three or four main contributors who have given me some pretty decent pull requests, have been extremely helpful to other users on the Compiler Explorer channel, and have offered really useful

insights into bits that I wouldn't know how to do. For example, like trying to secure the site. I was very naive in the early days and have become less and less naive as time has ticked on. And so if somebody who knows a decent amount came in and said, look, I think I can help you here. Here's how, by the way, you can bring your site down. Would you like me to help? I'm like, yes, yes, please. And we've become good friends. And that's been a useful, useful thing on so many levels. So I'm just really, really lucky. And Rust, you support Rust now. Supports everything. I know. It actually does.

Supports things that don't even exist. I'm looking at some of these languages. My gosh. Haskell, Ada. supports things that don't even exist. I'm looking at some of these languages. My gosh. Haskell, Ada. Clean? That's not a thing. That is not. I know. I made some slides the other day to show all the languages we support, and I have literally no idea what clean is. What is NIM? That's where the mice go. That's pretty cool.

Yeah, I don't know much about NIM, but I think that when cross, it sort of side compiles to C and then compiles to C or something like that. So this is, again, the beauty. Once you become a focal point for these things, people will come along and say, hey, can I add this? And I'm like, here's a document that explains how we add new languages.

If you can send me a pull request, it's in. And that's a great thing to be able to do and a great sort of position to be in. And I'm very fortunate but you and rust specifically i mean that's rust has its own playground on their own site but it's probably the second place language actually in terms of the statistics when people come and and and hit the site that rust is the second second place i like rust i haven't really done very much with it other than my usual writer ray tracer because that's a kind of fun thing to do in a new language.

I think Rust as a language has probably been the kick up the backside that C++ needed. So how many of these languages do you use? I mean, do you still use the site as you did when you first made it that's a really good question and really sort of cuts to the the chase of of a lot of things i've been thinking about how how the site has grown and and how much uh how much i don't use it anymore my current day job i'm spending more time writing python to sort of hold bits of larger infrastructure together and sort of almost comically python is supported but just to show you the uh intermediate byte bytecode that Python doesn't even really compile to parses. But one day maybe we'll use something more sophisticated.

I know people want Java and other sort of managed languages that have like a JIT compiling. But yeah, so I haven't been using the site as I intended for a long while. And that's sort of a shame for me personally, but it does mean that the tenor of the site has changed. And I'm sort of in that position of starting to let go a little bit about the direction of the site and start to trust the people that are major contributors to it and the community as a whole as it's becoming bigger than me. And that's kind of an odd thing to admit, really. And I'm only really thinking out loud now as I say these words, which is really an interesting observation. I need to perhaps let go a little bit more and let it go in its own direction.

That's a tough thing for an open source project that, I mean, it bears your name. Unintentionally. I just like to point that again for about the 300th time. Just change your name to Compiler Explorer and then everything will be fine. Exactly. For a while, my Twitter name was Mac Compiler Explorer, and I got a lot of positive feedback about the choice. But, yeah, no, it is interesting that it's it

the site is morphing in that way and so one of the things we added in the two years that since since we last spoke was the ability to actually execute the code that you've written which sort of dovetails in with the security issues that i alluded to before because there's one thing running a compiler with an arbitrary piece of code that a would-be attacker can manipulate arbitrarily. And it's surprising how big an attack vector running a compiler is. It really is. I mean, there are command options to load plugins. You can hash include files and have the error kind of print out whatever the file's in. So about once a week, some joker would email me and say, hey, you know, you can hash include slash, et cetera, slash password.

And then you can see the password file. I'm like, I don't think that the password file is useful to you since about 1994, I think. Shadow passwords were a thing by then. But sure, you can include any file you like off the whole file system but it's not a real computer it's not my computer but anyway that that that kind of stuff was the level of my experience um but you know like preventing certain um compiler options from being able to be used was one thing trivially preventing certain files from being read off of the disk by you know permissionings and things was another thing. But when you then allow someone to write any code they like and execute it, suddenly you have to take a much more serious look at denial of service attacks, of other issues. And in fact,

probably the single worst thing that could happen, because you could blow open, assuming you could escape the sandbox, you could have access to my my amazon account which is where it's running and i think i've locked it down so that the node doesn't have anything special but you know potentially you could steal the cpus that were on it mine whatever the thing people mine these days um on my on my dime but that's not really that big of a deal right right? The worst thing would be you'd be able, in theory, to look at other people's code, and that would be dreadful, right? I really want people to be able to trust that they can send their code to me and that it's not being snooped by anyone else. So we had to really think carefully about how to prevent that kind of

thing from happening. And thankfully, I've had some support from both friends who are in the security industry and there's just a wider community. But it's a scary thing. You know, hey, run this bit of code on your computer. Off you go. What could go wrong? How do I run the code? So there's two ways of doing it. If you've got the site up in front of you, there is on the one of the two dropdowns for like compile options, I think, is just execute code. It used to be a button, but we've run out of space for buttons. And user interface design is not my thing, as you can probably tell.

So if there are anybody listening who is really big into user experience, please ping me. I'd love to have some help in that respect. So the first thing, you can tick the execute code. The other thing is that you can add an entire pane. So although I said, you know, it's on the left-hand side, you've got the code, and on the right-hand side, you've got the compiler, the site's actually really fully adaptable.

You can move those panes around. You can add new panes. So in more complicated setups, you can actually have multiple editors compiling through multiple compilers, and then having the results of those compilers being diffed so that you can see the diff of the two things running. One of the things you can also add in is one of these execute panes that just has an output window and you can supply input parameters to your program as well. Have you been able to find those?

Yeah, I'm doing the latter one where on the left side it says add new, little plus, and new execution only. I have source editor DiffView. Oh, so you found the DiffView. The DiffView is kind of sometimes very useful, but mostly because even the most trivial change in code

will make a very big difference. It's like butterfly flapping its wings. Most of the time, the output is hugely different. But the DiffView is handy when you've got a single compiler bug where it's just generating the wrong instruction. You can say, look, it was fine before and it broke in this version. But you... So Christopher,

you've been able to find... If something can break, I will break it. Let's just get that out there. That's an absolute skill. It's very helpful sometimes. You're still on C? I was on C++.

Which compiler are you on? I don't know. Something normal. Okay. I opened a new incognito window. We can help you out with that if you want. A new incognito window. And I still don't see it. That's okay. Christopher will show me.

I was going to say, you two are in the same house, right? We're not facing away from each other. Oh, right. Well, he's running a Mac and I'm running Windows. All right. So, yes, everyone should go look at Compiler Explorer, which, if you can't find it, it's at godbolt.org. Which now supports 6502, I should point out.

It does, in the most dreadful way possible. There are two ways it does. There is one C compiler that sort of generates 6502. 6502 is not well-suited to C. And then a friend of mine has a sort of transpiler that takes x86 assembly and converts it into 650 6502 which you can add as a tool which is a bunch of fun he wrote a whole like bat and ball game for the commodore 64 in like the most modern c++ you could possibly write it was a lot of. Is it small enough? I mean, this is one of those things that's like, maybe I need to godbolt it. But it doesn't seem like something modern written in C++ would run on an old 6502-based computer.

I had a C++ compiler in 1990 for my 65816-based computer. It's definitely not something that makes that much sense i think probably the the single biggest trick is the use of constexpr which means most of the sort of the logic and the clever stuff is compiled time evaluated tables are built and then the main loop is very very small as a result of that there's i i don't remember exactly i think if you if you look we can put it in the show notes or whatever but it's jason turner's um i can't remember the name of the talk now, but it's like modern C++ on tiny computers, something like that. Jason Turner of the C++ podcast. Correct. Yes.

Speaking of 6502s, they have a special place in your heart. Everyone's heart. Everyone's heart. Everyone's. Mine's 8051. I'm sorry. It's a dark, dark, dark part of your heart. Oh, wow. But yes, the 6502 was, I was about to say it was the first chip that I programmed in assembly, but it isn't. Actually, I did a Z80 first, or Z80 as I would have called it then, but I've turned to the dark side since I've been over here.

Yeah, sorry. Although the accent suggests that I'm British, which I am, I spent most of the last 10 years in Chicago. Hence, the past makes more sense now. Hence, the Midwestern twang. That's right. But the 6502 was probably the processor I cut my teeth on most in terms of like learning how to actually write programs, interrupt handlers, graphics, games, those kinds of things were all done on the 6502. And as such, it was an important part of my childhood or sort of mid-teenhood was learning that, learning how to write games mainly.

So much so that I sort of recaptured it by writing an emulator some years later. And so that's the only other thing that's... So pretty much anything you type.godbolt.org will go to Compiler Explorer, and then what we'll do is we'll look at the bit before the.godbolt.org, and we'll try and match it against the language or a compiler

name or whatever, and select that by default. So if you go to rust.godbolt.org, you kind of get the Rust compilers first up, and if you go to cpp.godbolt.org, you'll definitely get C++. But if you go to bbc.godbolt.org you go somewhere completely different there is no bbc compiler although it sounds like it could be a compiler actually um maybe for the b programming language no but it's not it's the

british broadcasting corporation um and they made a computer in the 80s that was so important to me. And if you go to that website, you'll find JSBeep, which is a JavaScript emulator for that system. And my other passion. Okay, so here I am. It says BBC Computer 32K, Acorn DFS Basic. There you go. Acorn was a file system? So Acorn was the name of the company that actually produced the computer.

It was sponsored by the British Broadcasting Corporation, but Acorn was the company behind it. So they kind of both of them got their names on that page. But you can now go 10 space print and then the quote is, I think, shift two or shift two, I think, gives you the quote. Oh, sorry shift two or shift uh yeah shift two i think gives you the quote oh because i can't just use my quote it puts up a star instead it's it's yeah it's

emulating it's it's unfortunately it's not as simple as you think to try and like coerce a modern keyboard to be the same shape as that when really the hardware is expecting certain rows and columns to be probed it's like, that key is not actually a key. It's shift and another key. So how do I make that a key? So you can do the obvious thing there and 20 go to 10, of course. Back, you know, when go to was completely acceptable, I still think go to is actually okay from every now and then.

It didn't do anything. How do I tell it to run? Oh, run. Oh, look, there it is. So the thing about this is if you type something wrong, it doesn't say syntax error. It says mistake. Isn't that brilliant?

Which is a little judgy, I think. But it's also several characters less than syntax error. So you've got to save space. I suppose they could have just said idiot and saved another byte. Duh. Duh. So, okay, so this actually emulates the 6502 and it emulates the BBC Micro. Correct.

And you have discs and cassettes I can put in? The default disk, in fact, is the game Elite. Oh, really? Yeah. So if you do shift and then press and hold shift, then just tap F12, which is meant to be the break key. Now, depending on which computer you're on, you may not have an F12, so that becomes difficult.

Disk fault. Disk fault. Disk fault. Disk fault 18. Do you know which one that is? That means that it doesn't think there's a disk in it. Did you try and select a disk? Oh, maybe I selected something else. Load new commander. Yes, no. I think yes. There you go. And can you see the awesome thing that's being

drawn on the screen right now? Hey, it was a big deal. Which drive? Oh, yeah, you're saying to load a new commander,'t but did you see the spinning ship beforehand oh yeah oh yeah this is a two megahertz machine with it looks like multiply instruction with no divide instruction with no real graphics hardware to speak of, doing a full 3D game. That's amazing. Absolutely amazing. And it didn't suck that badly either.

It was quite a decent frame rate. And it's just miraculous. The two chaps who came up with this game were college roommates, and I think nobody told them that it was totally impossible to do this on a 2 MHz machine. And so they went off off and did it anyway which i think is one of the best things in life is you know the the constraints that we we have in front of us sort of set our our our stall but for to be no one's you know like if you'd have asked anybody they'd have said no that can't possibly be done it's great anyway that's just elite my favorite game actually on that is

exile which is a similar deal it's like a full um full 2d physics based um game where you know you solve problems but not necessarily by the way that the game designers intended they just you know you can use physics to get yourself through small cracks that perhaps you wouldn't have been able to get through all sorts of cool things it's just such a cool system and very important to me but this is cheating it says virtual megahertz 10 oh did you hit control and insert maybe uh yeah i don't know if you've just been hammering at the keyboard you've got a faster computer than me anyway i mean it's ridiculous if you just think what's going on so you can actually now download it i the weekend so in fact yesterday there was a meeting of the acorn and bbc user group that i presented how the heck this all works too and as part of that i i made an electron app

that then embeds um jsb so means you can now run install and run a single uh executable that is like the whole emulator. And just to think the technology stack. In fact, there's probably an hour-long presentation in the technology stack of emulating 6502 on modern hardware using this technique because it's Electron app to JavaScript, JavaScript through JIT, JIT through sandbox of the world, sandbox through the graphics hardware. Graphics hardware. CPU.

It's just so deep. All to do something that used to run at 2 MHz and... 32K. This is awesome. There's similar ones for the Apple II that I've used in the past when trying to get my actual Apple II to work

better. But this is very well done. I think that trying to get my actual Apple II to work better. But this is very well done. I think that, yeah, the Apple II over here held the same place in our hearts for people of a certain age, should we say. It was just that thing that was around that was super cool. Oh, am I supposed to ask more questions? We're still recording, so. You understand, I have to be the commander now. Amanda Jameson, by default. Okay.

Actually, this is good because we have listener questions. I can go back to playing games. Keep on playing and just read out things. Works for me. Krusty Auklit actually reported a use of CE when he was troubleshooting a really bad bug. He ended up proving it was a tail call optimization bug in the compiler using Compiler Explorer after he added IAR support. And then IAR had told him how to, quote, fix it, which it wasn't.

And he was able to prove to them that it was a compiler bug. That's so awesome on two levels. One, because the site was able to be used to show this. And, you know, if you look through, like, compiler bug reports nowadays, almost always they'll have a link to Compiler Explorer saying, yeah, this demonstrates that it's definitely in this version of the compiler. But even more awesome is that Krust Orklet obviously actually submitted the patch to Compiler Explorer to put IAR into it in the first place.

So this is like the community serving itself, and I'm just a happy bystander watching this happen around me. McEnhillock? How would you have said that? I don't know. I don't see it. Where's the notes? These are people that are members of the embedded. M-C-N-Hill-K.

I don't know. Yeah, these are people who are on the Patreon and listeners, and they get to ask questions when I know who the guest is ahead of time. Anyway, he asked if you have a good intro to compilers book. I mean, so my experience of compilers is very much pragmatic and from the other side, like as a user of a compiler. I've written compilers only sort of along the journey as one does when you're like, hey, I think I've got a good idea to write my own language. And you write a compiler for it, and then you discover that, really, no, it's probably best to leave it to experts.

So the Dragonbook is the one that I would recommend. But, yeah, I don't have any extra insight. People often email me and say, hey, how do I do this in the compiler? I'm like, I don't know. I just run them. I mean, that's totally fair. And I kind of want to know, has anybody used the site to support a compiler's class? Because it gives such a wide overview that if you could get people to look at this and to write 200 line programs and

be able to see how the compilers do different things with optimization levels, different compilers, different versions of compilers. It would be a neat way to augment a course. It's so cool. Yeah. And I'm glad to say that at least three people are doing that. I have usually people email me and say, will it be okay to do this? And then I've sort of, I'm always like, yes, absolutely. I think they're worried about the scalability of the site. Will it take 30 people in a classroom hammering at it for an hour long or how long their sessions are? So yeah, it's wonderful that people are using it for that. I know that there are several books that now refer to it. People email me and say, you know, those little short URLs that you can make. So that's another aspect of the site that is sort of like a paste bin for compilable code and its results. So you can get a short version. It's just a godbolt.org slash z slash and a small code.

And how long do those things stay valid for? Because I'd like to put them in my book. And that was a frightening thought. This is actually going to go and be very permanent. So it's something that we think about a lot. All of those short URLs are really important to us that they stay valid forever, which is partly why we're collecting compilers

as badly as we are. The reason we have those 600 is because we can never really phase out an old compiler because somebody may have bookmarked it and we want to use it to say, no,

it was fixed in GCC 4.1.2 or something. That's kind of a heavy burden i mean you're enabling becoming a footnote in many things if you if you were counting your um reference score like they do in the world of academic papers. Your reference score would be really good. I guess so. I'd never really thought of it like that before. I mean, we take it very seriously, this permanence of the URLs and the compilers and whatever. And various members, leaders of the communities have reached out to me before now and said,

are you okay? Is the site going to be all right you know we rely on it so much that we need to make sure that it's it'll it's got a long future and i'm glad to say that thanks to the patreon support that i've i get for the for the site and and a couple of corporate sponsors and some help from corporate folks you know i'm it's well well kept up the the bill the amazon bill is paid and so we don't have to worry about that side of things but obviously a permanence of things like urls permanence of the compilers i mostly by it being open source that means that if i god forbid something dreadful

happened to me that you know someone could resurrect the the site in a different domain or whatever those those short links would be a more difficult thing. But we have them backed up. We've got off-site storage for them and stuff. We try to make it as, I mean, this is a couple of amateur folks doing it DevOps, but we try really hard. It's important to us.

It doesn't have to be something dreadful. It can be that you win the lottery and retire to a beach. It's okay. It doesn't have to be something dreadful. It can be that you win the lottery and retire to a beach. It's okay. It doesn't have to be something dreadful. Oh, thank you, Alicia. You know, Power Explorer will get bought out soon by SoftBank. By Rational Rose. Rational Rose?

Rational Rose. Geez, that's a blast from the past. Gosh, they had... Oh yeah, let's not get into that. Do you think you'll be compiling UML anytime soon? I was going to ask, yeah, where's my UML compiling? Ow. Ow.

I thought we weren't going there. What are some things that, if you had the the time you would you would add at this point oh gosh that kind of comes back to that that point you made me realize earlier that like i'm not using the site myself anymore yeah so it's more difficult to know i know that the things that we have been most asked for are a decent mobile phone experience, which sounds terrible because who's writing code on a cell phone, right? Who's really... The screen's tiny. I mean, you're looking at the site now.

You can see that it's pretty busy. There's a lot of things going on there. The editor windows that I'm using are actually from Visual Studio Code. So Microsoft open-sourced VS Code, and so we actually use the VS Code editor, which is called Monaco, for the text. But they have explicitly said they'll never support mobiles. That's just too much work for them.

So that means that, unfortunately, we can't support mobile. But I need to do something. No one's really writing code, but Twitter is full of Compiler Explorer links. People are saying, hey, have you seen this before? And people just want to be able to click on it and look and see whether or not, oh, this is interesting. I should go and look at this on the real computer. So I've got that to fix.

And that affects me too because people do tweet things at me and I'm like, I'm nowhere near a computer, but I can't even click on this link. This is kind of embarrassing. Another thing people have asked for is to sort of turn it into even more of an IDE than it is already. I'm not so sure about that. I think it's kind of got a sweet spot of small bits of code. But you said, Alicia, you know, people generally put one or two functions in, and that's about the limit the human can really parse the assembly for. So I don't know about that. There are definitely some other ideas. I've been doing some interviewing recently, and there are lots of sites out there that will let you, you know,

collab edit type things where one person can type and the others can see. And I'm wondering whether or not adding that kind of support would be fun. Oh, that would be really, really good, actually. Oh, don't. Oh, now, Christopher, now I'm going to have to, now I've said it. Well, I've had some experience with those things, and they're okay. But a good one would be good. It failed on me. So the irony is i'm in an interview with with somebody and we were using collab edit and

collab edit went down and um then we're like oh i guess we'll just use a google document and so we just used a google document that works you know plenty fine for like the kind of i mean because all interviews are terrible i think we can all agree right now there is no such thing as a good interview. They're all dreadful and just in different ways. But it was sort of embarrassing that the person who I was interviewing knew who I was. And they're like, could we not have this on like another website that does code edits that people use to share? I'm like, oh, shush. But that would be a very big change from an architectural point of view. At the moment, we're very boringly traditional scaling

by just making more servers appear as if by magic from Amazon. But as soon as you have anything where there's sort of bi-directional communication between multiple things, I don't know, I haven't thought about it very much. But it would be cool. It would be cool. But if I were to retire and have all the time in the world i wouldn't have a need to use the site as much so you know that's the kind of problem i

think you know eating your own dog food yeah puts it in a particular direction well i mean i don't think this is a you should put this on your site i think this is aLab should put it on their site. Maybe so. So, yeah, we do. I mean, we have a RESTful API, which means that people don't have to use the site using the front end that we present to them. They can just post and get to particular magical things that are, well, I was going to say well-documented. Nothing's well-documented, right?

That are somewhat documented and somewhat do what they say on the tin. And I know that some people are doing that. There are some plugins for IDEs that let you access the site sort of more directly, and they use the back ends that we supply. I know that at one stage, a security researcher was using us as essentially a, can I compile this code on 300 different compilers and see if there's a particular compiler that generates the code that i want automatically you know with with a script so there's there's a whole bunch of use cases um and yeah so somebody could actually you know collaborate it could could back onto us for example i i keep i'm sorry i'm just spinning around thinking of all the things that that would be really cool for um let's see back to listener questions uh

christopher not my christopher um asked if there is do you have any suggestions about learning enough assembly to understand compiler output for whatever your favorite high level language is for whatever your favorite target is? That's a really good question. Because a lot of assembly books and things talk about the whole instruction set. And if you look at what a compiler emits, it's probably 2%. I mean, certainly on x86, we use MOVs pretty much, and occasionally some other instructions, LEA. And then everything else is just a surprising thing you Google

when you see it in the output. But, I mean, so Compiler Explorer does have some primitive hold-the-mouse pointer over the instruction, and it'll tell you what it is. It's not very good. It's sort of hackily passed out of the Intel manuals. But there are a few talks on YouTube specifically targeting Compiler Explorer, actually.

I think one of them is called Just Enough Assembly for Compiler Explorer, and there's another one called Just Enough Assembly to be Dangerous, which sort of broadly is the same kind of thing, like just enough to understand what your compiler's doing. And so those are probably worth a watch. But I think the thing that you said at the beginning, Alicia, is that compile something on O0 and then sort of squint and look backwards and forwards between the two sides and say, well, which bit is this? That's a move.

I guess I know what that is. That kind of thing, I think, is the best way to get a handle on it is to sort of just work through it slowly and use the built-in support or Google to sort of shore you up a little bit. KBX81 goes on with a question that's sort of related. It's not about learning assembly, but learning what the compiler does, how it uses various registers, how do you do the reverse engineering that if you're going to dig into the assembly to know which registers are you supposed to do or how all that works? Are you just supposed to know the return value is always in R15? Right. Yeah, that's an interesting one, too. I mean, that's mostly the ABI, the application binary interface, which is sort of the contract between various functions.

So if you're calling an operating system routine or another library's routine, that you all agree where all the registers go and what they mean. It used to be straightforward, and it still is mostly straightforward. you can go and look at whatever abi your system is using um i always end up on the same wikipedia page that has all of the information when i'm trying to remember what they are uh i do there's a couple of talks that i've given where i talk about this a little bit so i can give you some links for those to put in the notes or whatever but um the the thing that i would be cautious on that is that compilers are getting smarter and smarter and if they know that no one else is calling a function other than code that that compiler generated it can start breaking the abi rules you know inlining is an example of

that to start with you know one one way of of voiding the contract between i'm calling your function and eax has the whatever and RDI has the whatever is to say, well, I've just actually picked up your function, pasted it in line to where I was, and then I decided to completely rewrite it to fit where I am now. But with things like link time code generation or link time optimization, the entire program is seen by the compiler as a single entity

and it can start making observations like nobody else calls this function which is defined in one dot o file over here except for this function which is described in this dot o file over there which i realize i'm gesturing wildly and no one can see but they're on different sides of the room and the lto system can say oh i can do this now i'll rewrite the function i can make make up my own ABI for this particular use case. And that way I know that I don't have to preserve RDI or whatever it was that I thought was broken because I can look in the other routine and see that it's not being used. Things like that. So it gets a little bit more complicated.

But I think essentially this is a question about the ABI. Yeah, the ABI. I mean, actually, someone mentioned abis in the slack channel and i was like where is that from it's from eabi that's what i always used to think and it was getting gcc to work with this eabi or that one and the thumb and the it was just so hard to figure out which processor needed which ABI. And now that we have cortexes, that is so much easier. This is a pain that I'm glad to say I don't get to see very often. So my sympathies to you embedded folks with all of your different combinations and permutations.

It is something most people don't have to see anymore. But it's one of those things that stopped me from going to GCC right away because it wasn't consistent. And I mean, there's been talks of some languages having ABIs for certain processors, but that doesn't have to, as you said, that doesn't have to be followed. If you believe you are the only one on your processor doing this thing, you don't have to use R15 to return things. You just have to be consistent. Right. And it's that consistency. It's those rules that say, okay, what is used to monitor the return value?

That's your ABI. And they exist. to monitor the return value. That's your ABI. And they exist. You can make your own. Don't do that. But you do have to understand there's a layer. There's the... Actually, we should talk about this.

There's the preprocessor in a C compiler, a C++ compiler. There's the preprocessor in a C compiler, a C++ compiler. There's the preprocessor and then the processor, and then that all goes into LLVM. Is that right? It depends on your compiler. If you're talking about Clang, which is one of the more... Yeah, broadly, the preprocessor takes your code, does all the includes, does all the horrible macro expansion, and then you end up with just a giant, giant block of code with all of your includes pulled in.

So the poor compiler then sees from your 10 lines of code, which was like hash include this, hash include that, printf, whatever, the poor compiler sees the 200,000 lines of code that actually that expanded out to. So then the compiler does its magical thing, parses it, makes sure it understands how things fit together, allocates functions and whatnot. And then in something like Clang, very broadly speaking, there's this sort of internal representation. Well, every compiler will have an internal representation,

possibly multiple of them. But I know that Clang emits something called LLVM, which is this low-level virtual machine, which is kind of not really that low-level, and it's not a virtual machine either. So like all acronyms, it's fallen further and further away intermediate form where types and some amount of provenance information about where this value came from is still expressed. That then becomes like a common lingua franca between all the various subsequent passes of the compiler. Like the inliner can run and it can say, well, you're calling a function here, but I know what you're doing. So I can look at the function that you're calling and I can just bring its LLVM instructions in place. And then I can run other optimizations that like, hey, this is a dead code path. No one can ever get here because I know the range of values are

less than this or more than that. And therefore, this if statement is essentially an if false. So I might as well throw it away. And so on. And compilers are just an enormous, giant bag of heuristics that run over and over again over your code, looking for opportunities to generate better and better, in this case, intermediate representation. And then there's a lowering part towards the end of the pipeline that goes, okay, well, this now has to fit my target instruction set. And so a lot of those previous passes I was talking about could actually,

they're common to all types of processes, you know, like loop unrolling or whatever is the same for any process. So maybe the specifics of how many times you should unroll or not are a tweak, but the code doesn't have to be different between ARM or x86 or MIPS or whatever. You just say, this is the loop on roll pass. Brilliant. But then when you get towards the end of the compiler, you're like, well, now I'm generating ARM assembly.

So I definitely know that when I have an increment, then I'm just going to use an add R0, R0, hash 1 or whatever. And so that's kind of broadly the sequence of passes. And anyone who actually works on compilers, I'm sure is grinding their teeth listening to this, but that's my take of how it all fits together. It's the LLVM to assembly part that the ABI comes in. Right. Yes.

I suppose that's in the context of what we were talking about before. Knowing that the return value needs to go in a particular place is obviously a constraint, as you say, of the target architecture. In the LLVM stage, it's not necessarily obvious that it needs to care other than marking one of the names. So in LLVM, everything's just a name. Let's assume you have an infinite number of named registers. And in fact, you can only assign to registers once,

which is another weird thing. But that's just a compiler trick. But yeah, so then one would be like, well, I'm returning percent result, and then the point at which it's being lowered to assembly, like, well, we have to arrange for result to be in EAX because that's where the return value needs to go, and therefore that needs to percolate backwards through, and anyone who's using result will use EAX if we can possibly use the AX.

Otherwise, we have to store it somewhere and bring it back. But yes, I suppose you're right. That's where the ABI comes in. Okay. I just wanted to get back to that. Yes. Let's see. What else do I have to get back to?

The SACs, that came out really early. Right. The JSBeep and SACS and BERC and 6502 instructions. Yes. You still work on that. I mean, it's still something that is interesting to you, even though. It's 30 years past its sell-by date, yes. I wasn't going to go that far, but okay, yeah, as long as you mention it. Why? Why do you still work on it?

Is it just the joy of what you were learning when you first found it? I think so. It's a huge amount of nostalgia for me. It's a huge amount of nostalgia. It's very, very important. It's formative for me. My first paid job, such as it was, was with a good friend of mine. We used to write the magazine type in programs, you know, and get like 10 pounds or 20 pounds

or, you know, if you were the star one, 50 pounds. And it's just the most wonderful feeling to be able to pull out something from like the early 90s and it's still got your name. They misspelled my name. I'm so peeved at them. But yeah, they're bad now i mean if they had godbolt then it would have been you know but you know that's really important to me that time of my life was really uh really informative and you know push my career in a particular direction i mean i would my my

plan was really to go into to do physics. It was my sort of like stated professional goal was to do. And I ended up doing a degree in physics and I was planning to go down that route. But yeah, yeah, yeah. Yeah, yeah, yeah. He has a degree in physics too. Because the computer stuff was really always there and was always exciting. And so that's partly, I think, why it captures that nostalgia.

But I'm good friends with people who still write games and demos for this computer. And it's just amazing. And I think this comes back to this idea of constrained environments. You know exactly where you stand with a 32K Beeb. It's like everyone had exactly the same hardware. It all worked in almost exactly the same way so any trick you could find was fair game like the sax instructions like well this is a complete this is a flaw or rather this is a side effect of the chip fabrication process the seamless

consequence exactly it's have you um heard of hyrum's law is a chap called hyrum right he works for google i've had the pleasure of um having dinner with him before and you know the two of us are like we both have weird things named after us uh my site and his his law and his law is something like any non-trivial any any function with non-trivial users will become relied upon any anything that one it's called hyrumslaw.com i think we can go hit it whatever but but yeah it's it's so true but like on a on a 1980s piece of hardware it's like well if you can rely on it once you can rely on it every time because everyone's got exactly the same hardware. So there's lots of tricks to pull out. These demo writers will use and abuse the way that the CRTC,

the cathode ray tube controller works to trick out the way that the TV interprets the signal to get more colors, more screens, get smooth scrolling. It's just amazing to think that there is still things being discovered. And in fact, only two weeks ago, we discovered a new weird side effect of something that happened. That's amazing. How come 30 years of looking at this has not found all the things that can possibly exist?

Yeah. I mean, part of it, too those computers that the the the beam and the apple 2 and commodore 64 that whole generation of computers if you if you wanted to you could understand pretty much the whole thing i mean not at a hardware level necessarily some of them had sound chips and stuff but if you wanted to understand the system you could and right there were circuit diagrams in the manuals yeah i'd say so you've got to see that you know that's how that works i see yeah the apple 2 came with this assembly for all the roms and everything it wasn't they weren't

hiding anything and now if you want that kind of experience you can do that with embedded systems some but even now some of the microcontrollers are like i mean just a bog standard cortex m3 is too complicated for one person to understand it's also the the idea that these things are attractable anymore like it used to be like well ram runs at this speed the cpu runs at this speed the third clock cycle does this the fourth you know whatever so as things got more complicated things like caches came into effect and um micro-coded um and out-of-order execution. And these layers and layers and layers of things which came up with more, gave us more and more speed and performance, but took away our ability to reason about the systems.

And not only that, but vendors suddenly realized that those were proprietary pieces of information that shouldn't be put in the manuals or certainly shouldn't be relied upon because they want to be able to change how the hardware works without violating what they had said about how it works so i remember like going through the the first few consoles uh the playstation the playstation 2 the hardware manuals would come out and you could like it was fair game you could poke anything you liked and as long as it works you could go and get on within reason you could get away with it and then buy about the ps3 and the xbox 360 it's like no here's the api you have to call this function to do the thing that you

want to do because we want to change we want to reserve the right to change how it works but that means i can't reason about it i'm not allowed to rely on anything weird that i find it just works the way that you tell me that it works and i i can't i can't do anything which is it makes sense i understand why they would like to be able to do that but um it's it's a sad day as you say we can't hold it in our head we can't reason about everything ourselves anymore. Matt do you have any thoughts you'd like to leave us with? I guess yes I think I read this uh um earlier in the week and it touched a nerve for me because of some of the things I'm doing at work. But consider this.

Every program contains at least one bug and at least one too many instructions. So by induction, all programs can be reduced to a single instruction that still has a bug in it. Yes. Our guest has been Matt Godbolt, creator of Compiler Explorer, which you can find on godbolt.org. Thanks, Matt. Thank you very much. Thank you to Christopher for producing and co-hosting, and thank you for listening.

You can always contact us at show at embedded.fm or hit the contact link on Embedded FM. If you want to know more about that Patreon and the Slack channel, hit the support button at the top of the embedded.fm webpage. Now a quote to leave you with, this one from Peter Pan. All the world is made of faith and trust and pixie dust. Embedded is an independently produced radio show that focuses on the many aspects of engineering. It is a production of Logical Elegance, an embedded software consulting company in California. If there are advertisements in the show, we did not put them there and do not receive money from them. At this time, our sponsors are Logical Elegance and listeners like you.