Embedded - 209: Debuggerception

Episode Date: August 3, 2017

Pierre-Marie de Rodat (@pmderodat) joined us to talk about how debugger software works (and what compilers tell the debugger). Pierre-Marie works for AdaCore on GNATcoverage (among other things). His github repo is pmderodat. Note that the AdaCore sponsored Make with Ada competition is running right now but you still have time to enter! Last year’s winner, Stephane Carrez with EtherScope, made an Ethernet monitor for an STM32 board (github). GDB supports Python scripting!?!!!  DWARF is the most standard debugging data format. Before that it was stabs. To see this information in a Linux or Mac system, use objdump. (It is really interesting!) Foundation by Isaac Asimov

Welcome to Embedded. I'm Elysia White. My co-host is Christopher White. Apparently, some of you like knowing about the inside of the compiler. So we've invited Pierre-Marie Derrota to talk about the inside of a software debugger and get even further. Hi, Pierre-Marie, welcome to the show.

Hello, thank you for having me here. Could you tell us about yourself as though you were on a panel at a conference? Okay, so I'm Pierre-Marie Derrida. I'm a software engineer at a company called AIDA Corp. So on a technical standpoint, I'm a free and libre open source software advocate. I like working on tooling in general and especially software development tooling,

compilers and static and dynamic analysis tools, such as debuggers, code coverage, and that kind of tools. And yeah, that's pretty much it, what I love in software development. I had an engineering degree in a French school called EPITA and I've been working at ADECO for four years now. And I'm working there on topics that I love, that I described earlier. And I'm mainly working on

host tools in native configurations, but I also have to work sometimes on cross-tool chains, so I have some contact with embedded development. Excellent. We're going to ask you about many of those things. But first, we want to ask

you for short, short answers. And we will, of of course be asking short questions in order to get them. That was backwards. That was totally backwards. Maybe we should ask him to just give us answers and then we'll generate questions. Sure. I don't think that would work. So, first question.

Command line or IDE? In the IDE world, I would be working in IDEs, but it's hard to change and I still work on command line. Favorite language? That's a tough one. Let's say Ada to be an advocate. Technical tip everyone should know. To be on topic,

GDB has a Python, uses Python as a scripting language for extensions. It's really useful and powerful. Yeah, we've been using that. It's cool. Yeah. Sounds like I should try it. Oh, it's your turn. It's my turn.

Favorite keyword. Let's say package. Package. All right. Import. Yes. Almond croissants, chocolate croissants or plain croissants with jam i would say um if you say toast all right favorite movie or book or other media that you encountered in the last year?

I think this will sound like a classic, but I just read the foundation book from... Oh, yeah. Kazimov. Yeah, from Isaac Kazimov. And that was a really good discovery for me. Cool. All right, let's get into bigger questions.

People who come to embedded systems from the Arduino world and people without a software background don't necessarily know what a debugger is. A compiler, okay, that translates source code into machine code. We don't have to talk about the linker. That's too deep. But what is a debugger and why does it matter? Can't we just use printfs? Yes.

There's so many little questions in there. Okay. Well, first, I think it's a good introduction to debugging in general to acknowledge that, well, programs, once they are ready to run on your target, your computer or whatnot, it's essentially a black box. I mean, it gets inputs. Sometimes it produces outputs,

but you don't get to know really what's happening in there. And debuggers, well, I think there are tools that turn these black boxes into some kind of glass. You can see the execution happening. And so when things go wrong, like when you have a bug, well, you can actually look at the inside of your program and find where this is going wrong and where is your bug. It's kind of magical. You go from trying to figure out what's wrong based on what it's telling you to actually being able to see inside. So I like your black box example.

This is really how I feel. And I think people can relate when you have your program, it's yours, you've written it, and it's not working. And if you don't have a debugger and you've used one previously for example i don't know if if you're working on a system that doesn't have a debugging probe you feel powerless and you're like oh i i have to recompile over and over again flashing the the leads to hope to understand what's going on. And with a debugger, it's magical. So you mentioned earlier about cross-compiling, and you just mentioned a debugger probe.

These are more embedded topics where in software, usually your debugger is running on the same system that you're running your code on. But in embedded software, often your computer compiles your code, and your computer has the debugger, and you have to use JTAG or SWD in order to probe into the system. And I like that, and we're actually going to talk to some folks from SEGGER in a month or two. So we're going to talk about debug probes and how that path works later. I want to focus on the software debugging. And that leads me to the question of, well, how does it know about my program?

Does it just interpret my executable? That doesn't seem right. Yeah, indeed. Actually, I like to think that you can think of your debugging, I say your debugging technology as a kind of pipeline. You have the front end. So I'll take the example of GDB so it's indeed GDB is a software that runs on your computer on your host computer and

you have some part of it so if it's a native program I don't know Firefox everything is in your computer but if it's in your cross-configuration, then you have a small piece of debugging technology that runs on the target and that communicates with the host program that is your debugger. But I would say people could disagree, but the most complex and smart part of the debugging technology is happening on the host okay so let's talk about that so yeah it's a bit of a mystery right to some people how this works because i mean one of the great things about debuggers is you don't have to run your program at full speed and then try to pick up the pieces afterward. You can step line by line or instruction by instruction.

You can leave it sitting there at a particular line and go have lunch, think about it, and come back. So how does that actually work? How is the debugger interacting with the executable in such a way that it's got kind of full control and the executable doesn't even know. Yeah, this is the most fundamental part of debuggers. Well,

talking about native programs, generally the operating system enables debuggers to stop the execution of a program to inspect it. So this is, I would say, the first basic operation is to ask the program to be stopped, interrupted temporarily. The debugger can also ask with system calls to have the program to execute only one instruction. And the last piece is the debugger can modify the code

of the running program to put instructions that will stop the execution of the program and get back to the debugger. So this is the famous bug prints. I guess you're familiar with them oh yes and also well in in the budget programs sometimes you else well you rather use your hardware you have special registers well they are generally called debug registers, and you can put the address of your code or of the place in the code where you want to stop, and the hardware when it will try to execute this address it will stop and call out for the debugger. So this is the third major step for debugger.

And so this is how you can control the execution flow of the program from the debugger. Okay. Does that make sense? Yes, yes. Okay. But the way you're describing it, where there's the ability to stop and to start and to step through,

and even the debug registers that allow you to have an interrupt that causes the system to stop and have a breakpoint. Those all make sense, but how does the debugger itself know about the program? How does it know where main is? How does it know that my variable x is in a register or in memory? How does the debugger know anything about my program other than the ones and zeros that are in the executable yeah this is this is the really interesting part of the debugger and the the part i prefer personally as a as a debugger developer um so the debugger has two ways to get information. The first is to directly interact with the program and generally it means being able to read and write registers and memory.

So this is how the debugger can know about the state of the program, where the execution is happening at the specific moment. So to get the program counter basically. And on the side, if you want the debugger to know about your program, to get back to the source level you have also to provide debugging information to it and this is where you generally for example for GDB you pass just the the ELF binary and this is where the debugger will read the list of all the source information where which functions are available, which variables where, and so on.

So using these two sources of information, the debugger will understand the state from the process and interpreting it in light of the debug information. And so the ELF file, this isn't my.out, is it? I mean, in Linux, it would be.out, in Windows,.exe. Yes. Oh, it is that, okay. Yes, yes. The program that you normally just execute,

it's the file that contains the program. Well, actually, it depends on the tool chain. On Linux, generally, debug symbols, they are in the same file as your program, but they can be in another file. On Windows, for example, so you have your.exe, that is a program that you want to execute. But the debugging information is in a file called.pdb. And in Linux, one reason you would want it in a separate file is so that you could have production code that was harder to debug. And yet still not...

And still the developer would want to be able to debug. So you'd release the... I don't even know what the word... Undecorated? Stripped. The stripped version. You would release the stripped executable, but you'd still have the ability to debug

using this separate file. Yes, that's true. And, well, yes. So making it harder for people that don't have the debug information to debug is one reason. And another reason is that debug information takes sometimes a lot of space.

Sometimes it doubles the size of your executable. So you just want to distribute debug information only when needed. And so for those of you keeping score in the embedded world, you can bet that the debug information does not live on your processor. Pretty much ever. Yeah.

Okay, but back to native compiling. So is there a name for the debug information? Is there a word I should know or something? Is this where DWARF comes in? Yes, exactly. Actually, there are... So, debug information, it's like a principle. It's the compiler telling the debugger how to interpret the state of the program. And there are several formats that were created to encode this information. There is an old one called Stabs that was used in GCC. It was a default in GCC quite a long time ago. And it's not very powerful.

You can't describe complex things on Windows generally you what you use in with the Visual Studio 2 chain is code view and dwarf is one of these formats it's used mainly in the GCC and LLVM world. It's very expressive. You can think of it almost as a language. And yeah, it's even Turing complete. So yeah, these are the main debug formats. So formats in which the compilers encode this debug information using this. Okay, this is interesting because I've never really known about Dwarf

except it's the thing that the debug stuff goes in. So that's embedded in the ELF file. The executable or the separate debug file. You said it was Turing complete. What does that mean? I'm used to things being symbols, like, okay, my variable x is at offset whatever, and that kind of thing. But it sounds like this has actual scripting built into it. What does that do? What can you do with that? Or what does the compiler do with that? This is fascinating, sorry.

I agree. Yeah. If you enable optimizations in your compiler, your variables, they are not going to disappear. But say if you have a loop variable, like an index, the compiler doesn't have to put the value of this variable in a register or in memory. So because it could be more efficient

to directly add the offset to some address, for instance. So if I take an example, if you are iterating on an array, you have an index that you increment when the loop executes. So yes, the compiler doesn't have to preserve, to keep in memory the value of the variable, but it might be possible still to recover the value. And debug information, if the format is powerful enough, you can express there the different steps of computation that enables the debugger to recover the value, even though its value is not present anywhere

in the state of the program. That sounds really useful. Why don't I have that? Well, you probably do. Are there debuggers that are better at that than others? Because I've noticed with, for example, IAR, if you have optimization turned all the way up, stepping through is like just pachinko.

It's balls bouncing all over the place. But GDB seems a little better at that. Are there debuggers that are better at that, or is it just my imagination? To be fair, to be fair, the important word is that's possible in debug information. to be fair to be fair no it's so the important word is

that's possible in debug information but really the debugging is compilers and debuggers

working hand in hand the more sometimes the compilers will do optimizations and then they won't

if they do optimization too hard or if not enough effort has been put into debug information generation well the debug the compiler won't know how to compute the values of your variables anymore. And so, this is where in GDB you will get, so you will try to print a variable in optimized code, and it will say optimized out, and you won't be able to see the value of the variable. So, DROF enables to describe complex computations to recover the values, but the compiler has to provide it,

has to tell the debugger how to do it. So it's definitely compiler dependent. Exactly, exactly. And well, there are efforts put into this, having good debug information with high level of optimizations, but it's really hard. Yes. So my compiler takes my.c file or.c++ or.eta

or whatever compiling file source I have. And if I have debugging turned on so that it generates the Dorf information, but I also have optimizations turned on, in a perfect world, the compiler would be able to generate, would be able to create code itself in this Dorf language

that would describe how debugging should happen. And in an unideal world, the compiler doesn't generate anything useful, and my optimized code doesn't run the way I expect. Is that a good summary? Exactly. Yeah, this is what happens. And yeah, even to take another example that people can relate to, the more optimization goes on, the more the compiler shakes the generated code. So some instructions to implement a statement move, we'll get out of the loop,

or we'll move up or down. And in the debugger, you will see like, execution goes to line 10, and then 8, and then 11, and then goes back to line 8, and then 7. So the more the compiler transforms your program to be fast the more the execution will be different from what you intended previously and when we say the execution will be what you intended

we mean as you step through line by line various instructions are going to bounce around it's not that you wrote a program and it doesn't do what you want. It's that at the instruction level, when you're debugging, it might not do what you expect. Exactly.

And more precisely, actually, the compiler is allowed to do whatever it wants with instructions as long as it respects the semantics of your program. And this is where compiler people will tell, be careful with undefined behavior because we as humans, we think the program is going to do this because we say plus one. But if actually we reach an undefined behavior, the compiler is allowed to do something completely different. As long as it does what you told it to do, it can do whatever else it wants in its free time.

And if it thinks you're stupid, which compilers often think I'm stupid, it will do what it thinks it needs to do in order to do what I told it, but it won't necessarily do it the way I told it. Yes. Well, it's not because they are mean, but they will do this if you break the law.

Like if you break the law. Like if you're in efficiency, let me remember what's an undefined behavior. I think if you have assigned integer and it's already equal to the maximum value allowed for this type and you do plus one on it, I think this is undefined behavior. already equal to the maximum value allowed for this type, and you do plus one on it, I think this is undefined behavior.

And if this happens, well, the compiler can do whatever it wants to. Well, execution is not defined anymore, so it could very well erase your hard drive. Well, the compiler is going to do that. I thought you said it wasn't mean. Mean compiler. Yeah, as long as you break the law that if the system does say,

no, you're not allowed to do that and you do it anyway, anything can happen. This is the rule of thumb. I was going to ask if you had a favorite debugger, but now it sounds like it's the compiler and how it generates this debug information that's important. So maybe I should ask, do you have a favorite compiler? I work almost all the time with GCC.

So I wouldn't be able to say that something else is my favorite compiler but yeah LLVM is a

pretty good competitor I would say I don't know I don't have practiced

a lot with other compilers so I'm not able to tell more. But to complete what you said, it's paramount that compilers generate good debug information.

But on the other hand, it's very important for the debugger, well, first of all, not to have bugged itself, and to provide interesting features around that. What kind of interesting features? Well, I told about the Python API. Well, this is a good example. You can have that even if the dwarf information is not complete or I don't know. And if it has, well, good features.

So a feature that every debugger on the earth, I guess, has is being able to print a stack trace. So the various stack frames that, the chain of calls that led to the current state. Does it make sense? Oh, yeah. I'm very familiar with the stack trace and what happens when you trace the stack

and it says that you are at 0, 0. Beep! So that, yes, that is an excellent feature. And when it goes bad, it can go really bad. RTOSs make that, or all operating systems, I suspect, make stack traces a little more complicated because you have different threads. Right, yes.

And some are better than others, being able to switch between them. I won't mention the other compiler again. I was going to say, Chris, do you have any opinions about a particular compiler? Let's just say GDB is pretty good at switching between threads and something else. Basically, you're in the running thread and that's it. I guess I wanted to ask how that works, actually,

because I know we wanted to stay native, but on target, how does multi-threaded debugging work? How do you tell the target I want to... Does that require a special piece of the RTOS that speaks to GDB? This isn't really a fair question for him. Sorry, no. No, no, no, I can try to answer that.

I'll first start with something I approximately know, which is in ADA you can have... can have tasks are first-class citizens, so they are part of the language. Okay. And GDB knows about where to look at in memory to determine the list of threads. So the debugger has to indeed to have some knowledge

about the let's say the task table on the target. It has to. Okay. And I've heard so this is from an engineering point of view

this is not terrible because that prevents somebody from creating a new RTOS and to make it work like this in a debugger. The person has to enhance the debugger first. This is not terrible. And I've heard that people in the GNU community have started to formalize some kind of,

it would be like debugging information, but for the runtime, something that would say, okay, so you want to know the list of threads, go there. So that a debugger wouldn't have to know every RTRS and language out there to be able to display the threads. Okay. So is this an extension to the Dwarf debugging information,

or is this something else? I don't remember. No, no, it's not really Dwarf. I would say it's complementary information that, it's not really Dwarf. I would say it's complementary information that isn't encoded in Dwarf. But I wouldn't... Because it's part of the operating system, right? So it's more of...

Yeah, not really about the program, but it's not clear whether it belongs really to the language or something else. So I wouldn't bet on it. But anyway, it's more into debug information. Is there debug information? Is there information intended for the debugger that I can read? I'm thinking about map files and list files and I know those are

compiler dependent how they come out but most compilers seem to have something like that. Are these dwarf features? Are these debugger features? Does the debugger use them at all or is this just output for the human eyes? I'll talk about dwarf because of course this is what uh i know the most um dwarf is a binary format so uh you won't be able to open it in your text file but it has uh you can use, I don't know if you know the OBIJ dump program. Yeah, well, it is able to output the, to decode the binary information and to present you in a text fashion. And actually, this is what I, as a compiler

and debugger developer use to check that the dwarf is okay and that whether it's the compiler or the debugger that is wrong when things are not working properly. So well you can use this dumping tool if you're curious but really it, if you consider yourself as only a software developer that wants to make programs and not toolchains, it's not that useful. Well, I say that, but now if you know it and you as a user see that things go wrong, it's always, it could be useful to have a look and say, ah, yes, I see. So this is this variable that is not described properly and to report the debug to the project that is responsible for that. That brings up a question Chris noted down. How do you debug a debugger?

That seems, I mean, it seems a little recursive, but what tools do you have? Do you have, you know, GDB on GDB? Exactly, exactly. Oh my God. This is exactly what, so the, I guess if you have, if your debugger is in an IDE, so you have Windows,

it's easier because, well, you have one window that you know. Let's say you put it on one screen. It's the debugger that you want to debug. And on another screen, you can put the window that is the debugger of the debugger. But actually, I already had to do that. I had some bug in GDB. And so, indeed, I ran GDB and GDB.

And to keep things sane, I put the prompt of the GDB that was debugged in red and the prompt of the GDB that was debugging in another color so I could know where I was standing. But it's possible to do this in only one console.

It would be a little confusing, but okay. So what, what program was the GDB that you were debugging? Running. Debugging. When you were debugging the GDB. Sorry.

I figured I could go on like that for like another half an hour. Yeah. It's inception. Yes. Debug exception. It was a Jumi program that was just trying to reproduce a bug. In this case, it was an ADA program, probably with some complex type that GDB could not grasp yet.

All right. that makes sense. How many, I want to say, how many GDB bugs have you found? But that's sort of not the right question. How many have you written? Having a debugger that is solid is really important because I have to trust it. But it is a piece of software, so there are going to be bugs. Oh, yes. How do I figure out when the bug is me versus it? I mean, this is always a problem. I hear

about people with compiler bugs they've found. And sometimes it's true, but it isn't as true as often as I hear about it. What kind of things would I need to do to start getting into GDB bugs? I mean, it seems like they should all be corner cases now. Yes, actually, the bugs I see generally are not... What we are talking here is a very specific kind of debugger bug. It's like you print a variable and GDB is displaying some value,

but really it shouldn't be that value. Like GDB has been wrong decoding the states this kind of bug well this happens not well it's very rare and generally a debugger bug will will look like Oh GDB has crashed okay this is not the program that is buggy, it's the debugger, clearly. Or it will be like you print a variable and GDB will say, what variable? No, no, this is nothing. There is nothing called like that. So, well, as a programmer, you're like,

oh, no, no, I have two problems to fix. But you know, it's not your program that is at fault, it's GDB. Otherwise, indeed, this can be a problem. But generally, if you manage to get that, you can always... So if you print your variable, it says 1, whereas it should say 0. Well, you can recompile your program with a printf that will display the variable and you'll know which one is right. Given that the Dorf debug information is compiler generated. How often do you have to figure out if the bug is in GDB,

in the debugger, or in the compiler's debug output information? Well, fortunately, it's also very rare. Generally, the kind of bugs I fix is something... It's not that I have a wrong result, it's that I have no result at all. So it's the example of, okay, I'm trying to print a variable, and the variable does not exist, but I know it's there, so this is obviously a compiler or a debugger bug.

So this is very, very, very rare. And when that happens, well, that's the kind of bug that requires you to be more focused than usually. You have to... This is where you have to take a lens and inspect closely what the compiler and the debugger do. And you get a bunch of dump files to see the various steps of compilation and debugging and to identify on a simple example

where things are getting wrong. This is like an exploration. You have to see it like this. Yes, most debugging is to some extent. Yes. So before we leave the topic of debuggers, I did want to ask about something you mentioned at the top of the show, Python scripting with GDB, which you mentioned is quite powerful

and I know people I'm working with do some of, but what sorts of things can you do with that and why would I care? The first thing that comes to my mind is something very useful, at least to native programmers, that is called pretty printers. So imagine you are working with a C++ program that uses vectors from the STL. In a debugger, if you will try to print this variable, you won't see the contents of the vector from a logical point of view like okay this is a vector of integers it contains 10, 1, 7. Now, what you will see is a record with all the gory details of the implementation of the vector.

So you will see what the developer of the STL will see. Does it make sense so far? Yes, so far. So as a user of this vector library, you don't want to see that. And so one feature of GDB is the ability to provide special Python scripts that will be called whenever a vector will be printed and which will be able to say, okay, this is a vector will be printed and which will be able to say okay this is a vector the data is there there there and there and prints only the

useful information in this in this stack of records structures and pointers and arrays and so on so as user, you will print the vector and see the list of integers between brackets, and you will be very happy. Gosh, I've been doing a little bit with OpenCV and trying to figure out what I want to do when I move it from Python to C++. And it sounds like if I want to be able to look at intermediate images, I could

use Python to be able to read my buffers and then put them and actually display intermediates. That's a little bit beyond the vectors you're talking about. This is showing images, but that's easy in Python. Would that work? Yes, I guess so. Well, then maybe another option in this case maybe would be to call some debug function that is present in your program that I don't know would take a buffer in parameter and will dump it as a png image on the disk that would be another option but yeah an option would be to to write some python code that will get the data inside the buffer and then use some magic python library to output an image, even to open your image viewer to display the file in one single command, you can do that.

All right. I just wanted to make sure that I understood how this was going to be useful to me. But I do want to change topics a bit because you also work on code coverage and you work for AdaCore, which of course, Ada is their language. I mean, not theirs. Ada is their preferred language. I don't know how to say that, but what do you do there?

Okay, so this is the first program I worked on when I entered in AdaCall. So yes, I'm working on a program called GnatCoverage. That is a code coverage tool and I'm working on various aspects on this program. It's a program that, strangely, is not unlike a debugger in that it works hand-in-hand in the compiler because it has to know about the program, the source-level program, to produce a meaningful report,

a coverage report. Okay, what is a coverage report? Yes, well code coverage is a tool that is essentially useful to spot untested on an unused code. So for instance, if you create a function that has an if-else condition in it and you write on the side a test for it, if the only use case you're testing is that the if condition is true, well you have some part of the function that is not tested. And a code coverage report will tell you that. It will tell you, oh, these statements, they are never executed.

You're not testing them, or maybe they are even useless. You could remove them. Maybe it's a feature that it's implementing something that you didn't want to implement on the first hand. So, yeah, this is what code coverage is about.

All right. I do like unit tests, so that makes sense. Is this only for Ada or is it language specific? Well, code coverage is a concept that applies to potentially all languages. And Gnet coverage is the tool from AdaCore. It works on Ada and C. And it has some specificity that because it works really hand in hand with the compiler, we had to modify the compiler to make it work well.

And so it's unfortunately for now, but we are working on some other alternative. It requires to have a specially modified compiler. That's a little unfortunate. Yes. that's a little unfortunate yes um but not uncommon i mean you have a a tool that is well integrated into a tool chain so yeah i can see how that happens at the risk of reopening the gdb a gdb uh kind of question again how do you verify that your code coverage report is accurate? Well, there is no formal proof. Like, okay, I have proved that in all situations,

the report is correct. Well, what we do is, like most projects, we have a test suite that we try to make as exhaustive as possible that checks that the code coverage computations work as expected. Okay. So you start with something that you know executes a certain way. Yes. Either on paper or from some other tool,

and then you verify that you get the expected result from your new code. Yes. Either on paper or from some other tool and then you verify that you get the expected result from your new code. Yes. Okay. I was sort of hoping that he had some code coverage generator, coverage tool debugger that inceptioned back to. No, I wasn't expecting that. Well, you're almost right

because at some point to make sure the tool was, let's say, powerful enough, we computed the code coverage report on Gnet coverage itself. So you are not that far from the truth. Well, this is what's funny when you are developing software development tools, is that you can use them on your own tools.

How did it do the first time? I mean, how was the report, the coverage report initially? Well, for code coverage report, it's kind of easy because you don't have to have a working code coverage tool to start writing a code coverage tool. So it works with the most basic programs. Now the hard part is how do you write a compiler in a language

that has no compiler yet? Yes. But for CodeCoverage tool, fortunately, we're safe. When we had Fabian Choteau on the show last year, AdaCore was having a competition, Make with Ada. Is that happening again? Yes.

Make with Ada, the second edition, is currently running. So it's still a contest for Ada programmers on embedded targets. And unlike last year, where it was, I think, only focused on ARM targets, this year it's allowed for any target. So it could be AVR, PowerPC, Spark, Leon, or whatever. It started mid-May and it will end on the 15th of September.

So it's kind of late, but not too late if people want to subscribe. It's a good thing to keep in mind that the jury, what's important is quality over quantity. So there is one month and a half left before the end of the contest, but you can still do something that can be appreciated by the jury. And what contest entries last year were your favorite? That may be imprudent. Which ones won? No, no, I think I haven't had personally a look at all the entries last year,

but there is one of them that I keep in mind. Stefan Kares wrote an Ethernet driver for a chip on an STM32 board. And so, yeah, he wrote the full stack, like the Ethernet layer, the IP layer, and the TCP layer, which was really a nice piece of work.

Everyone should have to do that once. In Ada? Yes, in Ada. And he, he puts a lot of efforts that once in Ada yes in Ada and he he he put a lot of efforts into documenting

why well documenting his design decisions that was instructive that sounds

pretty amazing actually yeah all right any advice for people who are thinking about entering this year other than That sounds pretty amazing, actually. All right. Any advice for people who are thinking about entering this year other than quality over quantity and you don't have a whole lot of time, but you have enough time if you get started soon?

Well, it's pretty much it. If you focus on good development practices, like good documentation, have a good test suite and compute code coverage or use formal methods. It's really a good point and well, that's pretty much it.

All right. Yeah. Seems like a good place to wind up the show. Okay. Pierre-Marie, do you have any thoughts you'd like to leave us with? I would say the devil is in the details. This is probably where your bugs are. So don't hesitate to dive into how your programs work to understand and to find the bug.

I like that. Thank you. And this is a pretty fun way to program, I think. Yes. Our guest has been Pierre-Marie Derota, software engineer at AdaCore. Thank you so much for being with us. Thank you too. It was a great pleasure to have this talk. And thank you to Christopher

for producing and co-hosting. And of course, thank you for listening. I have a quote to leave you with this week. And of course, it's from Ada Lovelace. As soon as I have got flying to perfection, I have got a scheme about a steam engine. Embedded is an independently produced radio show that focuses on the many aspects of engineering. It is a production of Logical Elegance, an embedded software consulting company in California. If there are advertisements in the show, we did not put them there and do not receive money from them. At this time, our sponsors are Logical Elegance and listeners like you.