Embedded - 215: Heisenbugs

Episode Date: September 14, 2017

Alvaro Prieto (@alvaroprieto) joined us to talk about the basics of debugging, from software to hardware. Some of the programmer devices we talked about: SEGGER JLink and Black Magic Probe. Chris mentioned a visual frontend for gdb called "Vulcan" but which is actually called Voltron. (He's got graphics on the brain). How did we forget to mention the six stages of debugging? Alvaro Prieto and Jen Costillo's new podcast on reverse engineering! And on Twitter as @unnamed_show. Alvaro's Cheese Cave: making cheese and cheese-lapse photography of Brie aging.  

Hello, I'm Eliseo White. Welcome to Embedded. My co-host is Christopher White. I mentioned a while back about having some shows where we cover embedded systems basics for the career-minded. We talked to Dennis Jackson last month about the advice he gives to software engineers getting closer to the hardware. This week, we are speaking with Alvaro Prieto about debugging.

Our goal here is to help people thinking about a career in embedded know what that really means on a day-to-day basis. The other goal for the more experienced engineers is to find better ways to communicate this stuff to people getting interested, and maybe we'll share a few tips and tricks along the way, or hear from some of you to get your tips and tricks. Hey Alvaro, how's it going? Hey, good. How about you guys?

Good. Looking forward to the beach this afternoon. You can't start a podcast like that. Well, I am, I am. So it's going to be a short one. Sorry Alvaro. That's okay. You've been on the the show before but it was a while back and uh and things have changed so how how are you introducing yourself these days i'm still a firmware slash hardware engineer and hobbyist enthusiast yeah that's about it are you not gonna bother with cheese or podcasts or

anything else uh i don't usually introduce myself but yes i i do make cheese and uh started a new podcast since i was last on all right uh and that's the Unnamed Perverse Engineering Podcast with Jen Castillo, who has been on our show a number of times. I expect it to be a good show. Is it live to see how it worked. And we have two, almost three episodes. I just need to upload it. They're pretty rough, but the second one was a pretty good interview with Dimitri Greenberg. So, yeah. All right. Well, if we have time, we will talk more about it. But if we don't, people should still check it out.

So, you are familiar with Lightning Round? Yep. Are you Round? Yep. Are you ready? No. Alright, let's do it. Favorite movie or book fiction which you encountered for the first time since we last talked on the air? The Big Sick.

The Big Sick. Really good movie. Kumail Nanjiani's movie. Yes silken valley okay what do ducks watch on television ducks or dogs ducks quack ducks uh ponds no the incorrect answer is documentaries oh wow when baking bread do you use a sponge a starter or just yeast and go for it uh use a starter sourdough starter preferred voltage uh 3.3 why did the duck go to jail to record a documentary because i don't know was selling quack. Oh. What kind of cheese is currently in the cheese cave? Have nine little breeze going.

Sounds like the start of a limerick. It does, doesn't it? Nine little breeze went to market. Oh, no. Nine little breeze in a cave. Okay, let's actually do the show. I ran out of duck jokes. No, I got a lot more. Oh, do you? Oh, go ahead.

What do you call two ducks and a cow? I don't know. Quackers and milk. Alright, moving on. Holding it. So, you've used an Arduino in the the past right briefly yes and you are

i mean christopher you've used one yes i mean it's a goal this is not really an interview more more panel thing which i guess meant that he could have asked us jokes but luckily he didn't know that at that time. Do you ever ask jokes? But yeah. Apparently Alicia does. And I've used Arduino, and it was easy and fun and fast to get started. And there were ways to debug my code there, but it was not pleasant.

So I kind of wanted to start with how do you debug in an environment where you don't have a debugger? Before we even talk about a debugger or what it is. Before we define the jargon surrounding debuggers, yeah. I just wanted to, what do you do when you don't have one? This is one of my favorite interview questions, actually, for firmware engineers. Well, what do you look for in an answer then? Give us your interview question. Yeah, hopefully they don't listen to this.

I mean, you usually start with printups, right? And that works in Arduino. You can have just printup all this stuff. And then I will start taking options away from them. So it's like well what if what if printf is not working what if you don't have printf and then uh we'll switch over to a debugger you know a jtag or sled debugger and you know that way you can step through the code do all this stuff but then i tell them hey what if that's not working or not external and and then we switch to you know blinking lights leds gpios that kind of

stuff and on an arduino you don't usually have that uh debugger step which is what we're going to talk about more instead you have the printf where you can print out information on a serial port. And I usually point out that that takes time. And the less you use variable arguments, the less time it takes. If you just print out a string, at least it's a constant time. But if you print out a string that is a number related to something in your code, that takes more time. Well, and it also might change the issue. I call them Heisenbugs.

When you start looking at it, put in printups, it might fix the bug. And then you take the printups away, and it comes back. And usually those are timing-related, right? Exactly. Sometimes they're memory-related because you've added more things to the stack at that point. True. That's even more fun. Corrupting a string instead of something important.

Yeah. Well, and you might have a display too. So you might be thinking to debug by putting something on your LCD display, but that's the same, the same issue, whether you're printing to a serial port or printing to a display, there's going to be some extra delay caused by you doing this action.

So Heisenberg is a good, a good word for it because merely by looking at something, you're changing its behavior. And then you mentioned light output and GPIOs, which is nice. When you have a logic analyzer and oscilloscope, you can toggle a GPIO when you go into something

and toggle it again when you come out so you can see how long it takes or you can verify you got to the line of code you wanted. Or if you have enough of them, you could put a whole byte out. Yeah. And that could be pretty quick. I mean, you could have some pretty granular indicator of something. One of my previous employers,

we had a device that when it went into error condition, you know, pretty granular indicator of something. One of my previous employers, we had a device that when it went into error condition, it would Morse code some obscenities out through an LED. I have Morse code. I've seen the SOS LEDs. I have Morse code to LEDs. I don't think we used obscenities. I think we were too focused on our limited vocabulary and Morse code and trying to use it.

Okay, so those are our important debug techniques, whether or not you have an Arduino, whether or not you have a debugger. You should know about printf. You should know about the limitations of printf and timing. the ability to toggle lines external because that gives you many features you you need even when you can't open the box and put on a debugger but this debugger thing people don't understand why it's cool uh so i want to talk about that sounds great so why is it cool i mean i the people coming from i mean i guess you have two people right uh coming high level software folks coming down to the firmware world and then more of electrical folks coming to the software to the firmware world and i assume the high level software folks might have well i don't hopefully know what a debugger is, but basically you get to step through your code,

go instruction by instruction. You can stop at various places. So you can tell it, all right, when you get to this instruction or this function call, stop. One of my favorite ones is a watch point. So you can tell it whenever this variable or address in memory changes, let me know. And that's very helpful when some rogue process

or function is changing your variables without you knowing. You can just tell the debugger, okay, stop and let me know whenever this happens. You can look at variables, you can modify variables as it goes. Just kind of a microscope into your program, right? You can dump whole memory. Oh, yeah. You can bring up a whole page of memory and watch how it changes to see, you know, if there's a corruption issue, you can

oftentimes it'll highlight, you know, depending on your debugger, of course, but it'll highlight which bytes or words are changing in red so you can actually just

see what's changing over time. If you're looking at a buffer, it's kind of pretty to watch because it updates one byte or however big your buffer size is, but one little thing at a time

and you can just watch it go by and it's kind of matrixy pretty yeah sometimes just looking at it will give you hints as to what's going on and this is let's go with the people coming from software who've seen a debugger and explain that to uh maybe a maker or an electrical engineer who hasn't seen that before and not worry about cross compiling and cross debugging because that adds complexity. We'll get to that in a few minutes. You mentioned being able to stop at functions and those are usually called breakpoints. And the ability to look at variables and watch variables. And that's a little bit different. Looking at variables assumes you're stopped and watches.

Often it will go even if the system is running. Some ARM chips let the debugger access memory without stopping, which I've used before, but I guess it's not the norm. Yeah, and it depends on many factors. When we say debugger, I kind of wanted to not worry about cross-compiling because I didn't want to worry about the word debugger because it's an overloaded, confusing word. That's true.

If we stay, just you write a program on your laptop and then you run it on your laptop, and you debug it on your laptop. That's all. There's only one debugger in that story. But we'll get to more debuggers later. So let's say you write Hello World on your laptop. How do you start, Alvaro? How do I start writing Hello World?

Yeah, I mean, what operating system do you have? What editor do you use? What compiler would you go for? Okay. How do you start? So I'm usually, my desktop, I have Ubuntu and my laptop's a Mac. They both, kind of the same,

I use Sublime Text as my preferred text editor. And on OS X, I'll use Clang, which is a compiler. And on on ubuntu i usually will use gcc so i'll just make a main.c file put in you know that includes and main and then starting standard argument stuff and i just say print f hello world And then I do GCC and put main.C and then you get a.out, which is your executable. So that's how we write it. And then for debugging, you would run GDB,

the GNU debugger or LDB. I think it's LDB. LLDB. LLDB. And then you basically run the debugger and then tell it to run a.out, and it basically puts its hooks in there between the program and the OS, I guess,

and it's able to control the flow of it. What's LLDB versus GDP? Those are both different debuggers he could use to debug. One goes with GCC, one goes with Clang. Oh, Clang goes with LLDB versus GDP? Those are both different debuggers he could use to program, to debug. One goes with GCC, one goes with Clang. Oh, Clang goes with LLDB. LLVM, and yeah. Okay. And so you GDB A.out.

So you GDB your executable. And when you load it up, it loads up all of the debugging information. And we had a show about that not too long ago, but it was very technical and deep. And I wanted it to stay fairly surface. And then it will usually automatically breakpoint main so that you can step through it. Right.

And so you use your GDB cheat sheet to figure out what all the commands are, because there are a lot of them. Yes. And you step through and step through, and then it prints it out, and then you're done, and it exits. GDB exits.

Does GDB automatically exit? What? I don't think so. No. Does it? Your program exits. Your program will exit. Oh, my program, yes.

It'll still be in GDB, because you can restart it. I do want to say that this is a command line thing. This is, I mean, we're starting with a debugger that's probably the most intimidating debugger. That's true. on the planet. That's true. But it's usually available. I totally agree. And LLDB is very similar to GDB, so those can be lumped in together.

That was why my next question was going to be Chris' same question. You want to write Hello World or some tiny program. What do you do? I have no different answer because I'm very used to GDP and used it for years. But that was the only thing that really existed. I mean, I probably wouldn't use it.

But I think there's a better answer for new users. Yes. That's where I'm headed. I totally agree. So my answer for that would have been Visual Studio on Windows, because I do tend to run Windows. I don't use that anymore because it's gotten way too annoying. Instead, I use Visual Studio Code, which is a much better program. And it still has GCC and GDB in the background for this this problem but it would be all graphical instead of needing to know

the gdp commands s and r and all of the commands i can use like f5 to run and f10 to step through or you could use your mouse or i could use my mouse because there are these little buttons across the top. And it would graphically walk through the lines. And then if I clicked on a variable, it would show me the variable, what its current value was. And so that's a visual debugger. And that's different than GDB. GDB is incredibly powerful, but it's one of those expert-level tools. And a graphic debugger has so many things going for it if you're new to this. And I mean, many graphic debuggers use GDB in the

background anyway, or they connect to a GDB server. Absolutely. And so my Visual Studio Code, that is what I'm usually doing with that is connecting to a GDB server. And Visual Studio Code, the IDE, right? Not your code in Visual Studio. Right. VS Code. VS Code. VS Code.

I should stop saying Visual Studio anything, because I finally uninstalled it recently. But VS Code. And when I work in Python, I usually do the printout version of debugging. There are Python debuggers, but because Python's interpreted, I don't usually use them. And you have to change your code in order to use the Python debugger. You have to put in your code the set trace or something.

I know people do use it, but I just use printf. So that would be my other answer is I don't really use a debugger for Python. Okay, so that's sort of the basics of Hello World. Should we talk more about that or should we add in the cross compiler? I mean, you can add some of the basic stuff, but it's all going to carry over to the cross-compiler, I guess. Okay. Or a lot of it is.

What's a cross-compiler? So usually when you compile code, if you just say GCC or Clang, it will compile code for the machine that it's running on, the compiler's running on. So if I compile code on Ubuntu, it's going to compile it for Ubuntu Linux. If it's on OS X or macOS, whatever it's called, it's going to compile it for that. But if I want to compile

for an ARM Cortex-M on either of those things, I need a special cross-compiler, is what it's called. So it's going to generate machine code for a different processor than what it's being hosted on. And that's where the cross comes in. Yes. And when we talk about a cross-compiler,

there's always this step of moving it from where you compiled it to where you want to run it. Sometimes that's called downloading or firmware update. Flashing. Flashing. Lots of good names for it, but it means that there is a cable or box or something. Mystery wireless link. Between your computer and your target device. And this happens on Arduino. Anytime you're using Arduino, you're cross compiling. Exactly. Because there's no way that any of these small chips can run a compiler for themselves. They can't self-host it, given the resources they have.

So the reason we cross-compile is not just to be annoying. It's because we're targeting tiny little things that can't run a compiler on their own. Or if the compiler they could run would be too dumb to be really optimized for doing high level stuff. So anybody using an Arduino is already doing this. They may just not be aware of it.

And that's true for embed. Embed is cross compiling as well. On the cloud. In the cloud, which is kind of odd. So it's not just a cable between your computer and the device. It's the cloud to your computer to the device. And it's kind of odd. And the embed style of downloading is to treat it like a thumb drive

where you just copy your binary image and then it sorts itself out. It programs itself. A little program finds the file on itself and then moves it. Yeah. And there's often an extra chip in there, one that can talk USB and do the programming for you. So if you want to make an efficient board, you don't necessarily want that extra chip.

And on an Arduino, there's a little debugger programmer chip in there that is kind of pointless if you're doing manufacturing because you shouldn't need that on every board i think some of the new ones don't have it they just have usb built in like a rom bootloader okay um i know the arduino pro just requires a serial port but that's that's sort of a special cable so we're going to stick with that all right so that was a little side tangent, just to make sure people understood that cross-compiling is absolutely normal and maybe doing it without realizing it.

It's okay. Yeah. It's going to be okay. Okay, so now when we say debugger, that means like six different things. Chris, why don't you list all six? What? Are you putting me on the spot? Well, there's the software program you run on your host computer.

Like GDB. Like GDB. There's the piece of hardware, possibly, that connects your device to your computer. Those can be called, unfortunately, debuggers, but they're not. They're debug support hardware um there's the individual who you say go find this bug you said there were six things i'm struggling to come up with six just made the number up uh the other the other piece that i might have identified is that there is support hardware in your processor, in the ARM Cortex-M or whatever, or the AVR,

that helps the system be able to be debugged. And so there's that debug support hardware inside the microprocessor, as well as this other box that is between you and the device, and then there is the software piece. So let's not call those all the debugger, because that's just going to be way confusing. The software is going to remain the debugger. And then the widget in between, I usually call that a programmer,

although sometimes a JTAG unit, which is super confusing because it doesn't always use JTAG, which is a debugging test sort of protocol. And yet, many of the devices are called JTAG just because that was such a standard for a long time. It's even slightly more confusing because those boxes are often embedded. If you get an evaluation model from, say, ST, I think the discoveries have those built into them. So when you hook the USB, a separate box you might buy,

and they can actually act as boxes like that for other boards if you want. So if you have a dev kit from ST, you could use it to be the debugger box. We're going to call that the programmer. The programmer.

The programmer. Or something else, but yes. You could say hardware debugger, but... Yeah. Actually, I was going to use microprocessor debugging features for the things that the microprocessor offers as far as debug features go. That's not what I was talking about, though.

I know, but I wanted to make sure we had all the words. All three, okay. So if we haven't confused you yet. Okay, so the debugger is the piece of software you typeity-type or clickity-click into. The, what were you calling it? Programmer. The programmer is the box that goes between your computer and whatever board you're using to do both debug options

and put the code on. And then the microprocessor debug something. Debug hardware, debug support. Is whatever piece. We don't really need to talk about that much. Always, because there are limitations to it. Okay. And then that programmer could be a box.

It could come from Jlink. It could come from STlink. It could come from STlink. It could come from Seger, which makes Jlink, I guess. There's a Blackmagic probe. Right. We had them on. Buzzblaster. There are many developers of programmer boxes.

And as we mentioned, sometimes those programmer boxes are not separate. Sometimes they're on the board. On the Arduino, there is a little tiny programmer box on the board, usually, and on an embed and on STM Discovery and Nucleo boards. So don't always expect there to be that box, but that box does provide some features. And sometimes it's very expensive and sometimes it's not. And we'll get into some of those features. Okay. So we're all agreed. And when we say debugger software, we still could be talking about GDB. So what kind of arcane magic is it that GDB can be the software I use for my local printf hello world and be able to talk to my hardware why don't we ask Alvaro that one that's hard so how is it magic like the GDB how can GDB

debug my local software and be able to debug something that may be sitting across the room. Ah, so usually the, the way I've done it. So that's what the black magic pro does. And with J link, the, the,

what we call the programmer, or there's a program that talks to the programmer that sets up a GDB server, which is just a, like a network, a local network interface that, is a standard has a standard interface that gdb knows how to talk to so gdb will talk to your programmer either directly or through some intermediary intermediary uh using a standard kind of command set to be able to read memory and write memory and and once it's able to and stop the processor basically so it's able to gdb is able to tell the processor to stop and to

read and write memory and step through instructions and usually from that they can do everything else okay black magic offers this gdb server for most operating systems at this point and there's this thing called OpenOCD, which is a very generic, works with a lot of different programmer hardware widgets and many different chipsets. And I find it difficult to use, but many people love OpenOCD

for its incredible breadth across features and hardware. And these provide GDB servers that GDB can talk to. It talks to in a very standard way. GDB says, I want you to break a point at this point. The GDB server says, okay, here's how I interpret that for the hardware that is attached, and here is how I interpret it for the hardware that is attached, and here is how I interpret it for the processor that the hardware is attached to.

It's not just, am I talking to a J-Link? Am I talking to a Blackmagic probe? It's also, what is that then connected to? Is it connected to an ARM Cortex-M? Is it connected to an A5? Is it connected to an AppMega something? Or an Intel Core i7. Yeah, so there are these two steps.

There's the programmer hardware and then the chip on the other end. Both have to be handled by this GDB server. And that makes it a very complicated piece of software. That was all I had. I was stopping it. Somebody would jump in. Yeah, I mean, yeah's it's your your programmer and and the software that it might have it's just a translator to let gdb access the hardware

directly on your microcontroller the way that it would a regular program um so it just yeah it just lets it do all these things that that then allow as the debugger, the programmer, to... The person. The person, the human, to do whatever you need to do to stop it and do. One of the nice things that GDB can do is show you, not quite graphical, but it does have the the tui mode i don't know if you ever use it chris but not only do you see the the current instruction but it can actually show you the source code as you step through it or the disassembly or the register files on the command line some people don't like that but it's usually good enough for me just dumps out everything all

the time um yeah and there's there's tons of programs that sit on top of GDB that aren't giant IDEs that make it more graphical well you can script it yeah you can script it with Python and I've seen there's one called Vulkan I think is that right? I'll have to look it up I'll put it in the show notes

yeah and it gives you kind of a textual but graphical thing where you've got a register window and some things that persist. So you're not necessarily playing a text adventure game with your code all the time. This is exactly a text adventure game. That's what it feels like. Yes. A script that takes ARM provides kind of the... Well, every ARM Cortex vendor provides a register file with all the peripherals and stuff. And he wrote a script to take that and feed it into GDB

so you can actually look at the register names, like timer3, control register, status register, and it decodes all the bits. And you're able to do that automatically because of the scripting capability of GDB. And I want to get back to that. Actually, no, let's just stick with that for a second. So when I first started with Embedded,

those register files didn't exist. And the reason that we would pay a lot of money for a compiler would be because Kyle or somebody had these files that described the processor. And you wanted to be able to program for it. And you wanted to look at the registers and know, well, is this a serial register? I mean, at its heart, it's just a point in memory. It doesn't have any special things about it. But then you get this register description file that lets you look at what each bit is, or at least name all the registers. And so you'd go to Kyle, and Kyle would have this support for different chips like the 8051 or different processors. And you would pay them for their compiler, but as much

you were paying them for their debug support software, because it wasn't GDB with this server. It was instead, they were hard coding inside their compiler debugger IDE all of the information about the chip. It's one of the reasons that even now, as much as I like GDB, when I get a new system, when I get a brand new thing from Nordic or a brand new thing from ST and I want to try it out, I follow their tutorial first. Even if it means installing a compiler I don't want or a compiler that's limited by size where I can only use it for 30 days, I can only make small programs. I download that and use it because they've already configured all of the steps for me and I can trust that if I follow the instructions, it will do what I want.

And then I can go to GDB or some other compiler and debugger and tool chain and set that up with some confidence that I have a reference working model. Do either one of you use other compilers and do you find them easier to set up? I've only used, professionally I guess, I used Code Composer Studio because I worked at Texas Instruments. That was kind of, I had to. And yeah, it's Eclipse based and that's kind of a bad word.

Well, I mean there are times to's Eclipse-based, and that's kind of a bad word. Well, I mean, there are times to use Eclipse. It is somewhat standard. I strongly prefer VS Code, and I have used Eclipse a lot. And Code Composer Studio's basis of Eclipse is true and possibly the worst of all worlds. All right. I've never used that so uh but you've used other other professional paid for compilers yeah i've used uh greenhills um and uh ir of course and keel kyle whatever they're all very similar i mean in terms of setup they're you know they're designed to be used by professionals which actually somewhat counterintuitively means their setup is quite

simple i mean there's an installer you press go and you you know usually there's a dialog box says what hardware are you are targeting and you click on that and there's there's some fine-tuned setup but for most common things out of the box it kind of sets itself up and you're the way you go with gcc and gdb you know there's a lot more intricate knowledge sometimes you need to at least make things run smoothly yeah and when you set up your hardware when you when you open a dialog box and say i have a new board it says is it from stm is it from nxp and then you walk down the whole processor all of the letters and numbers associated with the processor that's one of the things you're paying for is yeah this giant list of chips that they've done the job to customize their software to work with.

You can just choose it. Instead of you having to do it yourself, most likely. Right. And so I hear about people going from Arduino to GDB and GCC and then being hugely intimidated by that leap. And I kind of want to point out there's a reason that the other developers, other compilers exist. And that's because that leap is a hard one.

I mean, it's very difficult. GDB is tough to use if you've never played text adventure games. I like that analogy a lot. And the pay-for compilers are intended to make it fast to start up. Well, they're also intended to be full-featured and have all the things exposed in an easy way, so you don't have to go figuring out, oh, if I want to see these registers, what do I have to do in GDP? There's a lot more self-education you have to do

with some of the open-source tools tools because they're not spoon feeding you. The paid ones are crazy expensive, right? So for hobbyists, they're kind of prohibitive. There are free versions of, I think, of the K1 with limited. So it's going to limit you in certain ways. But a lot of projects you can get pretty far with that. Yeah, but I just, I guess, just want to say for the hobbyists that are like, who would pay for this?

But when you think about engineers and how much their time costs, it might be worth paying thousands of dollars to save them just a day or a week of setting up your tools and running through issues. You have paid support so they can come help you. So it's something that I personally never considered until I was working kind of professionally. And I would say there are diminishing returns with that. That's true. You pay, you do pay upfront and that buys you time. Years into large projects, you might find

that the limitations, now the limitations of those big things and the fact that you can't customize them exactly the way you want, or maybe you can't integrate it with your continuous integration system all those kinds of things start becoming issues and people start scratching their head and going well maybe we should spend a few weeks thinking about every time i've used other than texas instruments i've used some of the open source tools and we haven't really mentioned the platform i, which I know a lot of people really love. I've never quite managed to get into it. Adam is the text editor.

Which is same as Visual Studio Code, I think. Yes. The backend. Backend, yeah. So it looks familiar whenever I load it. It often has most of the processors I use. But if I'm using a brand new system, it's more likely that IAR or Kyle will support the new chip

and then Platform.io will take a few months to get that support package put so that I can look at my registers and all of that. And I totally could do that myself, I understand, because it's open source and as a good citizen I should do it occasionally, but that's one reason I keep bouncing off platform I.O. is it never quite supports what I'm about to work on. I don't know if their IDE for debugging is awesome.

I kind of wish I should... There's a picture of it right here. It's got all the stuff. Is it GDB-based? Of course it is. Okay. I don't know. I'm just guessing.

If it works, you can just provide your ELF file. Executable. It works with all the debugger thingies it doesn't say it's gdp anywhere here but but if it's just a front end for gdp that would be fantastic oh yeah it is

if I don't have to develop my code there I can just use it to visualize that's something big because seger has ozone I'm sure you'll talk to him about it but you need to use the seger jlink programmer to to be able to use it but it's cross-platform and and it's supposed to be pretty good uh the debugger is pay for you pay for the hardware no and platform io oh so there's the basic plan, the professional plan,

and those have the PIO unified debugger. Yeah, so I'm not sure what's going on there. But we should note that Platform.io is not Platform.io. No, it's Platformio.org. Platformio.org. People say it's Platform.io, and I know that's how they pronounce it, but I always think of Platformio. Sort of like Mercutio.

Platformio, Platformio. Shakespearean something or another. Yeah, okay, so Platform.io, something, something, and VS Code are somewhat similar in many ways, as far as I can tell, not having used Plumio much. I don't think VS Code is similar. I think VS Code tries to be a shell

for editing primarily and building. And all the other stuff you do is plugins. So there's no overarching organization with VS Code saying, here's how you use this for whatever board. You've got to do a lot of work to make that happen. I guess it was so much less work than I expected, having done GDP server stuff before.

But no one's going to hold their hand through it. It felt easy. But yeah, nobody really held my hand through it, and I was slightly shocked when it worked. And I think Platform.io, it's more what's your board? Oh, okay. Yeah, they definitely had the well, are you running this Nordic board or that Nordic board or this STM board?

And that list is super helpful, especially if it's up to date. Okay. Okay, and that is a visual debugger. So we're not typing R at the command line. We are using the mouse

or using secret key commands to have it highlight the next line in code. And then you can look at all your variables or you can dig in and look at the assembly. I like how the keyboard shortcuts are secret. And that's how the pay-for compilers would be for embedded,

that it would just show you all of the stuff and look kind of magical while it does it because you can see all of the registers at one time. And you can say, oh, I want to look at all of the UART registers and show me UART0. It is nice to have that bit level decoding so that you don't have to look at the hex number and then the datasheet and then the hex number and then the datasheet and then you write it out in binary and then you wonder if you got it the right

way or if you got it the other way. This is what computers are for, to make this information presentation easier for you. So, yeah. But setting up a toolchain is hard. It is intimidating. Are there ways to make it less intimidating? I mean, nowadays, it's a lot easier than it was maybe five years ago.

On the Mac and on Ubuntu, you just brew install, use Homebrew to install the ARM GCC compiler and toolchain. Same on Ubuntu, just app, get, or whatever, and then you can start compiling just out of the box. And if you get a Blackmagic Probe, then just plug it in and it just runs. So it's not as bad as it used to be. But I could see if you've never done it before, it might be a bit intimidating.

You can't just get GCC. You have to get the special ARM one, don't you? Yeah, exactly. But it's in the package managers already and ARM's maintaining it now. ARM took over the ARM cross-compiler, which is nice. Okay. One of the things I wanted to talk about was black box versus white box debugging. And Chris, do you want to define those?

Sure. Well, okay. So black box debugging is where you have no knowledge of the thing you're actually debugging. So you might not have the source code. So you're trying to figure out what's wrong with something that you only know the inputs and outputs of. And that's kind of the high-level definition of it. And that's traditionally what a development test engineer or a quality assurance person is doing. When they're testing your code or a device

and finding bugs um they're they're black box testing it black box debugging it i'm not sure actually you can i mean that that's kind of a weird i mean you could run into it with the with the nordic soft device for example you have half of your code and then all the Bluetooth stuff is just a binary blob. When I think about black box debugging, it is the Arduino style. It's the not being able to step through, it's not being able to look at every line. That's not what I think of.

Because you have the code there. You can figure out what's wrong. Black box, to me, usually means you have no knowledge of that. Or control. And white box means you have no knowledge of that. Or control. And white box means you have access to everything. And so even if you don't have the best debugger in the world, you can, through trial and error,

figure out where the problem is, specifically in the code. So I wouldn't necessarily taint those terms with debugging. Those are usually associated with testing, to me. That is true. And the other reason I really think of it is because to some extent, there's the idea of whether or not it's in a box, whether or not you can touch the electrical connections or not. And when you only have a serial port and maybe an LED

and no uh programmer interface then to me it feels a lot more like this this black box i can't see inside i don't know what my variables are right now you have to be able to reprogram it though right otherwise what's uh some of them will program over the serial port. Ah, okay. But you still can't step through them. I've had... I get a board and they tell me, hey, can you program these? I take a look and there's no debugger,

no JTAG interface, nothing. And they're not programmed. So that was a fun one. Crap. Oh, no, we made it work. We found a resistor i think somewhere an led we're connected to the uart bootloader pins and then we're able to program it over uart and then it had usb and then i could use the usb bootloader after that that i put in

and i can do printf debugging over that but it was not i would not recommend it. Okay, so let's talk about JTAG and ICE and SWD and all of the different programmer-associated words. Trace? Trace, yes, Trace. Do you know I want to add that in yet? Oh, well, that's a feature of debugging. Okay, so we've been talking about the features of debugging, the watch variables, the breakpoints,

register decoding, the firmware programming, the download it to the chip. So yeah, Trace is one of those features. Talk about it. Talk about Trace? Talk about Trace. We're probably going to talk about JTAG first, no? Yeah, we probably should start with JTAG.

You think so? Yeah. Oh, all right. Okay, so... Trace is talking about the turbocharger when you're trying to describe how an internal combustion engine works. All right.

So JTAG is this fairly old protocol that was used to test components and all kinds of components. And then people noticed it was very good at testing processors, and then they noticed they could use it to debug what the code was doing inside the processor. And so it has a lot of history, and it has a lot of weirdness, because this wasn't what its initial purpose was. And yet we use that word even if we aren't using the JTAG protocol

because it's kind of, anybody who's used JTAG, that's just what you, I don't know, it's a word that we like. Well, there's connectors associated with it. Right. So like legacy stuff that now we're putting different things over it, but it's kind of evolved. One of the different things that you see a lot is SWD, which requires

far fewer pins, and yet you still end up with the whole JTAG connector because it's handy and standard and people know what it looks like. It's a single-wire debug, right? Yes. Yes, although I think it's two wires, so I've never understood the single-wire debug. We have clock and data.

Right? You have to have ground somewhere too. That's true. Is it clock and data or TX and RX? No, it's definitely, I'm pretty sure it's clock and data. Because you have SW clock and SWIO, I think. Or is it three wires? Oh my God, if only we'd looked this up before the show hurry up oh it's

definitely it's definitely two plus ground and then you have power if you want to be able to detect that it's there it's also sometimes abbreviated as serial wire t-bug so i think there's some some funny business going on oh yeah that might be i think there's yeah it might not be single serial word yes that'swire. Yes. That's serial. Anyway, SWD. We're going to stick with SWD. And that even though that may be the protocol, we still call it JTAG because it's hard to change.

And acronyms, JTAG is so much easier to say than SWD because nobody knows what it stands for, apparently. Even though we've all used it okay so uh those are just protocols it's kind of like saying are you debugging over i2c or spy who cares it's jtagger swd the electrical engineer is going to care and that person is going to have reasons for caring. Usually what they care about is you don't have 97 pins, and so you end up with SWD. It's a bridge. And now we go to trace. I don't know why trace had to be here, because I thought trace was primarily a programmer function.

Well, you need a lot more pins. You need support on your board. Yeah. Okay, what kind of support do I need for trace? Pins. Lots of pins. Lots of pins. function. You need support on your board. Yeah. What kind of support do I need for Trace? Pins. Lots of pins. More than JTAG? No. It depends.

You can do just a few. I think I've seen it done with four pins. We'll just say what it is first. Yes. Maybe. Trace is a way to get very detailed logging about what the chip is doing as it runs. So as the chip runs, it collects this debugging data and spits it out, these extra pins.

And you have a more expensive programmer box with a large RAM buffer. Much more. And it collects all this stuff. And it can be because chips are fast and it's collecting a lot of data about what it's doing, you need a lot of memory in those boxes that can collect that stuff quickly. And so at the end of the run, or at various points, you can dump all that stuff to your computer and then analyze it in very sophisticated ways.

So instead of stopping your program and stepping through it, you might get a trace of the entire execution, and it will tell you, oh, I went to this function, did this and this and this and this and this, and then you can do that after the fact. And some are so sophisticated that you can theoretically do stuff like run your program backwards. So you can

say, well, I had a crash here, but I have all the state information. Just rewind, yeah. So rewind to right before the crash, and let me see what was going on, and then you can go back and forth. It'd be very expensive. This is more expert mode debug kind of thing, right? Like if you have your stack got corrupted and now you don't even know where you came from, this would have told you.

I've never had the pleasure of actually using one. Well, I do want to point out that this is not the human coding in log traces. Right. This is everything that happened in the chip. Yeah, spitting out like your program counter, for example. And the state of the registers. Right. Yep.

And so it is big. And when we talk about programmer modules that are simple, that are SWT and JTAG, the Blackmagic probe is very reasonably priced. Do you remember how much it is, Alvaro? I think it's like 50 or 60 bucks. Yeah. And then the J-Link is, I want to say, 400 for students? No, for students it's like 70. Yeah. Okay.

Then it's 400 for the base level of professional. And then usually I managed to get the thousand dollar one because that's just faster. Usually. Yeah. Until you pay 2000 or so. And then you get the trace. Yeah.

There's the J link plus the J link pro and the J link edu. So you get all of these different features and tools. And I suspect that they're all pretty much the same hardware with different software configurations. But that's just me. Some have Ethernet instead of USB interface so you can remotely connect and do stuff. That's nice when you need to leave your hardware in the lab and you don't want to sit in the lab yourself. Yeah.

And Alvaro kind of alluded to this earlier, but SEGGER is going to be on the show in a couple of weeks. So if you have questions for them, feel free to email them along and I can ask them. I have so many questions. Do you? What do you have? Well, I have a trace coming in for work and I'm excited to use it. We'll see if it works. And of course you have to have the software, right?

You have to have, in order to use trace, you've got to have, GDB isn't going to cut it. This is where spending money on a compiler. Yeah. Really? I mean, and this is not even the base level of IAR compilers. This is getting the expensive version of IAR. IAR actually has a separate product that they offer

that is just for debugging. And they have a trace module and they have a programmer slash debugger. So you don't even have to use IAR, the compiler, to use their debugger kind of thing. So I believe we can ask Seger, but I think Ozone has trace support as well. And there were tools from Micrium, too,

that they were very excited about that had more trace and debug support. So, yes, email me questions for Seger, and let's get back to this show, which is trace is awesome but moving on expensive and don't confuse it with JTAG or SWD

and if your manager says do you want this one that involves trace and they're willing to pay for it say yes because you may need it and make sure you're doubly put in support for it otherwise you're going to have expensive some microcontrollers can't do trace that's it's an added feature their arm provides they have to pay extra to be able to have the hardware okay so we're talking about hardware one of the things

that i have found with different levels of debugging is electrical engineers whining about board size if either one of you had that problem of course uh yeah uh i know that one company i worked with had a board that was very very tiny and flexible and very fragile and so they used pogo pins and had a whole support for the board pogo pins are if you don't know they're these little spring loaded pins that come out of a connector usually and the spring loaded part is not on the board

and they make a dry contact with your board so it tends to need some pressure to work well and it's not a very good connection so there can be problems anyway that's what pogo pins are i hate them i like them and then there are these tag connect things that i've i've sort of fallen in love with they're they're kind of cable converters tiny pogo pin connected things to a larger programmer interface. So it takes the tiny, tiny footprint on your board and pushes it up to a JTAG or SWD connector that you can use with your programmer. And it's really just a mechanical solution that allows you to have a tiny board

space for a debug because debug interfaces are standard and they're kind of standard at big and not at tiny and so that i like the tag connectors it's kind of a balance between a full pogo pin bed of nails that lets you look at lots of signals and and it's small board because the full pogo pin bed of nails does tend to be a little flaky sometimes well and you'll see these contraptions if you have a lot of them you'll get this giant thing that looks like a hydraulic press that comes from your double e so your board might be one inch by one inch and then you'll have this thing that's three feet tall with a giant lever to crush it into the pogo pins to make sure it's making contact.

And there's not something you'd have on your desk. That's more of a manufacturing line. I've had them on my desk. Had them on my floor. The ones I usually get have like several thumb screws that you tighten down. Form factor kind of debugging, right? I always lobby for a larger side blown up board that has all the nice connectors, try to do all the firmware development there, then go to the tiny, tiny board. And by

then, hopefully the firmware is ready. But I hate debugging firmware and hardware at the same time. Yeah, I mean, that's definitely a consideration. But sometimes you still have to make the form factor board slightly debuggable. You'd like to, yeah. It's too scary not to. But yes, if your electrical engineer is saying, no, you can't have debugging ability because we don't have space

on the board. Do you want it to work? Exactly. Do you want it to work exactly do you want it to work and uh do you can you make the space for something smaller and explode it off your target board you should end up with a custom connector that'll have you know power ground and the two sld pins and some tiny board to board connector or something or your pogo pins, and then use that. Or the tag connect is a semi-standard way to do that. Yeah, but now that whoever's working on wearables,

then it starts to get too small for even those, and that becomes a nightmare. Wearables often have the problem that they have the flex boards. Yep. Which is hard to connect to sometimes. We've lost all the Arduino people in the last 10 minutes. I know, I know. I'm sorry. Back to basics.

Back to basics. Not board design. Board design. What else did we cover? It kind of covers my list of you need a debugger because you get a lot more features, features you wouldn't necessarily expect or if you've never seen them, wouldn't know to ask for.

Not just features. I think you need a debugger because unless you're a hobbyist and you're not really, you don't really care where your time goes, the speed at which you will be able to develop and fix your software is probably 10x, 20x, 30x faster with a debugger. And some bugs are almost impossible to debug without it. I mean, I've had some USB issues,

and if your interface for printf is USB and you're trying to figure out why it's not working, it gets really tricky. And there's a school of thought that says you shouldn't spend so much time debugging. That means that you're spending all your coding time writing bugs.

And I think that's nice for them. I end up debugging because there's integration problems where code meets from different people and you can't unit test everything on hardware and sometimes there are chip bugs.

Not usually. Well, if you wouldn't use cutting edge chips, you wouldn't find them so often. So often. You might still find them. It's not the edges problem. Yeah. Not writing perfect code is wonderful in theory.

Even unit testing code to be perfect is wonderful in theory. You know, eating well and exercising will keep you alive forever. That's also wonderful in theory. I'd rather have a fun life. Dark leafy greens, man. It's all about the dark leafy greens. Michael Cohen? Yeah. That's not where I got it from, but yeah, probably.

Okay. Yes. What else should we talk about with debugging? I mean, we've talked about a lot. Alvaro, what did we miss? I mean, that's the basics right uh you could tell you times i've used it but it's usually so i guess i guess okay here so if we're talking basics this is probably gonna open up a can of worms but i have an arduino i've been doing that i want to debug it

i want okay you sold me you sold me i i i'm gonna go get a debugger and and try it on my arduino code you can okay so let's let's talk about how to do that alvaro oh thank you i mean i mean i know it i would take it off the arduino if if you're going far enough that you have to start doing that. But if you did have to, I assume that the Arduino compiler produces both a binary and an ELF file, which is just the binary file with all the extra kind of information that a debugger might need to know which line number is where, what the function is, all that stuff. If you have that, then you could load it in gdb and connect it assuming

your arduino has a programmer slash debugger hardware yeah um which the newer ones if it's a cortex m you're fine i i do not know anything about the ad megis well basically i mean i think you have to move away from quote arduino to the. Yes. Which is, you're not moving anywhere. You're moving from the comfortable confines of the Arduino framework and all of that to using the tools that they're built upon, which is the AVR GCC support, AVR GDB support. I'd rather move to a Cortex.

Yeah, but... Arduino. Arduino. I think moving from an Atmega to a Cortex-M is a very large leap. It's just that the debugger support for ARM chips is just so much bigger. But Chris is right. If you want to debug your Arduino, the word you're looking for is AVR. The acronym you're looking for is AVR.

And there's a website called AVR Freaks, and they have so much stuff, and they're very nice. Basically what we're saying is the same chip. It is the same exact chip. You might need to buy a programmer that piece of hardware they aren't super expensive because lots of people want them and i believe there are different versions on tindy so that you can get them for like 30 bucks and you might have to get a different arduino you might have to get an arduino pro or something with that programmer chip removed because you don't want them to fight. But it is possible to do much of

this debugging with the Arduino. I do kind of agree with Alvaro that if you're going to bother, go to a Cortex-M because it's easier and there's lots more support out there for it. But it's completely possible to do with an Atmel chip. I mean, Atmel has always made debuggable chips. It's just that Arduino was made to be easy

to use. And I didn't, I couldn't Google fast enough to get all of the words right, but I know that it's AVR. AVR, dude, is kind of the program. There's AVR ISP. It's one of the cheapest programmers. And AVRfreaks.net is a forum.

Yes, that's over a decade. I remember going there. It's been there forever. Yeah, it's the AVR ISP. And we didn't talk about ISP in our word salad here. ISP should be in there with JTAG and SWD. ISP is something, something programmer. In-system programmer?

In-system programmer, yeah. And it is just a different way to talk to the chip. It's one of the ones that Atmel uses specifically. And when we talk about arms, that's where we get into SWD. Okay. Well, I think we should call that a rousing success and wait for the adulation to come in via email and not complaints or questions that we can't answer.

That's very optimistic of you. But our listeners have been really nice, and I should be launching into some of the really nice emails we've gotten. But sometimes it's hard to read it when you send really nice emails, because I don't believe that you're listening to the show. Be meaner, we'll believe you. Albert, let's talk about you for just a minute or two so that we can... What's up with you? Good question, I don't know.

You've been working on this project in your own time outside work, which we have so carefully not mentioned. You've been working on this project that involves cheese and building a cheese refrigerator. Not just a cheese refrigerator, but cheese-shaped boards. Cheese-shaped boards. Yeah. Where can people find out information about that? So, I haven't written too much about it yet. I submitted a talk for the Hackaday Super Conference, I guess.

I don't know if I'll get it or not. If the title of your talk isn't Let's Go Somewhere They Have Cheese. Let's Go Somewhere Where There's Cheese. Yes, Let's Go Somewhere Where There's Cheese. I will be disappointed. I don't know if I was learning something something by making cheese. I would have gone with Cheese whiz no oh come on there's a million wallace and gromit things yeah there's a lot of jokes oh yeah i i do have i do

have a gromit um anyway um i have a gromit people have not seen it are going to be confused. But yeah, I've been making a cheese cave. It's called to age brie type of cheese. And I've started, you know, big old circuit board with a bunch of sensors. And then I'm just kind of practicing my board layout skills and trying out different things and making the board smaller. And the last revisions actually cheese wedge shape temperature and humidity sensors. And then the other half of the cheese wheel is my controller board.

And you don't get those confused with actual cheese. They're way too crunchy. Yeah, well, and they're're purple so i need to make a i've seen purple cheese you're not eating it don't eat it even blue cheese isn't completely blue maybe it's just the rye that was purple that's the wax don't eat the wax yeah uh and so your sensors are humidity and temperature primarily? Yeah. So the main thing with Bree is you want to be around 80% to 90% relative humidity and around 10 degrees C, which I don't know how it's in Fahrenheit, like 50 or something.

So I got a little regular fridge and put a relay and then have my microcontroller control the fridge relay, just turning it on and off until it gets the right temperature. Well, you say you got a fridge. I seem to recall that this is the second fridge. I forgot. I told you guys about that. Yeah. So I got a fridge on Craigslist, just the regular mini fridge. And I had an idea of before I started all this, I was testing different sensors, testing ways of controlling humidity. So I had my controller board and I figured, how do I get the cables in there? So I just drilled right through the top and I was able to get some cables. And I had a little hose so I could have a mister. I never ended up using it, but I tested lots of different things.

And then I needed, I didn't have enough space for all my cables. So I said, oh, I'm just going to drill another hole right next to this big hole. I drilled and I start drilling and then I just hear it. All the coolant just sprayed in my, well, not quite in my face, but in my bedroom. It's probably not good. But yeah, that fridge died. I think the second day I had it. So I had to go buy another one.

When will your brie be ready? This one I started a week and a bit ago. So maybe in like four to five weeks-ish. That's a long turnaround time. That's pretty slow. It's pretty fast, actually fast actually i mean if you're thinking cheddars parmesans gruyere those can take years why cheese because i love it i am obsessed with cheese all right well it's good to know if you're

if you're going to hackaday supercon be sure to bring Alvaro some cheese. If I actually give a talk on that. Otherwise, I'll still just write about it. I have pictures on Imgur. I actually took a picture. I have a time lapse of the brie aging, so you can see the mold growing on it. And then there's a few pictures on Twitter of just kind of the cheese cave and stuff and whatnot. We can look them in. And you had a picture of the board today.

Can I use that for the show notes? Absolutely. Cool. And then the other thing we kind of mentioned at the top is that you are starting this podcast with Jen Castillo, and it's about reverse engineering, and people should listen. And what made you start this?

I mean, was it that Chris and I make this look so easy? No, no, believe me. We talked about this over a year ago when I quit my last job and I had lots of time. We always wanted to kind of reverse engineering stuff. We're both reverse engineering enthusiasts, I guess. There's like meetups here. I like going to DEF CON, that kind of stuff. So you guys have been able to talk to so many cool people in the embedded world with a podcast. I like how you use it.

It's an excuse to talk to cool people. So we're going to do it more in the reverse engineering, kind of deeper end side of things. I think it's a good idea. We didn't see anything else like it. Yeah, I look forward to hearing more about it and listening to that third episode.

Yeah, they've already got two up, and he's finishing the third. I know. What's the third one about? I don't know. It's just us talking about some of our projects and stuff. I thought you had insight. We still need some practice. But yeah, editing, I am amazed at how much work it is for Chris to do all this.

It is so much work. I just remember last time I was on, it was what, six of us or five of us? It was a lot. It was a lot. Yeah, it was a lot. That was episode 200. And it was a lot of fun, but it was a lot it was a lot yeah it was a lot that was that was episode 200 and it was a lot of fun

but it was a lot of people alright well Chris do you have any questions no no you ready to go to the beach

no no we'll have to make sandwiches and then we can go Avro what about you any thoughts you would like to leave us with? I would suggest people kind of try out some of the debugging stuff while they're sane,

because when they have to use it, it's not going to be any fun trying to figure it out when your stuff's not working. So kind of getting just the basics up and running. So, hey, how do I step through code? Just try it out. And then when you do really need it, you'll have kind of at least an idea of how to set it up. It's like getting a hardware debugger or the trace. You want to have it. It's not a kind of thing you want to order and wait a week to get it because you're going to go crazy.

So I would just suggest people try it out. Excellent. Practicing skills is a good idea. Yeah. Especially new strange ones. Thank you for being with us. Oh, my pleasure. Our guest has been Alvaro Prieto, embedded software engineer, master of the cheese, and co-host of a new podcast called the Unnamed Reverse Engineering

Podcast. There will be many links in the show notes for those. Thank you to Christopher for producing and co-hosting. And of course, thank you for listening. I feel like I should have announcements here, but I don't remember what they are. So, you know, have a good day. And a thought to leave you with from Brian Kernighan. That's not right. Kernighan. Kernighan. Brian, Brian, like the Kernighan and Rich's not right. Kernighan. Kernighan. Brian. Like the Kernighan and Ritchie guy. Alright.

Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it? Embedded is an independently produced radio show that focuses on the many aspects of engineering. It is a production of Logical Elegance, an embedded software consulting company in California. If there are advertisements in the show, we did not put them there and do not receive money from them. At this time, our sponsors are Logical Elegance and listeners like you.