Embedded - 466: Attacked by a Goose on the Way to the Office

Episode Date: December 14, 2023

Ralph Hempel spoke with us about the development of Lego Mindstorms from hacking the initial interface to running Debian Linux as well as programming Mindstorms in Python. Happy 25th birthday to Lego Mindstorms! Pybricks is a MicroPython based coding environment that works across all Lego PoweredUp hubs and on the latest Mindstorms elements. The creators are David Lechner and Laurens Valk. Ralph was the first person to boot a full Debian Linux distro on the brick, see EV3Dev, a Debian Linux for Lego Mindstorms EV3.  BrickLink was originally a site for third party resellers of new and used Lego sets and elements. The site was purchased by the Lego Group a few years ago. It's still a great place to buy individual parts - for example a 4 port PoweredUp hub to run the new PyBricks on :-) ReBrickable is a site dedicated to taking off-the-shelf Lego sets, and creating something new with the set. In particular see the MOCs Designed by LUCAMOCS, fantastic Technic vehicles as well as interesting designs for vehicle subsystems. Yoshihito ISOGAWA - YouTube is an absolute genius at coming up with practical applications of new LEGO Elements. Ralph recommends his books as “awesome to read”. LEGO uses 18 Cucumbers to build real Log House  Ralph highly recommends Test Driven Development for Embedded C  by James Grenning (who has been on the show: 270: Broccoli is Good Too, 109: Resurrection of Extreme Programming, and 30: Eventually Lightning Strikes). Origami Simulator and Elecia’s origami generating python code on github Transcript Nordic Semiconductor empowers wireless innovation, by providing hardware, software, tools and services that allow developers to create the IoT products of tomorrow. Learn more about Nordic Semiconductor at nordicsemi.com, check out the DevAcademy at academy.nordicsemi.com and interact with the Nordic Devzone community at devzone.nordicsemi.com.

Welcome to Embedded. I am Alicia White, alongside Christopher White. Our guest this week is Ralph Hempel, and we're going to talk about engineering and toys. Hi, Ralph. Welcome. Hi, how are you guys doing? Good, good. Better now that the computers are behaving.

Oh, I know. Could you tell us about yourself as if we met at, I don't know, Electronica? Sure. Hi, my name is Ralph Hempel. I've been doing embedded systems for about 40 years now, and I've programmed on everything from punch card-based computers to HP calculators to micros to embedded Linux systems. But my favorite thing to program is M4 and lower class MCUs. I led the firmware development team at the LEGO Group for the last seven years. And we developed the latest generation of LEGO Mindstorms, as well as all the powered up components. So that's all the recent Technic and Trane electronic elements.

And now I'm back in North America at Edwards Fire and Safety as a staff firmware engineer. So it's been a fun ride. Well, I want to ask about your career and your time at LEGO and all kinds of questions, but first we're going to do lightning round. Are you ready? I am ready.

Most underappreciated LEGO piece? Here we go. Most underappreciated Lego piece. Oh, here we go. Most underappreciated Lego piece. I would say, my goodness, now you're really stuck with me now here. Because that wasn't on the list, you guys. There's so many parts. Questions may include, but are not limited to are not limited to okay thanks i didn't read the fine print obviously um i'm most underappreciated lego piece i would have to say the uh the hassen pin uh it's not actually called the hassen pin but it's actually a right angled uh lego technic connector yeah that was used for joining beams together. And now, of course, the Lego group has actual square frames or rectangular frames that make that a whole lot easier.

But there's a really fun fan story about how that part came together, which we might get into later on in the call. When someone finds out what you do, what question do they always ask you? Oh, that must be so cool. Can I get some Lego? Yeah, that's it. No, but it's actually kind of fun at parties. It's like, what? You worked at the Lego group? Yeah, yeah. Oh, that's cool. Where in the States were you? I'm like, no, it's a Danish company. I was based in Billund in Denmark. So that's always kind of fun to just get around that. And then, of course, that's much more interesting than seeing iProgram-embedded systems.

That's computers you can't see, like in a microwave oven or in your engine controller unit. We write software for things that aren't computers. Yeah, that's my usual answer. Yeah, that's a real buzzkill. Lego group, that's an awesome, yeah. Favorite course you took in your academic career? Favorite course? Actually, that's an awesome, yeah. Favorite course you took in your academic career? Favorite course, actually, that's a great question. So in high school, it actually wasn't a university course,

it was a high school course. And in Canada, we have an algebra and functions and relations and calculus course in what was grade 13. We don't do that anymore here either. And at our school, at Jarvis Collegiate in Toronto, we had two excellent, excellent math teachers, and they taught a course called triple math. So instead of having all of those three courses with three different teachers, they actually made their own textbook. And we had

all of those courses with one teacher for the whole morning, with usually a one course break in the morning. That sounds really brutal, but it was the best preparation I had for university. I didn't see anything new in math until kind of the second half of second year. So I just, I'm really grateful for those teachers. Speaking of university, have you gotten bitten by a goose? Are you still afraid of geese and are the geese afraid of you? You've done a little bit of background work on the University of Waterloo. Yeah, there's a lot of geese. We have geese in our yard here up in Ontario as well. And I have not been bitten by a goose, but I've been chased by the swan that lives on our river. Not really afraid of geese,

but my son was one of my boys works at the Bruce nuclear plant and he was attacked by a goose on his way into the office. It's a thing. Yeah. Best language to learn programming and which one did you learn first? Well, the first language I learned was HP keystroke programming for Hewlett-Packard calculators. That's really what kind of got me hooked on it. The first language I learned, I think, was Assembler, 6502 Assembler on a Commodore. Yay! Yay! Sorry, makes me happy. 6502.

So that was kind of cool. I mean, the tools that we had available then are so much different than what we have now. And we'll talk about, you know, Kim Ones and all those other cool retro computers that are coming around again. But yeah, that's one. But if I had to learn a first language today, I would probably pick probably Python as a getting started language. Or Scratch if you're under 12 years old. Do you have a favorite fictional robot? I do, and it is Chappie. Cool. From the movie Chappie. Have you seen District 9? I have not seen District 9.

I know of District 9. Okay. Chappie's a cool robot because he's supposed to be a police and surveillance control robot, but he gets kidnapped and is put to work in interesting ways. So it's actually a fun movie. Well, it's not actually a fun movie. It's kind of a depressing movie. You're changing the story. The character Chappie suffers some real human things, and I think he's probably the most human robot that I can think of. Yeah. I'd like to thank

our sponsor this week. Nordic Semiconductor specializes in ultra-low low power wireless communication. They have a wide technology portfolio, including Bluetooth low energy, low energy audio, Bluetooth mesh, thread, matter, cellular IOT, and Wi-Fi. They have thousands of customers worldwide with 40% market share in Bluetooth low energy. They have nearly 2 million system-on-a-chips produced every day. Sounds like a system that you can't go wrong with.

So please check out nordicsemi.com. And if you would like to win one of the Power Profiling Kit 2s that our sponsor is so generously handing out, please send us an email that is show at embedded.fm and tell us what your favorite PPK2 feature or spec is. Again, thank you to Nordic for sponsoring this week's show. Okay, moving out of lightning round. So I remember Lego Mindstorms coming out like 20 years ago? More, sorry. 25? 25. This year would have been the 25th anniversary. I remember getting the first version of it. And it was really cool. And we were out of school, so we were, you mostly played with it, I think. Yes, but I also worked with computers for a living, so coming home to work with computers for fun was not working out.

But I was so fascinated by it. And then it seemed to go away. Was that just my attention? I think maybe real life caught up with you i think that's your attention no uh actually the lego mindstorms brand has been uh up and running for the last 25 years this year uh i think the lego group retired it or officially discontinued mindstorms so you would have been familiar with the rcx that's the yellow brick way back when. And that was where I really started hacking on Lego. So I'm going to go back and tell a little story about Usenet. Do you remember the actual Usenet system? I was the Usenet room administrator for our college.

Oh, okay. So Usenet had, this is long before social media, as you all know, and there's a group called rec.toys.lego. And on that group, we had a whole bunch of discussions on, hey, we should, you know, if we're electronic geeks or embedded systems geeks, we should make some sort of a controller for Lego creations. And, you know, some of us that have actually made commercial products realized that, hey, that's just not too practical, or at least it wasn't 25 years ago.

And then thego group brought out the rcx and a group of four of us that kikoa proudfoot mike gusberry myself and dave bomb we were the original uh hackers on the on the lego rcx so this is the reverse engineering story do we have time for a cool time for anything cool so got time for anything. Cool. So Kekoa, he was a grad student, I think, at Stanford, and he came up with the idea. The original communication with the RCX was over a USB, or sorry, it wasn't USB, a serial light link or a little infrared tower. And then he kind of sniffed on the serial wires and realized what the protocol looked like. And then was able to, because you had to download the firmware to the brick every time initially. And it was only, you know, 30k or something like that.

So it didn't take very long. And in the course of sniffing that out, he was able to figure out what kind of processor it was. Because you could figure out what the assembly language was and then he wrote a little dumper that would then replay the entire contents of the ROM back to on the serial part and from there we were able to reverse or he was mostly able to reverse engineer the assembly code and make a an annotated listing of it and And we were all kind of, that was kind of really cool. So then I wrote a fourth interpreter for the RCX. And this kind of got the attention of the Lego group, the work that we had done on the reverse engineering of it.

And they invited us down to MIT for the very first MindFest celebration. And there we got to meet the engineers that made the RCX, and they looked at our disassembled code, and they were like, hey, this looks actually better than our source code. Because it actually had dominance and everything. So that was my first introduction to the engineering team

at the LEGO Group, and I stayed in touch with them for years and years afterwards. So getting back to what your original question was, Alicia, the RCX was the first generation and about every seven or eight years, the LEGO Group would make a new generation. So then after that came the NXT, which was mostly a whitish grayish brick with orange colored highlights that was using an AT Sam, an at micro sam 21 or something like that and then the next generation was the nxt and that was the full-blown linux or a little that was an ti am 335 i think based uh based unit and then the most recent generation is we're back to STM32F413. So that's the most

recent Mindstorms that was released, I think in late 2018 or early 2019. When you were doing this at Disassembly, was that when you were also working at a company for safety and fire suppressant and yes yeah how is it different to be thinking about toys versus important products versus important products yeah actually you know it's it's an interesting leap between the two because in some ways they're they're extremely different and in other ways they are not different at all because they're both highly regulated industries and the lego group is is always really super concerned about child safety and uh more recently with security and online safety and things like that but the product itself had to be kind of intrinsically safe and not

disassemblable because of certain toy safety standards around the world. So the whole openness of the product for right to repair and all those things that are coming out in the next few years, those are going to be really difficult to reconcile with toy safety. But the LEGO Group made Mindstorms in quantities of hundreds of thousands every year. Every part was individually tested. So the motors, the sensors, they all went through separate functional test fixtures. It's really not that different.

The main difference with fire safety and safety equipment is that it just sits in the wall waiting for the big day. Waiting for somebody to pull the fire alarm or something to happen. safety and kind of safety equipment is that it just sits in the wall waiting for the big day, right? Waiting for somebody to pull the fire alarm or something to happen, something interesting, please. But it has to sit there and work for months, years, and if in some cases, decades without being power cycled. And that's a real challenge. That's pretty different from toys, which if something goes wrong, you just power cycle it. Nobody really, I really i mean sure there's the safety issue of making it large enough so that a child can't swallow it and making sure that it doesn't electrocute you very badly electrocute you very badly okay yeah we didn't know there were varying levels of electrocution don't

electrocute the kids very badly that's our that's our motto no no the key thing there though is that i mean there's you wouldn't think about it but there's things like uh light intensity so when the leds you know they can only be so bright you can't you have to deal with flashing for uh you know for other uh sense light sensitivity you've got to deal with heat on the batteries when you're driving a lot of motors install mode so that the batteries don't get overheated. So there's a lot of stuff that you wouldn't think would be relevant, but it really is. You listed processors that are off-the-shelf processors. You didn't do custom chips? Earlier generations would have done custom chips.

So they would be ROM-masked. The RCX was an H8300 from Hitachi, I think. And that was a masked ROM part. After that, the parts were mostly firmware updatable. But yeah, no custom parts. Those are generally way more expensive to produce and the quantities that were, you know, the overall lifetime quantities, they just didn't justify the cost or the risk. I worked at LeapFrog where we made custom chips,

but we used the same chip for multiple product lines, and that worked out well. Right. So, yes, masked ROM. And then there was a time when masked ROM was a little cheaper, and then all of these chips came out with Flash, and Firmware Update was just so nice. Oh, I know. I'll tell you a funny story about Firmware Update. So when I first started in the fire alarm industry, the signature panel that the Edwards group was making was you had masked, not masked, EE prompts in them.

So you remember the old windowed E prompts that you had to stick into an eraser or the sun if you didn't have an eraser with you. Or you were just by accident trying to erase your system without knowing it. Yeah, that happens too. So the trick there is that you can't just, you know, you could send a technician there with, and this is like literally 35, 40 years ago, you could send a technician onto the site with the files and everything. But, you know, on a 300 baud modem, it would take forever to send the files down. So at some points we were actually putting people on airplanes with a sleeve full of EEPROM devices that you could literally pull the old ones out of the fire alarm and put the new ones in. So the whole idea of field upgradable

or even over-the-air updates, as are much more common now than they were before, or DFU updates, that stuff, we didn't even think that that would happen. But it's here now, and we should really take advantage of it. It's changed everything, really. With LEGOgo i have a question from spec about what processing did you have inside the mario lego sets these were the ones that you could drive around and pretend to be in mario kart was that uh no maybe there was that one but there was also

yeah i'm not sure which one he's asking about. Yeah, so there was a line of products that the Lego Group did in partnership with Nintendo, and that was a small Mario character. So there was a Mario character, a Luigi, and I think a Princess Peach came out after that, right? And it had a little accelerometer and a gyroscope in it, and you could sort of bounce the character around and collect points, and it had a little accelerometer and a gyroscope in it. And you could sort of bounce the character around and collect points.

And it had a color sensor so it would tell if it's on a red tile. And there was actually some specially printed barcoded tiles that would give him different points or different actions. That's the same basic STI. Oh, no, that was a different part. That was a TI part with Bluetooth low energy enabled in it. Let me see. TICC2640? 50? 30? One of those. And then there was a bunch of sound files that went with that. And it was a cool little product. But I think some enterprising fans actually put it into a Mindstorms or a Boost.

Boost was another robotics line that we had years ago. And that was, you could put the Mario figure in it, and then you could kind of drive around. And they made a, that's usually fan type stuff, right? Where the fans take our product or take and i say our product i don't currently work there anymore uh but they would take the lego products and really extend them uh in in ways that we really didn't expect and that's cool yeah i mean that's not something you're going to get with fire alarms for sure yeah well you don't want that

look what i hacked my fire alarm to do. It's not something you want to hear. No. Do you have any good fan hacking stories? Well, I'm a good fan hacking story. Yes, yes. So I've made firmware for every generation of Mindstorms up until the one that I was actually hired to make. So, you know, everything from PB4 on the RCX to a Lua interpreter on the NXT.

Lua was actually, it's actually a pretty cool little language. And then we put the full Debian Linux on the EV3. So those are kind of cool hacks. But I mean, if you, i think i sent you some links uh for that i think will be showing up in the show notes and i am always so impressed by what fans were able to do with things that we just really never imagined i think there's there should be a link to somebody who's made a log house maker out of cucumbers so he actually he actually takes a cucumber and slices it into little log-sized

pieces with notches and everything and makes a makes a log house out of cucumber pieces so just i mean yeah you don't have to look far to find some crazy fans to to do some amazing things uh when you were initially reverse engineering the first one um it sounds like it was all external that you and your friends looked at the serial communications and figured everything. Was there a, were you tempted to open it up and start poking at things, or was it just easier to go from the outside? Because when I'm thinking about reversing things, sometimes I want to open it up and say, okay, what parts are these? Well, yeah, a combination of things there, Chris. That's a good question.

So we started with, you know, just, and I think it was, again, mostly Kikoa and Dave Baum that did the reverse engineering. But, you know, you start with the serial protocol, all the affordances that are external, that are easy to get at, you do that first. But then, of course, those elements were fairly easy to open up without destroying the package.

And you could see that it was an H8300. And you could kind of figure out where the external flash and the external RAM was. And you could kind of figure out where all the parts are, what the motor drivers were, and where the pins went. So we did a fair bit of wringing out of this PCBA. But it's really just a case of slowly reverse engineering, building up your knowledge of the product bit by bit. And then finally, the big day comes and you're able to do a complete dump and then you send that binary dump into a reverse disassembler.

And wow, there it is. It's all the source code. Cool. Now we can start commenting it and figuring out what goes where. And it was surprisingly complete. It was pretty cool. And the story again that the Lego engineers were like, how are we going to encode the serial stream?

So you have to think about it from the other side. First of all, is anyone even going to bother trying to reverse engineering? No, not really. How are we going to protect the stream well we're going to balance the data out by sending out the negative of the inverse of every byte so every byte really gets sent twice once in the positive once in negative just to balance out the uh the energy and they're never going to reverse engineer that because if they did, they'd have to, sure, it's binary code, but now you'd have to reverse disassemble it. Who's going to do that?

It's just a toy. Who'd bother? It's just a toy. And that whole philosophy, you know, we showed that that was, hey, it's just a toy, but there's a lot of people out there that are really interested in it and motivated to figure out how this thing works. And that be taken you know for good or for bad but fortunately hacking on lego was always a good thing yeah it's i mean when we think about hackers it's like scary people who steal our identities but the truth is sometimes you just want to see how something works. And Lego is such a good place to play with that. Yeah. And I think if you sampled many of your listeners or members of your Patreon group, they would say, or any engineers that you know, most of them will have had Lego in their hands

at some point in their lives, right? It's not that playing with lego makes you an engineer but most engineers if not yeah most engineers in my experience have touched lego and it's a great prototyping tool i've used it i mean the uh the pop filter for my microphone here that's actually a lego frame with some with some 3m uh sanding paper in between it just to sort of even out the air bursts. So we use it for all kinds of things around the house. It's kind of like a 3D printer, but with blocks instead of filament. And you're the printer. Exactly.

And you're the printer. Well, yeah, exactly. And then, of course, when you really complete the loop like I just did, we actually 3D print things like mounts for Raspberry Pis so that they have Lego compatible pins. And then you can build your Raspberry Pi into a Lego creation. So, yeah, we go all the way. You worked with Damien of MicroPython. How did that come about and where did it go? Well, that was a really happy accident. So when I joined the LEGO group to work on this latest generation of Mindstorms, I kind of looked around and surveyed what could possibly be put on the brick. So we had a rough idea of the processor that we needed and some of the features, but not all of the mechanical and visual or UI features. We'll talk about those in a bit. But I've always tried to put an interpreted language right on the brick.

So that's why I put a Forth and a Lua interpreter on the brick. So my goal is even if there's no app, so let's say the app support goes away, you can always program the device with a PC and a terminal, right? A terminal emulator. How come I didn't know that uh well go ahead the official the official stuff can't do it that's just my way of thinking i would love to be able to put an interpreter or a repl loop on pretty much anything and once you have a repl loop you can you know you can kind of hack into the system and do more but my goal with uh with the latest generation of mindstorms was to put some sort of an interpreted language and we looked at Lua and it wasn't all that popular and then I kind of poked around and I saw Damien's MicroPython was getting quite a bit of traction also with the MicroBit and the MicroBit has inspired a few

things about Mindstorms as well. So I checked out MicroPython on the micro bit and a few other processors and like hey this is actually pretty good and it might work and it totally fit on the processor we were using we had some room left for drivers so that we could uh talk to our motors and sensors and the display and it just was a natural fit but there was quite a bit of resistance because previously uh the lego programming app was kind of a labview type app and yeah and the idea of python wasn't all that or wasn't all that popular and then we also had to think about it from the app development side and uh you know labview's a it's an okay system i never personally i never really liked it but there's a lot of fans

and a lot of kids have done a lot of amazing things with it. So that's kind of cool. So we looked at Scratch and Blockly specifically. And we did a proof of concept with the EV3 using Blockly. And Blockly translated. At the time, there was sort of a thing that you could take your Blockly blocks and generate Python with those. So as a proof of concept, can we assemble a series of blocks, spit out some Python that we could interpret on a brick?

And once we had that proof of concept done, it was like, let's go full steam ahead. And Python was then the language of choice and then i talked to damien you know just to kind of make sure how would he feel about that being in our in our product and he was really cool with it he is probably um i i don't know exactly how to say this but he is probably one of the cleverest uh developers that isn't wasn't a developer I mean, you know he's a physicist. And he has done some just amazing things with memory management. And over the years, he has modularized

and isolated the modules, the support modules for all the different MicroPython variants really well. That code base has grown and I've learned a lot about how to write drivers and how to write Python code properly and also how to write C code properly from looking at his examples

and the way he structures his code. And I hope that that relationship has gone both ways. But he's just an awesome guy to work with and I just wish there were more people like him in the world i do like micropython the more i use it the more impressed i am by it it is a lot of code but it's good code it is yeah yeah and so micropython for mindstorms led to PyBrix? Yes. Now, the PyBrix is a fan-created product.

So that's David Lechner and, sorry, mainly Lawrence Volk and David Lechner together. And they established PyBrix a few years ago. One of the other things that you may not know about LEGO Mindstorms

is that currently all of the LEGO Mindstorms products, you could reflash them with your own firmware. There was very few people, you could count them pretty much on one hand, that actually made replacement firmware for those bricks. But Lawrence and David were one of them, along with myself. And so the PyBrixx guys guys they've made a full new firmware and an app that's that's a downloadable apps it's a web app essentially that you can put on your android

tablet or your pc and and it you can program the bricks now all of the bricks all the powered up bricks so that's the technic brick the boost brick the train brick and of course the mindstorms bricks uh you can program them all either with a block-based language that they've got developed as well, or with MicroPython. And that's part of the open, firmware-updatable product philosophy that I championed at the LEGO Group. How much did you have to push for that? How much were they against basically what you did?

Well, how do I put this nicely? I don't think they were fully aware of the openness that was there. So, I mean, the product has always been open. We've replaced the firmware on the RCX, the NXT, EB3. They all had kind of a way to put new firmware on it. So it wasn't really that much of a struggle to say, hey, we should open this up. The harder challenge is getting the documentation and everything out there. And then that's something that we didn't do all that well, frankly.

But the PyBrix team has really done a great job with documentation and making the app available. So it carries on. The nice thing about that open philosophy is that even if the manufacturer drops support, which Lego rarely does, but eventually they can't support things forever. So that's where fans like Pibricks have really kept those products alive. Switching questions or switching topics a little bit. This one's from a listener. Sahil asks, how did you get into test-driven development? That is a great question.

I was probably kicking and screaming. And that's really... In some ways, I mean, I'd read about it. And I read James Grenning's book, Test-Driven Development for Embedded Systems, probably eight years ago and or even more than that. And there's a couple of it took me three tries. So I don't know if this is a typical story or your experience as well. And maybe we can talk a little bit about how you two feel about test driven development. So for me, the first time I tried it was, hey, we need some unit tests around this code,

write some unit tests. And when you write unit tests for code that's already there, that's not very modularized or not very decoupled, you write really brittle tests, you write to get code coverage, and it's not really test-driven development. And the second time I did it, a few years later, I tried to pick it up again. And it was kind of the same story, but I could sense that there was something there.

It was getting better. And then I rewrote. I have a memory manager that I wrote called UMM Malloc. And it's basically a Malak replacement for embedded systems without a memory management unit. And that's when I wrote the memory manager and I wrote unit tests as I was going along, the way you're supposed to do it in quotes. And I finished it and I was like, hey, wait a minute, I didn't actually have to really touch the debugger while I was doing this because I wrote tests to match functionality. And after that, it was like, I'm never going back. And once we integrated it with one of our products, it just worked the first time out of the box.

So I'm fully convinced that TDD is the way to go. But how do you guys see and experience TDD? I love the theory, but I do not like the implementation. The whole CPPU test, the way the mocks work, I don't, it doesn't really work for me. When I'm bringing up a flash driver, none of those things help me. It's the code on top of that. And that I tend to do more in a sandbox way where I don't necessarily have the CPPU driving it. I just have a series of tests I can run

and I do it in a less formal manner. The thing that I really, really, really like about TDD is the idea that even before you start writing the feature, even before you really think about how you're going start writing the feature, even before you really think about how you're going to implement the feature, you need to think about how you're going to test it. So for me, it's a thought exercise because the implementation doesn't work the way I want it to.

But I think as a thought exercise of how am I going to test this before how am I going to implement it is just the way it should be. Waiting to think about how you're going to test this before how am I going to implement it, is just the way it should be. Waiting to think about how you're going to test it until after you've implemented it is going to lead to bugs. Yes, absolutely. And Chris, do you have any thoughts on this as well? I have different thoughts. One is just practical.

We're both consultants, and so we tend to be hired guns to come in and do something for some company. Usually they're very small companies. Usually they're in a hurry. Usually they don't enjoy paying a lot of money for things. And they don't like to see things they can't use right away. Or their code is already in some state., you know, so it's very difficult for us to come in and apply TDD principles in those situations. I'm thinking of a project I'm doing right now. You know, they want a demo sometime early next year and okay.

And they want some certain firmware features to work. And I'm building on top of, you know, a vendor's chip library with their drivers and things. And so it, you know, for me to sit down and say, okay, I'm going to do this in a TDD way. I don't actually even know what that means. I know what TDD is, I've read the book, but I don't know what that means in the context of a project like this, because I am not writing isolated code that I can test easily. And you're bringing up a driver of hardware. Right.

I don't want to write mocks for their hardware because I can't even see that deeply necessarily into their... There's a lot of complexity. But when you say things like, okay, I have this memory manager or I have a module or a piece of functionality, that's where I think it makes a lot of sense. And you can do that. But a lot of times I'm not in that situation anymore. I'm grinding low-level firmware stuff that's mixed in with ST's HAL or some random company's implementation of CMSIS. Or optimizing.

Like DMA is not something that is that interesting in TDD, because it's a cheap chip function. Yeah, yeah. So I just get confused by how do I apply this, and I end up going back to my old ways. But when I do have isolated functionality, and this was some stuff I did at Fitbit, where I was like, okay, this is the graphics library. It does mathematical transformations of stuff. I can put tests around this, and I did do that. That is very helpful. But I don't think it's universally applicable, at least given traditional embedded development and how chip vendors architect their stuff.

97 layers. Yeah, and we can talk about that at length, I'm sure. So one of the things that I do as a bit of a shortcut is, first of all, if I take a vendor library or any third-party code, I'm just going to assume it works, especially if it has test cases already, great. But I'm not going to bother testing a a hell like the stm32 hell so that's so we stop there uh the second thing is that we don't mock out their hell either because that's just not it's it's just a year just to do right exactly so the other the third

piece is don't test or that's worked for me anyways is don't test on the target. So assume all your testing is happening off target on your PC. And that kind of frees you up from having to bite off too much of a chunk. And when you're doing that, you might say something really interesting, Alicia, which is, I think about how to test it before I write the code. And one of the tricks I use is I turn that around and say, how is this supposed to work? What is this actually supposed to do? And then how do I test that that gets done? So that way, I'm actually writing the code incrementally.

So I don't really think about what are all the tests that I need to do to make this thing work. It's more what specifically does this have to do and how am I going to test that it did that thing? And we can talk, probably shouldn't talk a whole lot more about that, but it's an interesting topic. And once you get your brain around making sure that you can test off target and then leave all the stuff that thousands of developers use and just assume that it works, that kind of frees you up to work on the interesting part. But Chris, you know, when you're talking about a code base that's already there and, you know, your client doesn't want to spend a lot of money and they don't want to have a lot of additional tests, then yeah, you're kind of stuck to do it that way. And a lot of this stuff, like if I take it off target, there isn't anything. Sometimes it's a driver, and I'm moving a buffer from this driver to another driver,

and it's, you know, okay. And the question is how fast can you do that? It's a DMA and eight lines of code once you figure out how to do it. And the tough part is figuring out how to do it, not that you're writing a lot of code. Exactly. Yeah. So I think we're agreed that TDD isn't for the whole project. Yeah.

And it works in certain areas. But one other thing is that once you have a code base with TDD already baked in, then your newer developers are more confident making changes because they can just literally run the green test and hopefully everything works. And those should be tests that are run, you know, on check-ins and things like that. Yep. But then if you don't have somebody champion it, then when that person leaves the project,

they become failures. Fail tests and people stop paying attention. Yes, I've seen all that. Turn that unit test off. It always fails. Well, wait. Oh, wait. When was the last time we It always fails. Well, wait. Oh, wait. When was the last time we ran this test? Oh, yeah.

So that's the other part of the whole TDD thing is that it doesn't really work unless you have a pretty decent CI system. And one of the things that I brought in at the LEGO group was, you know, we were still, when I joined, people were still sort of bench testing it, you know, line by line or running motor tests manually. And then I said, hey, wait a minute. What if we kind of automate some of this testing? Well, we can't automate all of it. So what's the point? Well, okay. But what if we could automate 80 or 90% of it? Would that help? Yeah. Okay. Yeah. But that's so hard to do. So eventually, we kind of flew little pieces. You just have to bite little chunks off, right? And we ended up with a farm of like 30 or 40 Raspberry Pis connected to our Mindstorms bricks and, you know, exercising,

just continuously exercising code as we checked it in. And if you ask people now, hey, would you want to go back? No, but it does take a CI, you know do you want to go back? No. But it does take a CI, you know, you have to have one or two people there that are dedicated and, you know, love getting a CI, CD pipeline up and running. And again, use the tools that are out there. GitLab has a great series of, you know, GitLab runners and tooling that works in that realm. And then when we had one or two Raspberry Pis, yeah, you can set them up manually.

Once you get to 30, yeah, you're going to want to use Ansible or some other provisioning system to get your farm up and running. It's like saying you don't want a dishwasher because it won't reach into the sink and put them away in the dishwasher for you. I don't want to babysit 30 Raspberry pies and figure out what went wrong with them. I am not the person to set up your CICD system. I really respect the people that are. But when we work for little companies, it's sometimes hard to convince them. You need another person.

You need a whole other person just to do tool stuff because many engineers aren't geared to enjoying that. And it depends on the size. And there's some engineers that love it. So the trick really when you have a big team or even a small team is to not insist that everybody does all the things because they're going to hate, most people are going to hate doing 80% of the stuff. But the trick is finding, you know, finding a nice cross section of developers that where somebody really has a, you know,

a bee in their bonnet about TDD or about CICD or clean code or whatever, and make sure that they're the evangelists for that aspect of programming. So TDD is often paired with Agile. And Agile has a fraught history with actual hardware. I mean, some people love it and some people think, wait a minute, some of this stuff takes time.

You can't just change it willy-nilly. Some people think all kinds of things about Ag agile. Some of them are in this room. Well, that would be my first question to you is, in your world, what does agile mean? Does it mean just running in sprints or what is agile in your mind? That's the main problem for me. It's kind of like programming languages without actual specs. It could be anything because it's... It's Agile. Definition changes. Is it stand-ups? Is it points? Every time I've done, quote, Agile at a company, it's been mostly Scrum and stand-ups and retros and all of that stuff.

I can tell you, I don't want daily meetings. Point poker and a bunch of stuff that, frankly, did not help. I do want to interface with the users early and make sure that their needs are taken care of. Yes, but at every company I've worked at, the user was just my manager. So many times. And we have a whole product owner team

that deals with the customers, and you're not going to get in front of those guys. And yeah, it's a challenge. So how does Agile work with TDD? So first of all, I don't think that, well, in my mind, I haven't seen Agile work really well in a lot of places. I've been in one project, my last project that I worked at at the LEGO Group, where we had, it might have just been an amazing team where things just kind of clicked. I had a project manager that was really good with the upward communication and with all that sort of stakeholder management stuff, because I tend to say no a lot and I'm not good at it. And I kind of handled the technical side of things. And then we had a complete cross-functional team. So we had some electronics, we had, you know, board layout, we had

BOM specialists for when I say BOM, building materials, not actual BOM. And then we had, you know, the whole procurement interface and then the firmware and production equipment. So it was a real, it was really cross-functional. And and as you know in the life of an embedded product you're going to be heavy up front with electronics and and board layout and and bill of materials and procurement and then we have firmware and then once that's all up and running you might get your dev your dev boards in and you have to integrate and then eventually you have to put it on production equipment. And we took advantage of the kind of cyclical nature of that so that while the hardware guys were sorting stuff up,

we were writing the system or the firmware off target without an actual hell, right? So the whole life cycle of, you know, boot time and heartbeats and task management and all that stuff that happened off target so that when we got hardware, we just had to make sure that it, oh, and we used the nuclear development boards, of course. So those are, you know, those are, you can get most of your stuff working on a dev board. When the real hardware came in, it generally took a day, maybe two days sometimes, to get it up and running. We had exactly one pair of wires that were the wrong way, RX and TX, of course, on a UART. You never get that right the first time. No, no.

But fortunately, out of, I think, six boards, we only got it wrong on one board. But it was actually, it wasn't RX, TX. It was the wrong UART port. But anyways, we got that sorted out. And so you take advantage of those cycles. And how does this tie in with Agile? Well, over the course of the six or eight months we worked on this, we could kind of come together for a two-week sprint and say,

hey, what's the most important thing that we have to accomplish as a team in this two-week sprint. And setting sprint goals, Martin Dallman has a great book on agile development using sprint goals. And he feels that sprint goals are one of the most important things. If you can't state what your most important thing to do is as a team, then it's really hard to use agile effectively. But again, that worked for me and my team on that project, but it's not everywhere, especially if you're trying to push Agile into a waterfall world, right?

Because then you're just doing Agile ceremonies. It's not really Agile service, I guess you could call it, right? It's not really agile so you could call it right yes it's uh it's not really agile and agile isn't you know let's make all the changes at the end and it's not that either there's definitely some planning that has to happen but what martin says is use make humble plans plan more plan better later does that make sense? Yeah, that makes sense. And I agree with all of that. I mean, certainly it's always, it's difficult to not have visibility into short-term goals. It's easier to work on small pieces of things. It's good to have, and this is kind of anti-agile in a way, but it's good to have requirements and things you're working towards that guide all of that. Oh, yeah.

So, I mean, I don't disagree with any of that. And the best teams I've worked on have been small teams that kind of just did all that kind of thing naturally. And they communicate. Agile is about communication and not going off into your closed door office and doing something that you never- Closed door office? It's 2023-ish.

It's dream, okay? First of all, we don't go to an office. And second of all, if we did, it would be, you know, basically football bleachers. Somebody went off into their closed-door office and came back with a grand thing that didn't integrate at all. And that's no good. Yeah. Or, you know, the manager decides they're going to try their hand at code because they haven't done it in a while. Sorry.

That's always a good, yeah. Well, that was my downfall as a manager. When the team is shorthanded. Oh, me too. Just apply, I got a spare pair of hands and a keyboard. I can help. And I did. I mean, it's not that we didn't help and do things.

It's just, that's the other thing that's hard for developers as they transition into management is keeping your hands off the keyboard and keeping your eyes on the big on the big picture but usually when the team is shorthanded or you know stuff happens uh you're going to want to help and try to help and pitch in so that's that's it's hard to manage those things yeah were you already a manager when you went to lego group no i was not no so that was a i mean it's you know i've been around managers and i you know i've been a lead developer and uh heading up uh root causing teams for you know when there's really when problems really hit the fan and uh customers are ready to pull the panel out of the wall that's when i would

kind of go in and kind of do root cause analysis so So I'm aware and I knew how to handle, you know, a reasonably sized team. But this was the first time I'd been in a formal organization with leadership teams and all of that. But the surprise for me was, hey, senior manager at Lego, you know, you walk into the office the first day. So where, you know, where's my office? And right here next to the PA. And it's just everything was open. So there's very, very few closed offices at the LEGO Group anywhere. And I know people have strong opinions about that.

In some environments it works. And there's other days when we just work from home because it was, you needed to get work work done so that was a little bit of a uh covid was an interesting time for us for sure you know you work uh you were able to get quite a bit of work done at home and then there was there's some hey wait a minute why are people why are people also effective at home as they are in the office and i'm sure there's a there's a lot of debate going on around that today. Yeah, it did show some things we thought were true or we thought were, some assumptions were not correct. Yeah, well, people have a lot of investment in real estate, so.

Yes, they do. How did you, what advice would you want to go back and give yourself as you became a manager from a tech lead? I think I didn't really appreciate how difficult that first level of management is. So the interface between your team or an extended team, the director level and higher levels is you're really stuck. It's, you know, your team is counting on you to kind of run interference for them and get, you know, get resources. But in most corporate structures, you've also got, you know, fixed budgets and, you know, allocation cycles and all of that stuff.

So it's a really, it's a tough job. And I think the advice would be to listen to folks like Michael Lopp or Zoran. Just listen to as many manager-type podcasts as you can and kind of figure out, is this really what I want? And I wish more and more companies had a ladder for technically competent people rather than folks that want to get into management. Both are needed, right?

Yeah. And the bigger companies tend to have those now. Yeah. Smaller companies still, I think, are like, oh, you've reached your limit. It's time to become a manager. It's like, what? I think that in order to become an individual contributor at the next level, having some management experience is probably good.

Yeah, but you don't want to inflict that on a poor team. It's like, here, go figure out a few things. You know, that's not in the best interest of the company either. So there's a balance there. No, it's a completely different way of thinking it's you've really got a it's a it's a completely different way of thinking about uh about projects and resources and all of that stuff and and honestly i'm i'm quite happy as a as a staff engineer working on certain kinds of problems and then also not just trying to advocate for not being you know the the person that gets all the hard

and interesting problems uh trying to bring on juniors while you're doing your root cause analysis so that they can kind of learn from it because it's it's no good as a as a tech lead to be an additive effect you want to be a multiplier so that means getting your you know getting your knowledge and your wisdom and and any mentoring that you can do out into your organization rather than, because nobody reads documentation. There's no sense, you know, writing down your after actions because nobody reads that stuff. It's much, much more effective to be working shoulder to shoulder

with somebody who knows what they're doing. And then also to, you know, ask other people for, hey, what would you do in this situation? What's your next step? So that's what, if I was working together with somebody who is less experienced, it's okay, here's what we know. Here's where we are.

What's your next step? What do you think is the next thing to do? And that's an effective way of learning, I find. For better or worse, Uncle Bob, we all know, if I say Uncle Bob, we all know who we're talking about, Bob Martin. He had an interesting thing at the beginning of one of his talks, and that is that the number of programmers in the world doubles about every five years, right? And if you think about that from the other side, that means about half the developers have less than five years of experience. And that's kind of scary. I have thought that the role of senior engineers should be primarily to make more senior engineers.

Yep. Our role is to help people figure out if this is the career they want and then to grow in the career, not necessarily to get things done, which is hard because I really enjoy getting things done. Yeah. But it's also not our particular careers that doesn't work because... It works some. Okay. Well, you do outside mentoring and stuff. I do outside mentoring. And even in some of the gigs I have, I do manage to structure it so I have somebody to play with. But they aren't necessarily people who are becoming senior engineers. They're people who are learning about embedded systems on the side. But it's fine.

Some of them. But it's challenging as consultants to do some of these things. It is one of the things that I miss about regular work, full-time work, is being able to do a little bit more mentoring outside. I mean, I have formal mentoring relationships with folks, but they're all like, I can't help them in their careers directly. I can only advise. Right, right. How did you, so you said you're 40 years into your career. We're about 30 almost. And sometimes it's exhausting.

How do you stay engaged? Well, we've both been fighting burnout. I just thought you were asking the question for me. No, not just for you. Okay. Well, no, that's a great question. And you're going to go, I think if you're going to be in this, you know, life, I shouldn't call it lifestyle, in this career for any length of time, yeah, you are going to go through cycles. It's not all going to be a nice, you know, happy, happy journey.

Try to stay, Try to keep learning. So one of your previous guests, try to learn in your adjacent field. So if you're interested in how things work, you maybe want to get a 3D printer and try to figure out how can you optimize that kind of thing. So try to always be continuously learning. Watch what the cool kids are doing. So I learned on VI.

So I have all those VI keystrokes are in my fingertips, and I was very reluctant to give it up. And then I saw somebody using BS Code. I was like, hey, wow, this is pretty cool. And before that even, there was like, oh, we got to get into Docker because that's really awesome. I'm like, I don't need Docker. I've got VirtualBox or some other VM systems that we were using. And now for embedded systems, I use Docker and VS Code, so I can just spin up a Docker instance with GCC and Sphinx and Python

and all the stuff that I need for embedded development. I put the host GDB interface, so the ST-Link, if I'm using that, or Seger or whatever, run that on the host machine and then connect from the Docker machine to the debugger, to the GDB backend over a port. And hey, bingo, all of a sudden I'm using VS Code as an IDE. And that's just cool. So I guess the real way to fight burnout is to keep doing interesting things, find cool people to talk to. And always hang around younger developers because they're bringing in new ideas all the time.

And it's just a cool way to stay engaged. What are your preferred ways to learn things? That's kind of a question that I had for you as well, right? Which is, would you prefer to learn at a conference? Do you want to do kind of individual learning or learn as a group with a team? And for me, it's when I need to learn how to do something, I kind of go into it sort of shallow depth and just kind of get things done. So a little bit more cookbook style. So I'll find a decent web tutorial and follow that for a while and then

at a certain point you have to kind of decide to either dive in or just stay on the surface right and when i when i do want to dive in then i'll just take the time i'll book out a half a day or something like that just so you don't get overly burned out and just really deep dive and learn learn the stuff that i that I need to learn. But before I do that, I kind of write out a list of here are the things that I want to accomplish with this new tool or with this new idea and then try to apply it right away. Does that make sense? Yeah.

The way that I prefer to learn is to snooker people to come onto the podcast and talk to me about what it is I want to learn about. So that's been pretty nice, actually. You've had some awesome guests in the past. I've really enjoyed listening to some of the back issues of the Embedded podcast. Thank you. It depends on what I want to learn. I've been in a multi-year obsession with origami, so that continues, and I go in different directions,

and then I go back to the curves that I like, and I don't have a purpose for that, and that sometimes is the hardest part about learning, is when I don't have a purpose for that. And that sometimes is the hardest part about learning, is when I don't have a goal. My problem with the work you did with reverse engineering is that I would have gotten partway into it. And then I would have looked around and said, oh, wait, I can't bill for this. And it isn't helping anyone. And then I would talk myself out of it and i know i need to do silly things silly things are fun silly things keep you engaged but i'm not great at giving myself that permission

with origami i can say well it's mathy and you're using computers and you can use your hands and keep you away from screens which if you look at all of those together that can't all be true but yeah i so can you actually simulate uh you mentioned using computers to to uh work out your origami uh work so can you actually simulate some of that on the computer and then go from there or amanda gossi has this amazing web simulator that simulates origami. And there's a whole bunch of theory about developable surfaces and whether or not things collide and the pliability of your material. So her web simulator is just fantastic. I think it's origamisimulator.com. When I do math, it's usually in Python making patterns that I can then repeat, usually slightly changing. So if I want to get an ocean wave, I can do a big sea and then a sea that's a little less and then a sea has less, but a few more like sine waves at the end.

And then I can end up with a 3D curvy wave that is singular. Boy, is that not a great description. Well, I hope we'll add that to the show notes then, because I think that sounds like an interesting thing to look at. An interesting rabbit hole to go down for a little while. It is, and I have learned quite a bit about Python and SVG and plotting things and how snails actually evolved. Yeah, it's been fun. But you can't bill for that. But I can't bill for that, no. Yeah, so that's the trick there, I think into kind of model trains, or my dad was anyways. And I always looked at this Merklin, which is a German brand of trains, a Z scale, which is like 1 to 220.

They're tiny, tiny little trains. Anyways, I finally found one and bought a kind of a done table that was all, it was years and years old. And I needed to upgrade it. So then I thought, hey, this is a good opportunity to learn about Arduinos. And, you know, built a little Arduino controller for it, little pulse width modulators for the different track sections so I could run the trains at different speeds. Totally not billable.

But today I'll be working on a little Arduino tool that I can use for testing certain aspects of a fire alarm system. So you never know. Things just come around. And I think learning adjacent, like just a little bit off of where your day-to-day work is. And sometimes that sneaks back in and makes you more productive or gets you something that you can build for. You just never know. It's never really wasted.

I mean, learning five years ago about snail shells, I can't believe that actually affected my origami. So, yeah, it's all really connected in interesting ways. Chris is staring off into space. I think because otherwise I'll make him talk about learning more music stuff. Well, I mean, the answer for me is I hate computers, and doing computers outside of work makes me itch. And it's funny sitting in this room filled with computers.

Well, I know. They surround me. That you have put together. That's because it's the only way to do anything anymore. I mean, I'm not going to get a tape deck. No. Get Christopher a tape deck for Christmas. Uh-oh.

No, this stuff is still fun. You know, I'm, you know, yeah. I have a lot of things I like to learn. It's been trouble for me. I go in and out of having technical projects. I'm working on some ham radio stuff. That's not exactly computers. It's electronics and some ham radio stuff. That's not exactly computers. It's electronics and things.

It counts. That's how it is useful. It counts. But I, yeah, the last few years, the more computers I use, the less I just enjoy being around them. But as soon as you need to work on an instrument project, you are there.

An instrument project? Yeah. What do you mean? Somebody comes and wants to build a different arpeggiator. Never happened. We've had people talk to us about instruments. But not they've even asked me to help. No, but someday.

Someday they will. Your clients will come if you build it. I don't know. Well, just as a fun side project, once I built a simulator on an STM32, I built a simulator of the Game Boy sound synthesizer. Oh, yeah. Believe it or not.

So you don't have to tell me about the crazy stuff that you shouldn't do. But you know what's great? I think you folks are in Santa Cruz area, right? Yeah. So you're in an amazingly advantaged place because you can just sneak up over the hill and head over to Big Basin and, you know, take your dog for a nice walk and all kinds of really amazing wildlife. So I envy you your location. It's a great spot. And the beach is awesome, too. The beach, our local forests, our lakes.

Yes, we are. Lakes. They're ducks. They're. Lakes. They're ducks. They're ponds. They're ponds, yeah. Ralph, it has been really good to talk to you. It's fun to talk to folks who've done things. I mean, that have done lots of things.

I feel like we could talk for another hour and a half and still not really get to, you know, being bored. But we should let you go back to your regular life. Do you have any thoughts you'd like to leave us with? Back to our regularly scheduled programming is what I usually call it. Yeah, actually, I guess the final thought is 99% of the smartest people in the world don't work for your company. So, and code is a lot harder to read than it is to write.

So just, you know, when you're faced with third-party code or something else, just trust that it works or take the time to understand it. Don't just blindly rewrite stuff because you don't understand how this person understand it. Don't just blindly rewrite stuff because you don't understand how this person wrote it.

Basically, always look around first instead of writing more code. Try not to write more code. That's what I'd like to leave you with. Can I get really, really mad for 15, 20 minutes before I accept that maybe I should look and understand their code? Yep.

Okay, good. Absolutely. Absolutely. No, get frustrated and then, you know, time box rewriting. But just don't rewrite. I'm not going to rewrite anything. Don't succumb to the temptation of rewriting code. That's my biggest advice right now.

More comments. That's all. Just more comments. Our guest has been Ralph Hempel, formerly Senior Manager of Firmware for Product Technology at LEGO Group, currently Firmware Staff Engineer at Edward Safety. Thanks, Ralph. Thanks for having me. It's been a real pleasure talking to you. Thank you to Christopher for producing and co-hosting. Thank you to our Patreon listeners Slack group for their questions.

Thank you to Nordic for Slack group for their questions. Thank you to Nordic for sponsoring the show. And thank you for listening. You can always contact us at show at embedded.fm or at the contact link on embedded.fm. And our quote this week is from James May. Someone once told me that I was 12 inside. The only thing 12-year-olds crave is more Lego. Lego is fun. It's therapeutic. It's a beautiful sensation when you click the pieces together. How does Agile work for TDA? Am I supposed to cut that? I hope so.

Yeah, cut that. But knowing you, you'll put it back in at the end or something. Now you've told me.