Embedded - 423: Speaking of Aardvarks

Episode Date: August 4, 2022

Phillip Johnston joined us to talk about how engineering approaches can change over time.  This conversation started with Phillip’s Embedded Artistry blog post How Our Approach to Abstract Interfaces Has Changed Over the Years. His new course is Designing Embedded Software for Change.  Embedded Artistry has a Design Pattern Catalogue (though Elecia was looking at Software design patterns on Wikipedia during the podcast). https://github.com/embvm  Phillip is working with Memfault on an ongoing embedded systems panel. The first topic they covered was observability metrics for IoT devices. There is a panel coming up on how to debug embedded devices in production. Some reading that Phillip mentioned: Toward a New Model of Abstraction in Software Engineering by Gregor Kiczales A Procedure for Designing Abstract Interfaces for Device Interface Modules by Kathryn Heninger Britton, R. Alan Parker, David L. Parnas Designing Software for Ease of Extension and Contraction by  David L. Parnas (1979) Design Patterns for Embedded Systems in C: An Embedded Software Engineering Toolkit by Bruce Powel Douglass Best Paper Awards in Computer Science from Jeff Huang  Creating a Circular Buffer in C and C++ - Embedded Artistry Aardvark I2C/SPI Host Adapter - Total Phase    Transcript

Welcome to Embedded. I'm Alicia White, alongside Christopher White. We're going to chat with Philip Johnston of Embedded Artistry about changes. Hi, Philip. Welcome back. Hi, thanks for having me back on. Could you tell us about yourself as if we met at an Embedded Systems Conference? Sure. I'm Philip Johnston, the founder of Embedded Artistry. We are an embedded systems consulting and education firm.

I've been an embedded developer for 13-ish years now, and in that time I've helped ship more than a dozen products across a number of industries, including desktop software, consumer electronics, defense, robotics, drones, you name it. My main interest is helping to bring embedded software development out of the dark ages. And currently, I'm focused on doing that through a three-pronged approach of designing software for change, building testable systems, and leveraging automated software quality enforcement practices. Where's the fourth prong of getting people to listen to you?

You know, I try not to worry about that one too much because I can't control it. Are you ready for lightning round? I am. This is always the most stressful part. What was your first computer? You know, I don't know, but it ran Windows 95. What does that look for? I just expected it to be older than that. What was your first programming language?

C. When you were five, what did you want to be when you grew up? If you can't remember when you were five, pick any later date. I do remember that I wanted to be a paleontologist. I still want to be a paleontologist. I still want to be a paleontologist. Yeah, I think it would be quite cool. When you were starting college, what did you want to be when you grew up? A philosopher. Okay, continuing on this theme, apparently, what do you want to be when you grow up now? I guess now I'd settle for Gardner. When you were in college, how much RAM was a lot of RAM for your computer? In college, I want to say I was in the gigabyte range.

I don't remember exactly. But my first computer, I do remember my father paid, I think, $1,000 extra to get the four megabyte option. Yeah. All right. Let us... extra to get the four megabyte option yeah all right let us um you don't know how much an extra 16k was for for my first computer yeah and you had to put the chips in by yourself they went into chips into sockets yeah i luckily missed all. You recently posted a blog post about how your approach to abstract interfaces has changed over the years. And it caught my eye because you don't always think about how your design philosophies and implementation principles changes. Could you describe what you wrote?

Sure. So I guess backing up before the article, the main problems that I'm interested in solving are related to the fact that no matter how well we think we design our software, there's always like some change that comes in, either a part changes, or you got to use a new processor, or some requirements change comes in. And then it's not easy to make that change. It's always a huge rewrite, it seems. And you know, it's not just swapping out a couple files, suddenly, I find that whatever I'm changing has touched every file in the repository, and now it's a big effort. So that inspired a lot of my interest in, well, how can I solve this problem? And abstract interfaces, through the work of David Parnas

in particular, and the ideas of information hiding, were something that really struck me when I first came across it, and that I've employed with great success in my career. So I described how when I encountered abstract interfaces and actually use them in systems, I would, I guess, go through the typical exercise of, let's say, I have a SPI or an I2C interface for a SPI controller or an I2C controller. You know, by having a standard interface for that, I can swap out implementations using the same interface and any software that relies on that interface doesn't have to change. When you say interface for like those things, you mean like read, write, start? Read, write, start, stop, depending on what it is,

configuration may or may not be included i think spy configuration is probably a bit more standard than you know configuring an accelerometer where every accelerometer has different frequencies and filtering options and wake up events and things like that and i always i mean the posix open close read write, octal has always been in the back of my mind as a, let's just standardize the interfaces to everything. Okay, but now you don't think it should be like that? I mean, that works. That works for a lot of things. It works for a lot of things. I've done a lot of really cool stuff with it. Like nowadays, I almost always write new drivers for new components on my development machine

using like a Aardvark debug adapter. So I'm connected to whatever breakout board I have over USB and I can talk to the actual hardware. I can write my driver. I can write tests against real hardware. I can record data and play that back in automated testing. And most importantly, I can move it to my target processor and it's just going to work for the most part. There's always some gotchas that might come up with memory or timing or your implementation isn't quite correct. Things like

that do still happen. But for the most part, I can easily move stuff around. So that has a lot of benefits, but there are downsides that I think come up quite a bit. One is you're always locked into a particular interface. So the goals of like, okay, now I can change what hardware I'm working on have been met, but I can't necessarily take that driver and use it in a totally different ecosystem or a totally different client project that's not using all of my abstractions and supporting tools and things of that nature.

And so you're sort of become locked into whatever ecosystem or whatever set of interfaces you've developed for yourself. And two, and I think this part is, you know, it's very difficult to overcome it. It's hard to create good interfaces without a lot of experience and getting an abstract interface wrong and having a lot of different implementations and then finding out that you have to change that causes just as much cascading change to happen as, you know, if you didn't have it in the first place. Now, I mean, that's true of whether or not an interface is abstract or not, right? Interface changes are always painful. But the scope of the interface, I think, makes it

more likely that you're going to do something erroneously that you want to change later. And so, you know, my efforts shifted after being exposed to just, you know, randomly through a client project coming across a driver that did something totally differently, where instead of depending on some particular SDK or particular abstract interface for a spy controller, this driver just defined one function that you had to implement in your own application to handle the spy transactions and that was just a totally novel approach to me and you know it addresses some of these concerns of well i just have one function so the odds that you're going to define one function correctly that you know in such a way that the interface you create is going

to be able to meet the needs of that component within your system and to support multiple implementations, that's going to be a lot easier to get correct than if you're going to define a fully- to you, like being able to control a device in a standard way, you know, you're still locked in. Whereas if I have one function to define in my driver implementation, you know, I can, I can use it with an Arduino SDK, I can use it with my own framework, I can run it on Zephyr, you name it, right? Implementing one function is easy. Implementing a full interface always uses a particular IMU. And it runs library code on my processor in order to do this. So I understand, library code on your processor to speak to the device? Yeah, so they have a library that implements, say, the Kalman filter.

Right. And I'm not allowed to really look in there. Right. And what they want me to do is be able to provide the code that talks to the actual unit. Okay. Because they know how to do all of the registers on the unit, but they don't really know whether I'm using spy on a Cypress or spy on a PIC. Gotcha. Gotcha. Okay.

So I have this library that wants to be able to send and receive messages to the device that it was written for. And this is kind of a complicated example, but there's a lot of, if you think about the libraries, that means that you have something separate and something that you do want to think about changing for the future. And I can see how in the past, I would have expected that library to ask me to implement a driver that had an init, a re-init, maybe a reset, a read and a write that would do lengths. But actually, in fact, recently when I had this entire problem, it had me implement SPI transfer, read-write, and it didn't want any other functions.

It didn't want to know about ints. It didn't care about resets. That was all up to me. What it wanted was to be able to talk to the unit. And so it was only one function I had to implement. Is that what you mean? Or is that, am I on a different path? That is exactly what I mean. Same, exact same example. That's when you have control over something somebody else is going to implement the lower levels for. In this example, I would be the library developer. Or in a different example, I would be the GUI developer wanting to talk to an LCD, but not wanting to have to deal with whether it's hooked up in SPI or parallel.

I just want to send things. Mm-hmm. it's hooked up in spy or parallel. I just want to send things. Is this abstracting the hardware, or is it abstracting the software, or does it matter? I don't think it matters. I think that we can view it as a general technique that we can use in our systems. So, you know, one tool in the toolbox is the full abstract interface, and that's still useful. Like, you know, having a standard way to talk to a device from your application code that allows you to swap out the device you're talking to, or it could be anything. It could be a particular library. It could be a database, whatever. But having some abstraction

on top of that so you can swap out an implementation without impacting other code, that's great and that's one tool. On the other hand, I think another tool that we should have in our kits that we should be able to employ when we need it is to realize that, okay, we don't need a full interface for this problem. I really just need one function or there's one particular detail that I can't manage in this library that's going to change when some aspect of the system changes. That might be what implementation I'm using, or a specific configuration, or maybe some applications are threaded and some aren't, and I need to have a way to defer that kind of decision to the application. So I think there are a number of ways that this can really benefit our system design that aren't just decoupling from hardware.

The old way of implementing abstract interfaces in your blog post, it mentioned function pointers. And so the library in this case would say you have to implement a structure that includes these functions, the read and init and all of those functions, and you do the function pointers. That was an extremely TI thing. I mean, they did that, I would say, to excess because sometimes there were 20 or 30 functions you needed to put in. Where did you pick it up? Please don't say my book. I didn't pick it up from your book. I must have been exposed to that idea.

It was either at Apple. It must have been at Apple. I was working on the diagnostic firmware team. And I think they managed a lot of things that way. Like the code was based on the EFI framework, and we definitely had some function pointer structs and abstract interface ideas in there. That's funny you mention Apple, because this whole idea of having a framework that does something, but it needs help, and the help can vary, so give me the one function that will help me, is very straight out of the UI frameworks where there are delegates and things,

these callback functions that maybe something draws a table for you, but it needs to know how to draw each individual cell. So it can do all the machinery and lay things out, but it doesn't know what you want to do for drawing the cell. So here's this function, draw cell that you have to fill in. And then it takes care of that low level part of the drawing based on what you want to do, which is, it seems very similar to me of, okay, here's this complicated driver that maybe has lots of high level intelligence, but really it also needs to know how to talk to its device. And please tell us how to talk to the device.

And that's all I need. That's interesting. That's the word we should be using. What? This function that we need to implement for our API or our library. This one function. Or even the five functions in a driver that they want us to implement. Those are delegates.

Those are things that they have said, we need you to implement this. It's not a delegate like it is in Swift, but it is a concept. And I think for many things in C, we don't always have the word because we don't have the concept. The design pattern. I guess, but is the word delegate and how it works a design pattern? I guess maybe. I don't know. I don't remember.

Anyway, so delegates. And then I feel like, what about callbacks? Those aren't delegates. They can be. It depends on their function. That's another thing I see with some library implementations is that they don't want to tell me what's happening inside, but if they need to, like a UI library needing to get images from somewhere, it doesn't want to know where it's getting images from. It just wants to call a callback that says, give me more data. Yeah. Can you tell we've been working on UIs lately? I can tell. Okay. So do you use the word delegate in your blog post? So I'm going to get to that, but I want to address another point that you had made earlier and then come back to this.

Okay. So when you said the TI does it to the extreme and there's, you know, like sometimes there's 20-something interfaces you have to implement, I think that's an anti-pattern, right? Because you don't want to have to implement 20 interfaces to get something up and running. Like that's a pretty significant time cost, where essentially, you're going to take whatever implementation already have, and I'm going to shim it to this TI implementation interface expectation. Like being able to do that on a smaller set of interfaces is much easier, right? You're going to you're going to be able to react and change and swap things out and get things running much easier if you have a smaller set of things to work with. So I think that TI example is a good indicator of where this kind of abstraction approach can go wrong. It is. Or not wrong, but difficult, perhaps. It's like they have a whole operating

system and they do it with function pointers so you can replace things. And they do give it all to you for the default version, but you can change every little thing, which means that you have to think about changing every little thing, which I never really want to even think about it. Because then if I do it, I've done it wrong. But if I don't do it, maybe I'm doing that wrong. Anyway, back to you, Philip. Right. So, back to the delegate question. I don't do it, maybe I'm doing that wrong. Anyway, back to you, Philip. Right. So, back to the delegate question. I don't call it delegates. I think it's a perfectly suitable word. I view it as the template method pattern, and I think that's just, I was exposed to that before I was exposed to the concept of a delegate, but I think they serve the same thing. And so, that pattern comes from, as far as I know, the classical Gang of Four design patterns book.

Yeah, okay. And basically it says that you have a function, the template method, that has some steps that are controlled by that function and some steps which are defer function requires the user application to implement it, or it provides an optional hook, which I think a callback applies on the same conceptual idea that allows an application to override or modify slightly how an implementation is doing things. And with embedded systems in particular, I think that just that idea of like, okay, we can have a function, I'm going to have some steps that I'm going to defer to the user that they have to implement, like a spy transfer function for your particular platform. And some things that, you know, you can override if my particular default implementation doesn't meet

the needs of the situation. I think that's a powerful tool because, you know, we do care about performance in the embedded world, perhaps more than other software systems with multiple gigahertz processors and gigabytes of RAM have to care. And so I think it's probably more common for us to need to tune the implementation, although I'm sure, you know, now that I've said that a thousand counter examples can be thrown my way showing how I don't know what I'm talking about. It's funny, the template method, I was trying to remember why I don't think about that one. And it's, it's partially because it says very clearly that it is based on inheritance. And so

as soon as it says that, I'm like, well, let's just check that one out. It's not like I can't use inheritance. It's just that I don't have a language that makes it easy. So, but you're right. That is what it is. You're taking pieces out and allowing their implementation to be done elsewhere. I think this is one of the trickier points of design patterns, right? It's easy to look at a pattern and say, oh, this doesn't apply to me because of whatever thing. And yet there are still going to be aspects about a pattern that are crucial. Like, you know, we could debate and it's probably, there are very good points to be made on either side about whether or not inheritance is really key to the template

method or not. In my interpretation, I just kind of ignore inheritance altogether. Like, I think you could achieve a template method through any number of ways that don't rely on inheritance, although that was the classical application of the pattern. The thing with design patterns is that there are a billion of them. And at some point, I can't sit down at my desk and say, okay, today I need to fix this bug or make this LED blink at some frequency. Which pattern should I use? It's not that it's metacognition, but there is the chance of having design where you don't really need it. The best places I've encountered patterns and been able to use them is when the thing I'm working with enforces them. Coming back to Apple,

they're very opinionated about the way the language is set up and the way the frameworks work. And you will use these patterns or you will be unhappy with your life. And that's the best way I've learned about them is like, well, this is what's happening on this project. These are the patterns. Certainly I've never come into something and said, well, time to flip through this book and see if I want an adapter for this or yeah. Um, I don't, I don't know how to bridge that gap unless you're doing a lot of whiteboard design and trying to be very abstract. Well, I guess for me, I mean, I agree. I don't sit down and approach a problem and think about what, what patterns apply. Um, I, I guess the, you know, if I'm trying to analyze my own process, which is always fraught

with error. But I think when I'm reading about patterns or I encounter a new pattern, you know, in the back of my mind, I have a general set of problems that I'm sort of always dealing with or, you know, something that I'm struggling with at the moment or something I'm unsatisfied with how I implemented it. And when I'm reading patterns, often in the back of my mind, I'm running through my collected set of problems and seeing how this pattern might apply to this particular situation. And that can lead to interesting results. Or I might run an experiment and see how it works. And then having implemented something, it's more likely to be fresh in my mind know, I might run an experiment and see how it works. And then having implemented something, you know, it's more likely to be fresh in my mind or I might like a particular thing in

a given pattern and realize I can apply that one little piece to some problem I'm dealing with. So I think for me, that's how it usually ends up working out in practice. And there's definitely a couple of them that I tend to reach for because I've learned to use them and, okay, this is the way I do things. Like you said, there's a billion of them, but there's like three or four that are sort of key core concepts that whether or not you know you're using them, a lot of times people will be using. Yeah, but sometimes I find them confusing

because there's publish-subscribe. That's something I'm very familiar with. I've worked with the robot operating system. It's all about that. If you ever want to see publish and subscribe on a large scale, go there. But then when somebody said something about the observer

pattern, I was like, which one is that? And it's PubSub, but they have different names. And if I use the state pattern, which is a state machine, which I use all the time, do I have to call it a state pattern? I mean, only if the people who are reading it care. I don't think there's any board of certification saying, well, I'm sorry, you called this something wrong, so your code doesn't work. But like Philip was saying that he comes across it and then he thinks about it for a while in the future. How do you come across them? I'm always reading. I think that's just, it's been a perennial thing in my life that's ongoing.

I'm probably reading something an hour or two a day whenever I get my free time. And so, it's not always technical. It's not always a book. But whenever I encounter ideas that are interesting, I either play with them in my mind or I have some project I can apply it to to see if it solves my problem. Or I just make some notes and start coalescing the ideas over time, especially if I keep running into a given situation or a given pattern, you know, and I notice that my notes are building some momentum and actually of sufficient size. It's probably something I should pay attention to. What kind of books do you read to randomly come across software patterns? Well, there's great design pattern books.

I mean, design patterns, the classic gang of four books is one. Your book has design patterns. The head is one. Your book has design patterns. There's the head first one. If you're new to design patterns, I've really liked the head first one. It's written in Java, but it makes things physical in many ways. That is what I need to make things work in my head. I'll have to look for that one. I don't think I've heard of that.

There's the Bruce Powell's book. I can't think of the name right now, but he has some, it's something along the lines of like embedded software patterns and see, or, you know, we can, we can find that book title and link it in the show notes. So we're not misleading people, but that's another one. And papers. I mean, there's like a whole conference on the pattern languages, the pattern language of programming, something like that, where people are submitting new patterns. And I occasionally look through that to try to find anything

that might be relevant in the embedded space. You actually go out and look at academic papers for design pattern conferences. Are you research shaming him? I don't mean to be. I guess I'm research shaming me. I read a lot of academic papers. I don't necessarily seek out papers about patterns. That's just one that's in my radar. But I found a lot of really valuable ideas from papers. Like one of my favorite papers of all time is a David Parnas paper from 1978. And again, I don't remember the title off my head, so we can link that one too. But it's, I mean,

Parnas describes like the exact same situation that I deal with all the time, and embedded software being difficult to change, you know? And so, there's ideas that are still to be discovered or not well socialized, because I don't think a lot of embedded software professionals are reading the software academic literature to look for new ideas. Another paper I like comes from the 90s. It's in a similar, this one's called Towards a New Model of Abstraction in Software Engineering or something like that. I should have practiced all these titles before. I didn't. But that one is related to Joel Spolsky's famous Law of Leaky Abstractions, where the author of this paper notes that, you know, we have these ideas of what abstraction provides us, like, you know, it supports change, we don't have to worry about

the implementation, you know, the general things of that nature. And then he, like Spolsky, points out that no, generally, when something goes wrong with an abstraction, it like is related to the implementation causing some performance problem that impact your system. And the general recourse to that, which I think we're all familiar with, if you're using an open source project that doesn't quite work the way you want, is now you have to take over the implementation yourself, right? So you went from having this nice abstraction or library or, you know, driver or whatever you got from someone else, and it just doesn't work well for you. So now you have to reimplement it. And I mean, you know, nobody's going to be surprised

by that, right? But in the ideal world, you'd be able to tune something about that implementation without having to take on full ownership of it or write your own implementation or, you know, whatever other recourse you're stuck with. And so he talks about this idea of, well, don't just make one interface for your components. Why don't you make two interfaces? You have the traditional interface that, you know, we use to interact with the component that we'll design according to information hiding principles. So we're hiding the implementation and we can swap out implementations if we need and our application code can remain unchanged.

Like we can still have all that. We just also need a second interface that lets us tune the parts about the implementation that we need to change. And when you think about that problem, like, okay, this is a paper that was written in the 90s. I've never heard anybody tell me that I should think about all of my components as having two interfaces, one which

is used to, you know, make my program work and the other which is actually used to change how it behaves if I need to. And so, there are a lot of ideas like that that have really impacted how I approach design or solve problems or just, you know, I have new tools in my mental toolkit to reach for whenever I notice I'm in a particular situation. And so, I do, you know, I don't spend all my time in the literature, but I do try to at least read the famous papers and, you know, the papers those reference or papers that reference those, you know, I do a lot of bibliography diving to find what I should read next. It's funny because I feel like so much of computer, not computer science, but professional software engineering has been this struggle to find ways to stop doing things over and

over again. But there's this tension between making things abstract so that we can reuse code and be portable and having things be well-suited for the platform we're running on. And it keeps going back and forth where, oh, this cross-platform library is very popular. Oh, but it doesn't work as well as developing native. So now everybody has a native team for every mobile device that writes a different app.

And then, oh, it goes back and now we're all using the same thing. I just feel like this has been this constant struggle between how do we reuse code, and, well, we should just give up because it ends up not working that well. And

that's not necessarily towards the thing we're talking about right now, which doesn't really have that problem. It's a very focused thing. But I do feel like that's been a struggle that hasn't been resolved and kind of goes back and forth. And I don't know if it's resolvable in the general sense. But it just seems like, why are we writing all these things over and over again? Why, when I start a new client project, am I rewriting some of the same code that I did at the previous client project? There's ownership issues, of course,

but it just feels like we should be building on the shoulders of the previous code and we don't get to do that that much. I have a good story about this. It's from Classpert. When people were turning in their final projects, one person turned in a final project that was pretty good, but there was one section of code that was really good, like really nice. And I asked them to be sure to check the licenses and to document that in their final reports.

And so it was documented in the final report, but in the code, it wasn't. And I was about to write a nice, long, complimentary paragraph about how this code was super clean, and it was good, and it looked like he had tested it very well. And then I thought about it for a little while, and I went and I looked and I noticed the license. And it turned out it was written by Philip. It was his circular buffer library.

So you're just saying I should take everything from Philip? I believe so. He's got this whole embedded virtual machine. Problem solved. Maybe, no, that may be the solution, partially, not to take everything from Philip, but to have a better... Please don't, that's a lot of pressure. Have a better awareness of what's already out there, or go looking for things, not in a stack overflow sort of search way, but at the start of a project, oh, is there something that's already been done that does what I want that is licensable? Instead of saying, you know, time to implement a state machine framework for these people. And, oh, a time to implement a logger for these people.

I mean, the chip I'm using now has 97 application notes for how to do bootloading. And each one has something just a little different, whether you're doing it from serial or Bluetooth, whether it's secure or very secure. And digging through this information has been horrible. And I knew what change I needed to make the first day, but I couldn't figure out how to prove it to myself. It took me like a week of reading their documents and trying out their stupid bootloaders before I got back to the original decision to just change this thing that the compiler said I shouldn't. Now, I might argue that that could be worth a week's worth of research time for a

bootloader, right? You want to make sure that thing is actually working correctly and that thing the compiler is telling you not to do isn't really a problem. Yeah, don't push on that. It's already been a week. They're going to want results soon. Yeah, there is that. Well, and I think this general idea... I've worked in the private sector. They want results. And I think this general idea goes back to a theme that has come up over and over again in our conversations, which is like, you know, maybe you can take two weeks to research things instead of starting immediately. Now, I know that no team in the world wants to hear that, and I've never convinced any

startups I've worked with ever that two weeks is an acceptable period of research and prototyping and design to figure out what you're actually going to do. But, you know, I think it is certainly worth it to figure out what's out there and what components can I reuse that exist. I understand the reading for an hour a day, but how do you choose whether you're reading academic software papers or working on a personal project with an Arduino? I let my interests drive what I'm spending time on more than anything, at least in the learning regard. You know, obviously I have constraints from the work I'm trying to get done, too. And sometimes that drives my reading too. If I need to figure something out for a particular project, you know, I will try to focus on things that are relevant to

that. So, you know, it's a little bit of a choose-your-own-adventure and a little bit of the needs drive the situation. Where do you look? For papers or just in general? Papers. Again, I do a lot of bibliography diving. So in any paper I come across that I like, I look for references that seem interesting. And then I'll usually go to the ACM or IEEE digital libraries to see if the papers are there. Sometimes I Google stuff. There was, I can't remember the name, but there's a list of famous software engineering papers or like

award-winning software engineering papers by year for a particular journal. So I'll look for collections like that in books. If there's papers mentioned in a book that seem interesting, I tend to pick those up. So, usually my reading is guided by other reading more than it is from me just going out and randomly trying to pick things. All right. Well, I'm not going to feel bad that I would rather do projects. I think that's okay. I don't always get as much software education as I want, but if it's a matter of staying motivated, having it do something is more pleasing to me. You just have people come tell us things about software.

Well, that is true. Yeah, you have the come-to-me model. That's a pretty good one. Tell me all the interesting things you know, and I'll pick and choose what I'm going to focus on. I read Philip's blog post, and I learned a ton. He's already filtered all that research into a blog post. There is that. That is true.

A lot of research went into that over the years. One of the things that your blog post says is that your opinion has changed. And how you describe it, it totally makes sense. But that sort of change is hard for us. I mean, we get stuck in, I still hear people argue about VI versus Emacs. Nobody cares. Right. Yeah. Why aren't you using VS code? You can have VI and Emacs in one beautiful package. What other changes have you identified for yourself?

I mean, that's hard to even identify, but have you identified other ones? Tying that point back into what Chris mentioned earlier, you know, I originally approached a lot of my design and development efforts from the idea of like, okay, I've written literally the same driver for six clients. And the reason I'm doing that is because of IP reasons and whatever implementation is out there doesn't meet the needs for whatever I have to do. And, you know,

and that kind of thing happened over and over again. You know, I can't tell you the number of TI gas gauge drivers I've written over the years, or it's just over and over again. You know, I can't tell you the number of TI gas gauge drivers I've written over the years or it's just over and over again. See, the thing is, this is resonating because for a while I was a very specialist for some reason in doing optical coherence tomography, imaging front-end software and driver software for medical startups. I don't know what any of that means. It doesn't matter, but it was complicated and I had to do UIs and graphics and talk to specific weird hardware. And for some reason, after doing it once, all these clients came out of the woodwork and said, oh, you know, I just kept getting referred to one to another. And I kept writing the same exact software package. I think I did it four

times. And I got, by the fourth time, fourth time i didn't i just it was autopilot i just wrote the same code i wasn't stealing there was no ip but i'd done it so many times that i was just autopilot okay well here's this here's this data acquisition class and here's this ui thing here's how i'm going to set this up and type type type type but yes i mean so does it matter i mean by the time you've done it six times, it's pretty easy, right? Pretty easy. But, you know, it's still like, it's not how I want to be spending my valuable working time, right?

It's not satisfying to rewrite the same thing over and over again, practically from memory. And, you know, and part of the, like, you start thinking about, okay, well, I could have a standard offering where I've already got all this stuff in place. And think of how fast it's going to be for my clients to actually make this work really quickly. I can get them set up. They can have all the drivers they need. If they need a new one, it's like, okay, it'll take me a couple days to bring it up and it's going to work fine. It's like all the machinery to support it's already there.

Great. It's going to be great. We're going to live in a perfect world. Everybody's going to be great. We're going to live in a perfect world. Everybody's going to be happy. And that just never, you know, the goals of reusability across products and portability across systems, it didn't play out in the way that I envisioned it would. You know, a lot of times clients didn't want that. They, regardless of the benefits that would come, they didn't want to pay a licensing cost or they didn't even want to just not own all of the IP, right? They'd rather have you re-implement it for them even in a worse way just to own the IP outright. So there are a lot of other concerns like that

that end up taking precedence over some of the engineering ideals. And I really had to change my views on what was the real value of portability and reusability, at least in the embedded system space. And for instance, a lot of my, like I don't focus on portability and reusability as a goal anymore. I'm more interested in how can I use these same things that would enable reusability

and would enable portability to just make it easier to change the product I'm working on in response to changes in requirements or changes in the hardware availability or changes in like you found out one implementation for whatever library you're using isn't suitable and you need to have another one now. So the same ideas have been sort of shifted to a different goal, but you know, I'm sort of heartbroken now with the fact that my reusability and portability dreams just didn't play out when I had a chance to actually make that happen. Have we switched to talking about the embedded virtual machine? We can. That was the project. Okay, that was what I was wondering. Portability and

reusability project, yep. Tell me about this embedded virtual machine. Well, nowadays I would describe it as my thesis on how to build software or how you could build software if you wanted to apply these techniques in practice. And so the idea there was I would have a set of design patterns built in that you could employ to make it easier to set up portable applications. And are, you know, easily changed applications, supplying like a standard set of driver interfaces and ways to keep your application code decoupled from specific hardware and just reusable machinery that comes up again and again, like a subsystem for managing different power states in your system or a logging implementation, things like that. And the goal was, again, I could license this to clients and bring up their system quickly, like it would be easily um supported you know we could make changes very very fast if they needed new hardware it would be easy to add new hardware or even outsource some

components of that and um other benefits come out like you know it was built with full static analysis tool support on had like three or four static analysis tools running everything was analyzed it worked with multiple compilers and i think my favorite feature was you know with the way it was designed you could supply an alternate what you might call platform definition so what is my what is my application running on in terms of OS and hardware? And you could have an application running on your development machine, like a simulator for whatever you're building, and most of the code could be common across your simulator application and the application running on the target. And all these goals are achievable, but like I mentioned,

nobody was interested in licensing it or even using it for a number of reasons that I think are totally understandable now, which is one, not owning the IP. And I think two, like, it really ended up being an expert system where, well, I guess there's three, but two, it ended up being an expert system where like you had to know C++ pretty well, and know the full framework to really take advantage of it. So for that reason, you end up with vendor lock-in where I'm the vendor maintaining the thing and your people can't necessarily maintain the lower layers just due to lack of knowledge or, you know, you don't want them to invest their time in that way. And so that was a pretty big eye-opener for me and sort of the impetus for

me changing some of my ideas. So you made a system that was everything Christopher wanted with reusability, but then found out you really do have to rewrite everything. Because that's what people want, yeah. But not just because... For real reasons, you know. For real reasons, know for real reasons that's yeah and i'm very sympathetic to the vendor lock-in because you know it's hard when part of the reason that i was working on such a project is because i don't want to be locked into a vendor

to then become that vendor like being accused of vendor locked it locked so many clients get i mean aren't they locked in anyway? Everybody's locked into some frame. You know, oh, I've got to have a file system. Nobody's going to write their own file system, so they go buy a file system. Nobody's going to buy their own UI, write their own new library.

I've seen it, but nobody should write their own UI library. You know, people go buy those. So is that a real concern, or is it more they don't want to be locked into things that they feel like are the core of their product? You know, I think it's a complicated answer, but that's a reasonable hypothesis. Yeah. I wasn't expecting, you know, expecting speculation, but it just seems weird to me. But, you know, there's many choices that are weird to me. Yes, agreed. I mean, yeah, the same clients would are happy to have their, you know, their software layer directly interact with their processor SDK, right?

And so now you're locked into a particular processor vendor. So, you know, I'm not saying from the client's perspective that it makes sense, but I'm saying that I do understand the accusation of vendor lock-in, even if, you know, regardless of any other things that might happen that might make that claim dubious. You know, I think it is true that you're locked in in a system like I built because going back to the full-fledged abstract interface conversation, you know, you can't use one part of the system without another part of the system because you're dependent on specific interfaces. So you now get the problem of like, okay, well, I can't use one part of the system without another part of the system because you're dependent on specific interfaces.

So you now get the problem of like, okay, well, I can't just pull out this particular thing I want to use because now I have to pull out this interface, which is dependent on these types that are in this place. And great, you have a modular framework for yourself. And I mean, this kind of thing, like inside of one company, certainly has benefits. But for me, where I'm developing software that for multiple people, it becomes more problematic if there's not a way to just take the pieces that they want and leave the rest, especially when that comes with the cost of complexity that may make you not want to take the full system on.

I think of, you know, Miro Samick's QP framework. I'm a huge fan of the QP framework. And I just can't tell you the number of times that I've heard people say, I like it. I just can't convince anyone on my team to use it. So we just rewrote everything ourselves, you know? So I do think there's a complexity argument to it. And like, I just can't take the pieces I need argument to it that impact everything. And what's so sad is most of the things we're talking about are infrastructure. They're not, they're not the special thing about anyone's product, right? Nobody, nobody on you know that was something i emphasized in my own marketing or attempts to convince people to use this was like you know the part of your project that makes you money is not the drivers or the file system or the logging library or how your code boots or you know supporting over the air updates that's not key to your business

model right you have a product that's going to do something unique that provides value to your customers. And like, that's what you should be focusing on. And all the rest should be secondary, right? It should be, you know, whatever I can get off the shelf or however I can invest the least amount of time in it. Over-the-air updates being a bad example with that particular point, but. Wow.

Yeah. I mean, TI does it for you, sort of. Yeah, the lock-in and the tools and the framework. I think about Grinning's TDD, Test Driven Development for Embedded Systems, is based on CPPU test. And I want to do more test-driven development, but a CPU test is horrible. Break breaking news. I mean, I'm sure once you have it working and set up, it's acceptable, but that step, it's just no fun. And I did it for one client and afterwards I was like, no, never again. I don't want to set this up. It's just, and so I, I get, I get where you're going with this and it is a complexity thing. It's a complexity and flexibility. It's very complex because it's very flexible.

But no actual product team often needs that level of flexibility. Right. The only reason you need that level of flexibility is because you're trying to support everything at once. Or you have dreams about platform development where you're going to make eight products on something and port them to different chips, which sometimes comes true. Oh, at LeapFrog, we had just like, had that part locked in. The only thing we had to do were the state machines and the behaviors. And we did not mess around with how it outputs audio. And the frog voices. You had to change the frog voices. Well, yes, but that was a matter of putting the voices into the memory.

I didn't have to worry about once they were in the memory. You had to get the frogs to talk, which is the harder part. Yes, that's true. But, you know, there's the little squeaky one. That was the best one. Philip, are you still there? Hi, I'm still here. If so, why?

When we started the show, you said that Embedded Artistry was a consumer service, no, a product services, but something about... Consulting and education. It was the education thing. You used to say consulting, and now you say consulting and education. When you were here last, you were starting a program to have a member section to your blog but that's extended so what's going on with the education world of embedded artistry so we do still run our membership program and that's doing quite well i don't have the current country count but i think we have representation from 40 more than 40 countries.

We just added Morocco yesterday, so that was pretty exciting. And continue to publish members-only posts that go into analyses of some of these papers that I read, but also technical approaches to different problems or what you know, what you might call like focus topics. Like I have a section on handling code reviews in an organization, not just code reviews, I should say code and design reviews, or, you know, capability maturity model for, you know, what a device and are connected, like IoT device, what kind of features do you really need to have on that device to take full advantage of your connected capabilities? So there are explorations like that, but we're also working on a number of courses.

The current one that should be finished, version one, by the time, well, not the time this comes out, but maybe a couple of weeks after this comes out is our Designing Embedded Software for Change course. We have a couple others like using C++ without the heap. I think some people in the embedded FM Slack group used our CMake course in the book club. We have a Maison course, we have a Make course. So all sorts of, you know, I would call them in-depth courses that are targeted for professionals. So, you know, you're a professional embedded software developer.

You need to solve some particular problems in your field. Like what are the, you know, what's the latest and greatest techniques that you can actually apply to your systems to achieve these goals of building a system that can be easily tested, of building at least pieces of your system that can be easily changed, of automating software quality enforcement, things like that. You have a pretty long list. Are they self-paced or cohorts? Right now, everything is self-paced, and it is a long list, but some of them are also in development. So there are, I would say about half the courses they are completed and then half are in a different states of development, but all of them, you know, I have hundreds of notes for each course that, you know, when I get the time, I organize them and release that content on a rolling basis.

I would like to do cohort based stuff, but I haven't figured out a model that works for me yet in that, um, you know, just managing the time and how to interact with people and what aspects to focus on. So that is something that I have in mind. But it's probably a couple of years out. The cohorts are fun. You get a lot more interaction. But they're also fairly exhausting. Right.

Are you having fun doing the classes? You must be because you're expanding them. Having a lot of fun, yeah. It's probably the most fun I've had in my career the past couple of years. Do you have to give up much in the way of consulting services to make time for the education? I do, especially now with two kids. But I still consult. I have three clients now, but it tends to be, you know, I'm doing more advising work or helping review code or, you know,

someone comes to me with a file and says, we don't like how this file is implemented. How could we make it better? And then, you know, coming up with ways to do that. So that tends to be more of the work I'm doing now. And it's good because a lot of it is related to the courses I'm building. So I just advised one client on, you know, what does it look like to have a reproducible build environment with a standard build interface? How do I take advantage of that with a CI system? And these are all topics that, you know, we're developing for our automated software quality enforcement course. So a lot of it is, you know, it helps me test out some of these concepts and, you know, see how they hold up in the field. It's not just

like I've made up a bunch of ideas in my head and put them on a course for people to learn and try themselves. So for that reason, I don't think I'll ever give up the consulting because it's good to have some way to test all these ideas out. The Designing Embedded Software for Change. What's that one going to be like? So that course goes through, it's a mix of theory and coding, I would say. So there's the theory of like, why does it matter that we design our systems for change? And what are some fundamental design principles that we can use when approaching systems that we want to design for change? And what are some fundamental design principles that we can use when approaching

systems that we want to design for change? And those are topics like loose coupling, separation of concerns, information hiding, and the open-close principle. And then, okay, great, we have this theory. You need to apply that to a system somehow. So there's some system-level strategies for approaching a system to make it changeable. And then each of those strategies I've mapped to specific coding techniques. Some of them like are the template method and callbacks that we talked about, you know, so showing examples in the code of how I can actually apply these techniques to support change in my system. And on top of that, we're trying to find real-world software.

I think that this is part of the problem in my own career and that I've seen in others is we read a lot of code and we copy a lot of code to do whatever we need to do. And so if the models that you're studying or you're basing your own designs off of or the code you're copying are poor, I mean, you're going to get poor results. And I'm saying poor, at least on the axis of changeability. So I'm trying to identify like what are real world projects that use these techniques that you can look at and see like how they actually support whatever, you know, aspect of change they need to support. Showing, and we're also going to, you know, I have some demonstrations I've put together showing,

again, like, okay, I've got this driver that was written for the Arduino SDK. How can I refactor, you know, the implementation just a little bit to make it so now that could be used in any framework I want instead of just on the Arduino? I'm back to where do you find the time? Little bit of progress every day. Adds up. That's all I can say.

What other design patterns do you think are important for embedded software engineers to know? So state machines have been mentioned. You know, I'm continually surprised, I think, based on how fundamental state machines are to my own work to find that it's not the case in a lot of the projects I've worked on and teams I've consulted with. I think state machines can clean up a lot of code if you can think about structuring your code in that way. Just moving to event-driven systems, which comes with state machines, but it's just so much easier to

think about and the since the topic has changed that's one of the things that's changed in my development is starting out with a very choose your own adventure approach to software development where there's lots of if then else's and functions and things and just kind of i wrote very bad code when i was sort of starting out to where i hope I am now, which is like, okay, what are the things that, what are the main actors? How do they need to communicate? What are the things that happen and how are those conveyed to each other? And it ends up in, you know, lots of event loops and state machines and things that I think are pretty readable. And how do you delete code and how do you delete shared states? Or add code. How do I extend functionality? State machines make that quite easy too. I tend to realize that the code I wrote initially was like 90% deletable if I could just replace it with a state machine or something.

Yes. Not more clever, but something more canonical. You have a panel. You've done one panel discussion with Memfault, I think it was in April? And you have another one coming up, right? Yep, that will be at the end of August, the 28th. What's the topic on the... The first one was scaling embedded systems, right?

Sorry, the 25th. Man, I'm not going good with all my memorization effects. The first one was on device metrics and how we can use metric collection in the field to help our debugging and I guess all sorts of prioritization efforts.

How do you figure out what features to develop next? Things like that, using metrics collected from the field. And then this upcoming panel, we're going to be discussing more generally, how do you debug devices once they've gone out into production? Cool.

And people can sign up for that. We'll have a link, I think. We'll have a link in the show notes. We'll have a collection of links in show notes. Collection of links. I have a bunch of tabs open, so I'll be copying those over before we close them all. I have one more design pattern that came to mind. Oh, sure.

Okay. Communicating through queues. Yes. I mean, it ties again into states and events, but just more generally sending data through queues instead of directly passing data between components is a, it's a great way

for decoupling things. And for, you know, you can isolate stuff to test you. It doesn't matter what's at the other end of the queue, as long as you have a standard definition of the data that they're both modules are using.

That's been something that's really something that's really helped a lot of my designs and it's easier to like instrument i mean you said you can put something on the other end but you can also instrument it transmit and receive everywhere so you know oh i i this happened and this guy didn't get it or which i mean you can sort of do that with calling functions directly but it's's very messy. And you don't have these central, you know, kind of everything funnels into a central receive. So it's much easier to say, oh, I received this message and just have a little debug printf that says, got this event. I totally agree with you about...

I totally agree, but... The queues, because I've seen that it does make things better. But as I'm on the Wikipedia list of software design patterns, there's no cues here. Well, I mean, there's no design pattern board. Well, that's the thing is I feel like both, I don't know all of these patterns as well as I want to. And many of the things that we do commonly aren't even on here. So, so design patterns are broken. Let's just give up. No, wait, that's not, that's not what I meant to take away from that. I don't know what I meant to take away with. Oh, I guess, Philip, could you fix that for us?

I will try. I have my own design pattern catalog. So I'm working on it. That would be kind of interesting. You mentioned design patterns in Embedded C, which is up here somewhere. Ah, Design Patterns for Embedded Systems in C by Bruce Powell Douglas. Maybe that's where I should be looking instead of Wikipedia. Yeah, we know that Wikipedia is always 100% factually correct,

and there are no political motivations or people trying to claim territory on Wikipedia. That never happens. Well, I mean, the software design patterns page seems fairly straightforward. You can go in there and add queues. I was going to say add queues, see what happens. I consider message queues and event queues to be a design pattern. But maybe, you know, like I could see message queues being under a message passing design pattern. Right. So certainly names are hard.

The loosely coupled aardvark or something. Yeah. Maybe that's what we need, more memorable names. Yeah. Yes. Yes, I think so. Speaking of aardvarks, you mentioned the aardvark tool. Oh, wow.

Nice. And I, like I said, I have a lot of tabs up. Could you explain it? I hadn't come across it before. So I discovered them through their USB debug adapter, which is excellent if anybody's working on a USB device and you really need to do analysis of the messages that are going over the bus. But this device is a USB to SPI, I squared C and GPIO, I guess, debug adapter. So they have a little GUI program that you can use to talk to devices and, you know, send raw SPI or I squared C commands and see what response comes back. But they also provide an API that you can use.

I think you also need lib USB, but so you can programmatically work with the adapter. And so I have a, you know, for my standard spy and I squared C interfaces, I have like a Aardvark driver and I have spy and I squared C interfaces that use that Aardvark driver to I have spy and I squared C interfaces that use that Aardvark driver to talk to the device. And yeah, I can control any spy or I squared C device, you know,

chips like lines can work because you can do spy plus GPIO. So it's a pretty powerful tool for one debugging those devices. If you know you're, it's not working in your system and you want to just talk to the raw device itself to make sure things are, like, whether you have a fundamental flaw in your understanding or whether there's something else going on. But also, yeah, for developing devices without having, like, a full board in-house. And so that's been a real beneficial practice in my career. Have you used the Bus Pirate? I have not. Should I? I don't know that what, how you describe it is largely how I would describe the bus pirate, but it's an order of magnitude less in cost price. Um, and so I think there must be

some qualitative difference. No, you're definitely kind of bashing at it, but you can make scripts. Yeah. I don't think that you need to necessarily use an Aardvark. I've done this same thing in the past using an FTDI chip, so I wouldn't recommend that necessarily. It's a little more painful than using an API that somebody else manages for you. But yeah, you can generally use any number of adapters for this. And I have, I think I have a couple of little, you know, China design boards that I've used from time to time. I don't, for example, I wouldn't travel to China with my Aardvark. I usually took

a little multi-protocol board with me that I could do JTAG, SWD, SPI, I2C, GPIO controls, all sorts of stuff. I mean, now for bringing up a new I2C or SPI device that I wanted to just talk to it, I would seriously consider a MicroPython board. I mean, it's just so easy to use. And you can really, I mean, you can use it both to do the communication and to monitor the lines. So you get both logic analyzing and communication. I don't know. I haven't tried it yet, but I've been thinking a lot about it. Well, Philip, we have gotten off whatever the topic was. Clearly, we are not in a rut because we went all over the road. So, do you have any thoughts you'd like to leave us with? Check out our website if you like this kind of conversation. I explore all of these ideas in great depth,

much deeper than you can get to in a live conversation without code. It's hard to conceptualize some of this stuff without code. So if you want to learn more about this stuff, check out our website, look at our course list, become a member, check out some of the great members-only content we have in the Field Atlas. And you can always contact me if you have any questions, and I'll point you to the right stuff, if I know. Our guest has been Philip Johnston of Embedded Artistry. There will be many links in the show notes, but if you just search for Embedded Artistry, you can get to his site. Thanks, Philip. It's good to talk to you again. Thanks for having me.

Talk soon. Thank you to Christopher for producing and co-hosting. Thank you to our Patreon listener Slack group for some ideas about how they have changed over their careers. And of course, thank you for listening. You can always contact us at show at embedded.fm or hit the contact link on Embedded.fm. And now a quote to leave you with from Maya Angelou. I wonder if I've used this one before. Anyway, from Maya Angelou, we delight in the beauty of the butterfly, but rarely admit the changes it has gone through

to achieve that beauty.