Embedded - 440: Condemned to Being Perfect

Episode Date: January 13, 2023

Chris and Elecia talk to Jeff Gable and Luca Ingianni of the Agile Embedded podcast, discussing the definition of Agile, agreeing about some things, and disagreeing about others. Agile Embedded can be found in your usual podcast locations or get it from the source: https://agileembeddedpodcast.com/ Jeff’s website is jeffgable.com and Luca’s is luca.engineer Transcript

Welcome to Embedded. I am Elysia White, here with Christopher White. And I am Jeff Gable, and I'm joined by my co-host of the Agile Embedded podcast, Luca Ingeani. So we're doing a crossover episode this week so we can get accustomed to each other. You are the Agile Embedded Podcast. How long have you been around and what do you talk about? We've been, it's been about two years, Luca. Sounds a little grudious. Yep. And I think we publish mostly bi-weekly. So we're about, uh, 43 or 44 episodes in. Um, and we essentially,

the thesis of our podcast is that you don't have to choose between development speed and quality and that agile techniques can apply to embedded. Uh, and we just talk about different ways that, uh, you can apply agile techniques to embedded development and, and use that to develop more effective products more quickly. What do you think, Luca? Is that a fair summary? I think that's a very fair summary.

Yes. All right. Well, I'm interested. I'm interested to be, to be convinced. Convinced of what? I have some experience with Agile,

but we can talk about that at some point. And you can tell me how the people that were doing the Agile where I was were doing it wrong, because I suspect they were doing it wrong. But I have a mixed relationship with Agile. And it's funny, we almost feel bad naming it that. I don't know what else to call it.

Um, we, we've talked in the past about, uh, you know, the, the big a agile has been so overloaded and overused and now means different things to different people. And, um, uh, nimble is, is the word that I've heard a few people say recently that I liked. Basically, if what you're doing makes you more nimble, then it's a good thing. If what you're doing locks you in and makes you less nimble, that's a bad thing. It's maybe a word that helps you get back to first principles. I like that. Well, not short. When you and Luca were talking about your Agile manifesto, there was some words that made a lot of sense to me, which was risk management through feedback loops, which is going to make an awful acronym. It's not really any worse than rolling on the floor laughing out loud but yeah

if i could tell a manager that instead of using the loaded agile word i think that would be an easier sell at least in some places yeah the the problem is if you deviate from this term agile then you need to explain yourself at length. Like, oh, we're talking about risk manager and this is not what we're talking about. What I'm interested in, I'm interested in, I don't know, software development methodologies or what have you. And you need to sort of rein them in and say, no, no, no, this is exactly what you should be caring about. I promise you that what I'm about to tell you is a helpful thing. But on the other hand, of course, if you stick to Agile, then you're inviting all kinds of people. You have to explain that to me.

You have to explain yourself either way. And you get people like me who are like, oh, no. Well, because you've had experience places. We do Agile Scrum, which means we have stand-up meetings, and that's about it. And we have retrospectives where everyone says, yeah, it was great. Exactly. We have a waterfall process with stand-ups. A waterfall process with stand-ups.

Well, yes, that's what it has boiled down to in the past for me. My first experience with Agile involved stand-up meetings that were two hours long. And so ever since I've had this little Pavlovian response. I'm so sorry. Oh God, no. Well, I've met somebody who told me that their team had trouble keeping the stand-ups short. And so they made a point of having the stand-ups on one leg well see that was a problem every time i had stand-ups uh we nobody stood up

yeah so everybody we have remote mostly so nobody you know everybody was happy in their house sitting in their comfy chairs yeah and and we've we've mentioned this on our podcast in the past like agile doesn't mean scrum like you don't have to do sprints if it doesn't make sense for your team you don't have to do stand-up if you are if you are communicating and constantly letting some letting the team know if you are blocked and need help then you don't that's the purpose that's the only purpose of a stand-up is to enforce that. So if you are satisfying that some other way... Not even enforce,

but facilitate. Encourage. Yeah, exactly. So don't do stand-ups if they don't make sense. I don't know. We do not advocate at all a particular methodology. We kind of advocate principles of do things that, you know, make the forward flow of work, like from the idea all the way to as getting value out of that, which usually means shipping it to a customer.

But if that's not appropriate, because you have, like for me, you know, a medical device. Those are what I work on. The release cycles are slower, but I try to do faster feedback loops within the development, like the engineering development side. And then I encourage, as much as I can, the business to do faster release cycles. But there is a limit on that. Like if you do an FDA submission, they're going to take a long time to get back to you. So there are ways around that. But the point is, is that just trying to encourage that fast flow of work to get value out and encourage the feedback loops, you know, to get back upstream as quickly as possible. And there are lots and lots of ways to skin those cats. Yeah, and I think that we're getting back to what Alicia said about agilist risk management. The point is to de-risk developing your product because you're working in this field of double uncertainty. Technical uncertainty, we don't know how to build it

or if it's even buildable. And product uncertainty, we don't know what we precisely should be building. And so we need to de-risk this double uncertainty by proving to us that we can build something and that what we have built carries value for somebody. And that is the entire point. And by the way,

the Agile Manifesto was written in 2001. It's not like Agile sprang into existence then. You know, I remember when I was studying in mechanical engineering in Germany, my professors would bang on about this thing they called Ingenieurmäßiges Vorgehen, the engineer's approach. And that was, at its heart, Agile. The idea was to come up with an idea, build a prototype, and then kind of look at it, and then go from there.

That makes a lot of sense to me. You know, the best times I've had have been working in small teams where either I was leading the team or a part of the team, but there were small teams, less than 10 people usually. And we never had an official methodology in a lot of the team, but there were small teams, less than 10 people usually. And we never had an official methodology in a lot of those places, but we kind of settled on that kind of iterative process, even at medical device companies. But I guess where I have some trouble is when the teams get larger and when you have multiple teams, I don't, I don't personally know how to do that, how to keep that spirit going or that effectiveness. Yeah, that's, of course, something very difficult.

And that's something that the Agile community at large is currently struggling with. And you get all sorts of proposals like, you know, say, for instance, the scaled Agile framework for our enterprise and less and, you know, all of these things. And they all try to give you some kind of a framework to scale up beyond a single team to teams of teams or even teams of teams of teams. Or in the case of SAFe, I think up to teams of teams of teams of teams. So like a couple of tens of thousands of engineers working on, I don't know, a car, let's say. And that is just really difficult and it will slow you down because at the heart of it, you need to have that easy flow of communication and that starts to break down. So the only way you really have is to remove dependencies. But, you know, there are just some dependencies that you will not be able to get rid of.

So I think, in principle, it will just have to slow down. If you are interested in aviation, there's a really interesting book, which is called, I think, Skunk Works. And it talks about exactly this, you know, the special projects branch of the aircraft manufacturer Lockheed. And the guy who founded Skunkworks, I can't remember his name. Kelly, Kelly Johnson. Yeah, Kelly Johnson. Yeah, exactly. He said, Skunkworks must never grow beyond 150 people, including like the cleaning lady and the secretary secretary and whoever because once it grows beyond that i cannot maintain the speed that i have and they did you know exceptional things the cycle

time in in the aviation time is like 10 years roughly usually and they were able to go from signing of a contract to first mission flight in 18 months for aircraft that are supposedly impossible, like the SR-71, which I think still holds the speed record. And he maintains that he was only able to do it because he had such a small team and he was just sort of very aggressively keeping the iteration small and focused. I mean, Skunk Works was kind of special in that they had insane amounts of money and intelligence concentration. It's like HP Labs was really good at doing things because they could just get whatever they wanted. And the Skunk Works book is fantastic. I totally agree with you there. I just don't know how to apply that to everywhere

because somebody has to write the grocery store application. Somebody has to write the software for the refrigerator. The principles don't change. Like, fine, you don't have a shed full of jet engines at your disposal. Why not? Yeah. Actually, yeah, why not? I had a

professor at university who had, like, two rocket engines sitting out back in his shed behind his house because, like, nobody wanted them. Exactly, like you do. But the principles all stay the same. Back to

your question of how to scale this up. I think Luca has more experience with it than I do. I certainly don't have an answer and I think if you just... Nobody does. A lot of people pretend that they do. Right, and you say, you know, if you just, nobody does, right. And you say, okay, we're going to do safe. I, I, I have never used safe myself. I really doubt that it is effective, but I don't know. Um, but I guess, uh, the, the overall principles are, you know,

look at your value stream. What's, what's all of the steps necessary from when someone has an idea to when it actually gets in front of a customer and starts producing value? And how do you shorten and automate that? That chain is going to happen over and over. Waterfall, the big fallacy of waterfall is that chain happens once. Right, right. Which is just not how life works. So if you know you're going to do that chain multiple times, smooth out that chain

and, and speed it up as much as possible. And that's it. Like if, and if your organization is 10,000 people, the, the solutions that you come up with to do that are going to be very, very different than, you know, with a team of less than 10 people, which is most of my experience too. Um, and right now I'm a solo developer. So I am usually at least as far as the software, a team of one. Um, so I get to, I get to speed that value chain up as much as I want. Um, and I don't answer to nobody, but I still work with, uh, you know, product development teams. Um, uh, so I'm, I'm having to work with them and, and try to speed our, our collective development cycle time up. Um, but the solutions you come up with to speed that up and then get information back

upstream are going to be very different depending on the size and the, uh, of your organization, the complexity of your product, kind of what hand you're dealt. Do you have a team of all stars? Do you have a team that you inherited? That's not great that you've got to improve over time? There's no one simple fix here. How is Agile that different from a sort of waterfall spin that has good milestones? I mean, I've never done a straight waterfall. It's funny you say that because I was about to mention that when I've done FDA products, you have to include your software methodology as part of the submission and all that stuff

and document it. And what I had done, this was many, many years ago before Agile had taken off, but, you know, I would take waterfall and I was like, well, that iteration, that's dumb. So I would write in there and change the diagram and make a little loop in there we we do this about eight million times right here and then i'd submit that waterfall with this little spin like waterfall spin and and you get milestones and and those are what i often think of as agile releases right i would characterize that as maybe an agile engineering development process within a waterfall organization. Um, which is sometimes the best you can do. I think we literally even have an episode on this. Like, you know, you're an engineering, you're, you're a product developer and you want to get faster, but you're stuck in this organization, sometimes the only thing you can do is, is do lots of spins in your organization. And, and you just have to, you have to kind of push that, um, that practice of trying something, getting feedback, and then going back and fixing

things. You have to kind of push that practice into the boundaries of the organization that you touch. So if you're working with, um, you know, kind of whoever's specifying the high level requirements, uh, maybe it's in the end marketing. Um, but it's some kind of, it's, it's essentially a business problem of what product are we going to develop? The faster you can finish your cycle and then show that to those people and hopefully, you know, show it to a user who talks to those people, kind of facilitate that larger loop. You know, I think that kind of achieves what you're talking about. In medical companies.

Make sense? Oh, yeah. And in medical companies or things like Skunk Works, the end user often has no idea what they want. Right. And will change their mind many times. As opposed to anyone else. Well, true, indeed. But sometimes consumer products,

you do get that feedback pretty quickly. And if you have the right sample set, it actually works out. Or you just ship the wrong thing. Or you ship the wrong thing. But especially with large contracts like Lockheed does, you have a contact that you show it to, and they're the customer. And they're not going to take it back to their team until it's good enough. And then once they take it back to their team, they ask for a bunch of changes.

How do you deal with agile in your company being good, but your customers not being particularly agile? And so you're giving them all of the stories and asking for the feedback. And they're like, it looks good. It looks good. It looks good. Oh, now I've shown it to my boss and he hates it. I, uh, Luca, maybe you can speak to this as well. I, um, uh, that's just more of an interpersonal thing. Like that's, that's an example of, you know, agile doesn't save you. That's just a basic customer interaction thing. Um, and again, I think, uh, um, uh, I, uh, an episode we're going to record that I've had a conversation with Luca, your, uh, one of your clients, uh, who's a hardware developer company. And we, uh, we, um, uh, actually wanted to record with them because

they're, they have hardware expertise and they have very rapid hardware iterations. And they were like, they came to Luca for help. But our software is, we're very slow on our software iterations. We're like, what? That's backwards. It's backwards. It's so funny. They were saying hardware is the hard part.

Yeah, it's so easy for us to iterate in hardware every two weeks, but it's so difficult for us to iterate in software quickly enough. And it's like, what? Never had that problem. I know. They have been in existence 10 years. And at this point, they've now built up the reputation where they just specify how they work and don't take contracts with people who can't work that way. But I don't have a good answer for that either, Chris. I think if you, again, if we go back to what Alicia pointed out that we had said before, if you couch it as risk management, which is essentially what it is, don't you want to manage this risk as much as possible? Are we sure we're building the right thing? Don't you want to show it to your boss a little bit earlier? Uh, and, and you're right that some people, if, if people don't know what they don't know, if you're, you know, you know, Steve jobs, Apple vision, like I'm going to,

like, I know what the customer wants before they do, then maybe it's you're satisfying Steve jobs instead of the end user. But either way, whoever makes the ultimate decision, um, get there, show it to them as soon as possible, but it does have to have a certain level of polish maybe. Um, or maybe you can come up with prototypes before you've built any product that, that just gives some idea of, of what it's going to be and getting feedback on that the point isn't what uh what exactly you're building the point is to gather feedback and end up building the right thing um and yeah so just like jeff said this is really not about agile being wrong like you can run into the same kind of situation

in a waterfall process. In fact, you will, because you're setting yourself up to only show it to them after you've invested all of this effort into building it. And then you're going to get it wrong anyway, let's be honest.

Not because you're incompetent, but just because nobody knows the future that well and no customer knows what they want that well and so in the end yeah this is and by the way we keep making that same mistake that i also always make of speaking of the customer in the singular yeah for any most trivial products you will have a bunch of people who very reasonably have demands on that product. Shall we call them stakeholders? If you insist. I don't like the term, but I don't have a problem.

So I'd actually turn the question back to the two of you because both of you have a lot of experience developing products for customers and in teams of different sizes and purposes. So how have you tended to manage those interactions or to kind of stumble through them and hope they don't happen too often? Do you want to go? Backup plans. Backup plan A through F, sometimes G.

Oh, risk management. That's interesting. Well, but risk management in a different way. Say I am presenting a tool to go on a drone, which I've done, and the tool may work in multiple different ways. And so when we get to the point where there's an official milestone, where we're meeting, where we're showing things off, I have already thought about what are the top three things they're going to ask about and how can I mitigate those risks. And so it's actually the opposite of agile. I am not asking them the questions. I am trying to anticipate what they want so that when they say, oh, but we wanted the drone to fly in loop to loops, I can say, oh yeah, I have a drone to fly in loop-de-loops, I can say, oh, yeah, I have a loop-de-loop sort of process.

And that means that I've wasted some time because I may have implemented things they don't need yet. But the idea is to try to anticipate what they need before they've told me, which is always dicey. I mean, what about you, Chris? You're so much better at that than me. My answer is anger and frustration. Yeah, I'm trying to think of examples. So there's been a couple of, I've worked with several medical device companies, various levels of rigor. And at the last, the last time I did a medical device, it was a very founder, founder oriented company. The founder, the founder was very in charge of things, but he really didn't know what he wanted. So we spent years in the iterative cycle, delivering prototypes, him testing them in the lab, delivering prototypes, him testing in the

lab. And that just became, you know, I was worn out to the point where anticipating what he wanted was impossible. It was just, okay, he's not going to like it next week for some reason, so we'll just keep writing the code and changing the hardware. So that took years and years, and it was very, very difficult. So in situations like that, I don't think there's a good, there's not much you can do. That was, I mean, Agile, Waterfall, Special Magic. Nothing. There was nothing going to work on that. That was acute founderitis and there was no help for that.

But other places, you know, I think I've tried to do what you do and anticipate what people need, especially when I'm more in charge of the software design. But in a lot of teams, I wasn't. And I was just, you know, a member of the team who owned a particular portion of the code. Even if I was senior, it was like, well, you, you know, you're doing the DSP stuff or you're doing the UI stuff. And that's more limited. You don't have as much opportunity to kind of, oh, the customer might do this, so we have this other version. We could do it in small ways, but yeah, it's always frustrating for me. And I think I get into a situation, and this is just a personal problem, where I second-guess the customer. In most cases, the customer, I'm referring to the customer here, it's somebody very senior at the company.

Sure. Because I was usually very senior in the software team or running the firmware software team. And so I was sort of reporting to the CTO or the CEO and here is, you know, here's the software, here's the firmware. And they would, we'd go back and forth. So the times it's been most frustrating for me is when they don't know what they want, even though they're very opinionated about what they want. And so you, you know, I would deliver something that my team did that I thought was fantastic. And they'd be like, Oh yeah, no. And so, and then trying to explain to them why they were wrong, because I thought they were wrong. And I thought I knew what they wanted.

Anyway, I, to sum up, I have not had great success with managing customers. And at places like Fitbit, I was so insulated from actual customers that that never became a thing, really. Right. Hmm. I'm trying to, I'm just back over over the interactions i've had i maybe i maybe since i've gone solo i basically just see those customers as having red flags and don't work for them yes that is the correct answer but it's too late when you're working full-time in a big company yeah i that's that's part of that's why i went solo is that i have complete control over who i work with and i do a lot of pre-qualification and i could just say

no i don't know that this is going to work out so well um and just don't take that particular project yeah but we're touching on something really challenging here, which is the entire subject of culture, isn't it? You know, if you are working inside a company and you've got bosses like this who don't really want to listen to you, but at the same time, they want to tell you what to do. Quite clearly, that is not going to work. That's the end of the story. So there's just no magic bullet. You can't have, you know, a stand-up here, a stand-up and that makes it right or something.

This is just something you, as a collective organization, you need to work on this if this is something you care about, or you can just sort of go on floundering about, which is, you know what what tends to happen because culture change is just hard and it it it's very personal it makes people very nervous uh it makes people very defensive because maybe if you if if one of those executives were to take a look at themselves and say, well, I've not been very helpful, have I? That stings.

There's just no way around this. And so long story short, this is the holy grail. You know, call it agile, call it DevOps, call it what you will. an organization where you can have open discussions and where those discussions actually have a meaningful positive effect, that's just really difficult to achieve. And the same goes for customer interactions. You can have awesome customers and you can have quite terrible customers. Just like Jeff, one of the things I love about being a solo consultant is that I can have no other roles. I just don't work with others. End of story.

Amen. Yeah. And I should note that Elisa and I are both consultants. Yeah. Of course. For reasons, for similar reasons. To some extent, I worry that this is sadly not as applicable to most folks who are in companies because they don't get these options. And I've been there and I mean, I used to rail against design by committee, which sometimes Agile encourages. I want someone to have a great idea and to stick to their guns. The Steve Jobs method. And I know that that's not always possible.

It's not always good. If the person doesn't have a good idea, it's just not good. But there is a lot of downside to design by committee, especially the very long arguments from the two people who are never going to agree and I still have to sit here and listen to them? I think the position that sums up the situation that I think we're talking about that I find most difficult is all the responsibility with none of the authority. Of course.

Yes, that is why you're going to consulting. You were only allowed to get things wrong and you were never giving credit for doing anything right and so that that's that's the very difficult position and like you said luca it's a cultural problem and people i think a lot of the problem i have with soft software methodologies and development methodologies writ large is a lot of times people do try to sell them as this will fix your company do this as a company and that's where i like like, you're not going to fix it. Yeah, completely agreed. Just like you say, none of this will fix your company.

The hard work is in human interactions. It's so fascinating. I frequently talk to engineering managers, engineering leaders of whatever description, and they all tell me, you know what? I used to start out thinking that engineering was about, I don't know, differential equations or C++ or what have you. And as it turns out, it's really about

engineers having conversations. And the differential equations and the C++ code are just side effects of those conversations. If you don't have the right kind of conversations, you will not have the right kinds of side effects. That is it. Yeah. I'm not, it's yeah. In these,

in these cases, I, yeah, there's no magic bill. There's no, um, there's no advice we can give except, um,

you know, recognize that the, this, this kind of principle of get to value and get feedback, apply that within your own sphere of responsibility and try to slowly, gradually expand that outward. And that's really all you can do. And I, and I bet Chris, in the story you told about your, the customer with, or when you were on the team with founderitis and every other week you were presenting something new to the founder that they would inevitably not like, I bet you got really fast

at, you architected your stuff so that you're not spending six months building something that they can inevitably reject. Right. I bet you sped it up and minimized it so that you were getting to that moment of inevitable negative feedback as it turned out. But that feedback, I bet you were getting to that point as fast as possible. yeah no the development cycle was very quick very rapid yeah congratulations very adjunct view but but to do that the an important point to do that we existed in prototype and this was a medical device which makes this a different situation we existed in prototype mode and once we kind of shifted toward okay now we're going to start making the product i threw all that out and started yes and that was that was an important step because all

that stuff was developed with a level of care and quality that i would not want to exist in the world and that happens too where you're iterating so fast and you don't realize this is a prototype this is a prototype do not ship a prototype. Do not ship this. Which matters for some products differently. Well, this is contrary at least to the Agile scripture, which says that you should always create production quality artifacts. Yeah, yeah. But I can't do that with a medical device.

The FDA would come in with their guns and tell me to take a walk. You can still do it. Okay, wait a minute. I couldn't do it in that situation, given the level of change and requirements change. I could not document that. There was no way to do any traceability or document that in that situation. I'm not saying you can't do it with a carefully set up system. But at that point, I didn't even have a document control system.

There was just nothing. That wasn't going to happen. I want to go back to Luca's, you should be writing production value code as part of the Agile. Is that part of the manifesto? I think it is, yes. I've read the manifesto, but... Part of me says yes,

because you want clean, good code. And part of me says no, because I do make a lot of simulators and prototypes that are just to show off one particular piece. Looks like mock-ups or works like mock-ups, neither of which can be shipped. But if I can just make them go together, somebody will buy that product. I'm surprised. Yeah, but that's not the product. That's, as you said, it's a mock and that's a valuable thing. Don't get me wrong.

It's just not the actual product, is it? No. So, but it isn't production code either. It's something I'm doing for faster feedback. So I feel like I'm breaking one rule. If you're building a clay model, if you're building a clay model for wind tunnel testing, nobody will think that this is going to be whittled down into the actual car that you end up selling. Of course not. It has a clear purpose.

It has clear value. And so within the confines of clay models for wind tunnel testing, it should be a good model. But it is not the car, is it? Moving that to software, there is a place for not writing production code. There is a place for making mock-ups and models, simulations and quick and dirty tests. Is it a statement of quality, it a statement of quality luca or production part yeah yeah i i also got hung up on that same statement so explain yourself sir yeah well i i don't have to explain myself because it's not my words

manifestos no nonetheless though uh the point is if you're working on your product then it should be as you know as as the agilists say it should be potentially shippable you should be in principle able to ship this whether you know whether you end up daring to do so or not or whether the fda forbids you, is beside the point. The point is you don't build sloppy products. You're welcome to build sloppy prototypes and then throw them away.

But you're not allowed to build sloppy products. I feel like this is one of those pieces where Agile is for the software world and not for the embedded world. How so? I love that argument because everybody comes up with it. I've not yet been convinced that there has been a true instance of this.

Like if you are building an actual product, not prototype. A prototype, what is the value of a prototype? The value of a prototype is in answering a specific question. And, you know, within those confines, it should be well-crafted. It should give good, clear answers to the question you were asking. And then you throw it away because clearly it is not your product and you will not ever ship it or anything like that or implant it into patients. So, you know, even if we're building hardware or things that smell a bit like hardware,

like simulations, for instance, you know, this is the point why mechanical engineers or electronics engineers do simulations because actual physical prototypes are just too complicated and too expensive compared to software. Software prototypes are cheap and instantaneous, or free and instantaneous, in fact. Whereas if you're a civil engineer and you want to build a bridge, you know, a prototype, and you would love to have an actual increment of your real bridge standing, you know, in your real valley, but that's kind of awkward and expensive. So you make do with some simplified version of that. Software has the awesome property

of enabling you to build real-life bridges and having real-life trucks drive over it, metaphorically, 10 times a day. But you should be as close to that as you can, and you should still have a very clear distinction of what is my actual product that I am working on and that I will eventually offer to my customer. And what is just a prototype

that has the value of answering specific questions that I have, a technical question that I have. Should I build it this way? Can I build it this way? That's fine. You answer the question, you throw it away, and then you work on your product and you do it properly, whatever that means in your specific context. Is your prototype in this case, a branch of your firmware or is it a separate repo? I know that that's a technical detail. I don't care as long as you throw it away afterwards. Yeah, see, I have

plenty of times where my prototype becomes part of my product and I don't throw it away. I throw away the six other pieces of code I wrote. The surrounding driving code. The

cruft that I put in because I wasn't sure which direction we were going. And so as I move to more production, it's a matter of getting rid of that code. And so I don't write production quality because I'm writing extra stuff I don't need, that I don't want to be in production because it's a maintenance nightmare if we're not using those features. Well, yeah, that's an interesting point because there's, with embedded software, there's often a lot of debug tools, right, that aren't part of the product. Debug printfs, that serial console, yeah. And code paths.

Is that production code or not? Assertions that you take out, things like that. So that's, yeah, I don't know how to answer that question. Oh, that's a fun question because um as far as i'm concerned logging is an aspect of production uh not if your code is so if it's not not if it's if it's not you know if if it's you know print apps during development and during debugging and then you will throw them out later then then no then it's not part of your product and do whatever you want but if it's like logging then no, then it's not part of your product and do whatever you want. But if it's like logging, then, you know,

then it's just a first class component of your product, just like documentation, for instance, just like anything else. I know I'm really strict about this. Okay. So I've worked on children's toys, educational toys for LeapFrog, a company which sadly no longer exists. But one of the things that we had was debug printfs, like everybody. And we had logs, and we could turn them off in the final product, but we didn't usually.

Because there were times where if something went systematically wrong, we could get to those logs. But if I did the same thing in a medical device and those logs included private information, that's now a security risk. Although I need it for the certification process, I want to be able to turn those off. I want to be able to clear those because they are a security risk. So is that production code or is that prototype code? It is something I need to leave

in the final code, but I need to make sure they don't get... That's production code in the sense that you need it to get to a releasable product, isn't it? You're releasing it, therefore, I mean, it's in the system, so yeah. Okay, but it's not creating value. It is creating value because it makes the FDA happy.

And if the FDA is not happy, then your customers will never get your device, and that will make them unhappy. I feel like the distinction here maybe isn't useful. I mean, I think everyone knows like an Arduino prototype is a prototype. Yeah. And, you know, and then you throw that code base away and you write it on top of a real artos or something else um but but to me like i i would say like whatever code you're writing writing it write it with quality that is commensurate with how it's going to be used and we just all know the anti-pattern of someone threw together something

slapdash and then it works and then that stays in the final product uh and it's it's technical debt um and technical debt can be okay but you you, you know, if it's, if it was thrown together in the beginning without regard for error edge cases, or what if, what if this particular thing happens and that's going to go wrong and you're not handling it, you know, that can't make it into production, at least in any safety critical application. Um, but it sounds like this code that you were writing that, that stored logs and you use that during maybe the V and V process and then cleared them as the device went out the door. Like the last step is you, yep, those logs look good. Now the step is in manufacturing is clear them and disable that functionality. And then the device goes out the door. I don't know that it's really useful to like debate whether that's prototype. To me, that's production code. Because it doesn't interact with the patient. Maybe you don't have a line in your traceability matrix that where that could cause patient harm, because that's mitigated by the fact that you're turning it off before it goes out the door. So I don't know that it's really

useful to get hung up on that debate. Does that make sense? It was an example of other code that I can't tell the difference between prototype and production all the time. I don't think any of us are disagreeing that you should write good code. Oh, no, I think. And that, you know, when you write code at a company for a product, whether it's prototype or not, it probably should follow whatever software development process you use and be reviewed and have a design maybe and, you know, some documentation and stuff like that. I think where I got hung up with Agile, and maybe this is a misinterpretation, is, you know, this is the sprint model, which is maybe not Agile agile but it's conflated with it is oh you

should have a minimum viable viable product shipping every every sprint and when embedded it's like fantastic you have a spy driver for a light and a cli where you can type hello you know so after after two weeks that's what you get so um that always struck me as weird because like there's nothing my minimum viable you're going to have until six months from now. Um, but that's, that's just, I think, uh, and I think that's my misinterpretation of that. That's what it feels like. I'm sorry, you wanted to say something, um, Jeff?

I was just going to say like that. Yes. I mean, like there, there is going to be a certain number, a certain amount of time before you get something that's minimally viable. Um, and to me, that's just the example of like, you can't fix, you know, scope and time and what's the other leg. I can't remember, um, content or well, no scope time and, and team size maybe. Um, like if you fix all three legs of that triad, you're going to fail. So kind of the classic agile way is get to shippable, like define that minimum scope, get to that minimum scope as soon as possible, and then improve from there until you're out of time or budget. And a lot of teams, that's explicitly how they work, and that's fine. And then once you get to that minimum scope then try try not to do

a lot of work that doesn't get folded into something shippable right that's that's by kind of by definition your cycle time that you want to shorten like do a little bit more and then incorporate into that shippable product and a little bit and you want to get that to be as little as possible that you're doing without it actually being shippable. Does that make sense? It makes a lot of sense. Yeah. I just want to also speak about MVPs for a little bit, just because it's such an overused word and people are often not as clear on the definition as would be helpful for the

conversation, I think. So what is the point of an MVP? The point of it... So the thing that you create after, let's call it a two-week iteration, is not an MVP. An MVP is sort of a particular milestone, if you want to, which is the minimal set of functionality that you feel confident to show to your customer

in order to get feedback. It's not even, it's far from complete. It would be terrible if it were complete. It must be as incomplete as you can make it so that you can optimize for fast feedback. So in that sense, it's sort of similar to a prototype. But I like to say that a prototype answers a technical question. Can we build it this way? And an MVP answers a value question, can we build it this way? And an MVP answers a value

question, should we build it this way? Does anyone care? Can we get anyone to buy it? And so it can also, by the way, be very, very different from your final product. I once talked to a founder of some kind of a job market website, you know,

some typical two-sided market where people could, I think they could post jobs and then people could apply for those jobs, that kind of thing. And what they did was they created this very simple website without any business logic. I think they had like a contact form and it dumped it straight into a CSV file or something. And then the quote-unquote business logic was him and his co-founder sitting down and manually sifting through those CSVs and saying, okay, this is a match,

this is a match, and driving the supposed business logic. And the whole value of this was that they didn't actually get burdened by building the business logic and the whole value of this was that they didn't actually get burdened by building the business logic they could slap again slap together this website you know between the second and the third beer um and then see if they got anyone to care could they even get traction in the market and of course once they discovered yes yes, they got traction, then yes, they built the business logic.

But only then, only after they had answered the question, does anyone even care? And that's the point of an MVP. Like the saying of the Lean Startup book, if you're not embarrassed by your MVP, you waited too long. So make it as simple as you can. know don't confuse it with your final product and don't confuse it with a prototype because they answer different questions so i it's funny like we we have argued over about this i'm going to argue with you and i think i have a feeling elise and chris have the same uh uh concerns that i do over that statement. So when you're working on embedded, you're like dead crickets.

So when you have an embedded product and you have distribution networks and you have something, something has to be hanging like wrapped in shrink trap package, hanging on the shelf in a store. It's January. It's CES. Yeah, exactly.

If it's not in the stores by october you can't ship it for christmas for christmas so so there is like that is different from a sas application where to the user like it looks the same and you're i think they call it fred flintstoning in the background where you're like manually walking instead of the wheels actually being powered. But to the user, it looks like that's the experience they have. They input their data and they get their answer and they're happy with it. And they don't care that you did it manually behind the scenes. There is a fundamental difference with embedded products that are actually hanging on the shelf. And to me, I would just say, I would define MVP as the minimum set of

functionality that you're okay with hanging on the shelf for the user to take off the shelf. And anything short of that, I would care, like if you're showing it to users, getting feedback and going back into your development process, I personally would call that a prototype. Don't want to get into a semantic argument about it but it um i i would call that a prototype and it's the mvp is what actually hangs on the shelf uh and you just if you can de-risk your program by building updates updateability into that if you know that's it's a big step because you know for a child's toy you don't want to say well you have to now we have to make it an iot device just to get software updates to it you got to make your best yes that's that's adding a whole other level of risk that that maybe wasn't appropriate so

the number of the number of times yeah so of course your your approach your approach will will depend on the actual product that you're building maybe maybe your first iteration of a physical product that you know where at some point a truck will come and take it off your hands and then it's gone. So in order to iterate more quickly, maybe your first iteration will have a stronger processor and you'll do a lot of, I don't know, filtering, post-processing, you know, in software. And then later on, you'll do cost-saving measures and, you know, put some of the filtering in hardware and go with a cheaper processor, what have you. So your specific approach will depend on your specific product and the environment. Yes, of course.

And some things that work in the SaaS space don't work for embedded. Many things, perhaps, even. That's fine. But the point is the approach, not implementation details. And I really like that specific example

of, you know, the first few products that you ship out the door that actually are used by real customers, you don't care as much about bomb costs. Right. That's maybe an example of something tangible that you can latch onto. You're like, yeah, I won't care about bomb costs for the first 200 units because I want to see if they actually sell.

And if they fly off the shelves, then I can, you know, yes, I'm losing money on those, but I can do then a bomb cost reduction effort. And then the next thousand that hit the shelves are cheaper. And then beyond that, they start actually making a profit but i don't even i don't want to spend all that extra time optimizing my bomb cost if i don't even know if they're going to sell yeah and that works great for a lot of things medical devices low volume high high ticket price things and i've done that for other stuff like fitbits like well maybe maybe i mean they didn't they didn't start out as a big hit to start with. I mean, we always had users. Yeah, yeah.

Even before things got shipped. a feature that has to go into the production units, but doesn't have to go into the user test units. And yet that's such a huge feature. Especially since in later days, most products don't ship with firmware that works. Right. It's just the over-the-air update. To buy the poor firmware people some more time, the hardware...

To buy it six months between... more time. To buy six months. The hardware in Target before we're done. It's the zero-day updates. Yeah. Yeah. I hate those. And I would say something you just said, like the internal testing development modules don't have to have remote over-the-air updates, but the production modules do. And that's where, if I were coming in and looking at that team, I would push them really hard to get that OTA functionality done as early as possible. And that is what you are using to update your

internal testing units. Cause then that process becomes more bulletproof and you start to, you know, it's all about risk management. Like you don't want to do the very first time you do an OTA update is when you've got 10,000 units out in the field. I totally agree with you, but that's anti-agile. Why? Because over-the-air updates are not something you're getting feedback on from the user. It's not something that is part of the user stories. It is part of the engineering team's ease of use and updatability. It's still an end-user feature.

It is kind of, but... It's not one they like. But so many times I need to test the functionality of the device to figure out if it is market viable. And adding over-the-air update before figuring out that it is market viable is not good. Sure. That's a lot of work to put up front. Even though I totally agree with Jeff that if I go into a place, that's one of the things to do is because that's important and you should do it.

It'll make your lives easier but then you still have to have back-end software that may take six months to update that and it's just over the air update is one of those features that i have trouble with with agile no i i i don't agree actually because like just just like jeff and just like uh the two of you i would totally insist on having that sort of thing early on, just like logging, for instance. One of the first things you incorporate is a logging framework, so you can always tell what the device is doing, so you're closing a feedback loop.

And so maybe don't make the mistake of saying user value is only created if something is actually visible to the user and perhaps flashy or something you know a working i don't know data backup is really important even if your user never sees it and hopefully never needs to use it but you know clearly this provides value and the same goes for ota yeah i think i can fix all of this for all of you what if we made the the software update have a really cool animation with fireworks and little narwhals swimming across? And so the user does like it. Oh, yes, please.

And so they can have feedback on the animation, and then it fits into the framework. I mean, there's always exceptions to this kind of stuff. Of course. This is why I get nervous about software methodologies, because a lot of people who are very rigid about it and you talk to them and it's like, I can't do that. I have to make this exception here for this or this, this, but I can do most of this. And my thing for software methodologies

when I talk to teams is like, please have one. Yeah, it's like a coding guidelines. Please have one. I don't care what it is. Please have one because having one is so much millions of times better than not having one, even if it's a terrible guidelines. Please have one. I don't care what it is. Please have one because having one is so much millions of times better than not having one

even if it's a terrible methodology. And then improve it over time. Yes. If something about it doesn't work, fix it.

Exactly. Be agile in an agile way. No, but he's right. Adapt, adapt it to your organization. That's what I've done many times with stuff. When I've run

small organizations, it's like, well, I'm going to take a piece from here and here, and this seems to work for us. But, you know, I don't have a textbook. I've got it on the table saying, chapter four, we didn't do this right, so we've got to start over. Yeah, and I think this is so important because people feel like they must get it right the first time. No. You're going to get it wrong in some aspects the first time around. So, and that's, I think one of the beauties of working in an agile way, you're not condemned

to being perfect or trying to be perfect. You know, you can figure stuff out as you go and find your own way. And that's going to be good for you. And what the best way for you is now is going to be different from what it's going to be like in five years. And that's perfectly okay. I want to change subjects for a bit. As we were preparing to chat, unit testing came up, which I've always strongly associated with agile development. But you think it has some pitfalls and some cons?

Oh, please tell me. I don't have to do it. Tell me. So just a little background for the listeners. So I think this was someone who maybe asked you a question, like a listener of yours. One of our embedded listeners, Maddy C, asked about what are the cons of unit testing and CI and CD and short feedback cycles, anything. And I love the wording of Maddy C's question. Most agile TDD for embedded discussions I've read, heard, have been a part of focus on unit testing. Good. CI, CD, pipeline tools. Also good focus on unit testing. Good. CICD pipeline tools also good.

And also short feedback cycles. Very good. So basically, yes, just all of those things are wonderful and do them, but what are the cons? Anything that shiny is hiding something under the surface. Um, I love that phrasing. Um, so, so focus on each one of those things. Maybe I'll get to unit testing first. Cause I think that's the most subtle one. Um, CIDC pipelines, I would say are an unadulterated good. Um, personally, I think the only downside to those is that yes, you have to invest a little bit of time if you've never done it before to learn how to set it up, um, suck it up and do it. And like I, you, you must have,

I think you should always have a CI pipeline, a build. I would maybe call it a build pipeline. Cause as we've talked with Jonathan Hall recently on our podcast, CI and CD have definitions that, that you may or may not be following. Oh, right.

Continuous integration is that practice of merging back into the trunk as quickly as possible. Trunk-based development is now kind of a more modern way to describe that, whatever. But the automatic build pipeline that runs automatically as many steps as you have created for it, at a minimum, it's building all the artifacts,

hopefully also running automated tests. That is an unadulterated good. So do that. Um, short feedback cycles is just kind of a, a global principle that yes, I say is an unadulterated good. How you implement that at your organization is what we've been arguing about for the past 45 minutes. Um, unit testing. Um, I would say it is good, but it is very hard to do well. And you can very easily unit test your way into a corner where you have now this brittle suite of unit tests that is slowing you down. Oh, yes. And I want to be very careful where I'm going to say, well, if it slows you down, then you're just doing it wrong.

Like that kind of no true Scotsman garbage. Um, it's, it's hard to do well. And there are, there are, uh, parts of software I've written that I don't bother unit testing. Luke is a big advocate for this. Like, he's like, if you don't, I would say basically, um, things that are more risky where you can mitigate that risk with unit tests, please do so. And I think that test-driven development is a skill that is very hard to do well, but provides a lot of value once you get good at it. But there is a learning curve there.

And on the early part of that learning curve, you can actually decrease your development speed. And it's just difficult to get right. But it does have a lot of great benefits. It enforces an architecture that is easy to test. I would say it is bad during the prototype stage. When you don't know what you're building and you are very rapidly just trying to answer those prototype questions, don't bother with a lot of unit tests because that's not where you're trying to solve right now. Unit tests are a way to make sure that your module is, um, correct and handles

a lot of corner cases. Well, those are the kinds of modules for which it works very well. Uh, if you're just trying to throw something together to answer a question of whether or not this will, someone will buy this. I personally would advocate against advocate against um doing a lot of unit testing and certainly not doing tdd yeah and and similarly also if you've got an old code base which is you know as old code bases tend to be uh not very well covered in tests then Then as you start unit testing and maybe even TDD, some people think that they must now cover everything in tests. I say don't. Because what's the point?

What are we trying to do with testing? We're trying to increase our trust in the product. We're trying to sort of prove to ourselves that there are no issues with it. Now, with your old code base, you already know. You know, it's been running out in production for 10 years, for better or worse. You know what to expect.

What's the point of testing something when you already know the answer? So don't bother. But when you're going to go change something in an old code base, maybe add unit testing then. Then you should cover in tests. Yeah, that would be a very good point. I would say, and just from my experience doing tests and how test suites tend to,

I'm going to use my favorite word, accrete over time. At Fitbit and Cisco, at Cisco at one point, the example I love to give is there was a static testing suite that ran any time you tried to make a commit and it would take 24 to 48 hours. By which time? By which time the trunk had moved on and you were doomed and had to start over.

Yeah, been there. My feeling toward unit tests is you need to rate the code that you're considering writing a test for on a few factors. How often is this likely to change? And how critical is it to your core functionality? And, you know, just in general, how much risk is this?

And I think you've mentioned that. Yeah, we're going back to trust, aren't we? But I'm not, you know, I'm not going to write a unit test for my spy driver. I'm just not going to do it. I'm going to get it to work. And then that thing is going to stay there for a thousand years and no one's going to touch it. I mean, just historically looking at the kinds of code that gets touched a lot over the course of a product development, there's certain things that once they're written, they're just, nobody goes in there anymore. And once they're written

and working and tested, they need to be tested, but it doesn't necessarily need to be a unit test and have a unit test that runs with the suite. Not unless you can put a logic analyzer in there, too. And the higher level you go up in the stack is when you start to get things that have a lot of churn, a lot of people in there. Algorithms. Algorithms, you know, customer-facing UI stuff, things where there's timing and race conditions and intermodule communication. That's where unit testing really has a lot of value because people are going to break

that stuff constantly. And it's kind of a paradox. Like, how do I know what people are going to break constantly unless I'm testing everything? But that's where some experience and understanding of the architecture and some history with the kinds of modules that people write is useful, I think. I think this is one of the points where you can't really trust your instincts as an engineer like what aspects of your code do you feel uneasy about those should be covered in tests

the things you don't worry about you know whatever test them don't test them doesn't doesn't really move the needle and and it tends to be that engineers have a fairly good understanding of where where the critical aspects of their code are like if you cannot if you ask an engineer what scares you the most about your product at the moment they will be able to point to it and this is what you might consider covering in tests yeah i would write a test for every piece of code where you have a comment that says i don't know why this works but don't change oh dear yes oh dear uh and and i don't know why this works, but don't change it. Oh, dear. Yes. Oh, dear.

And I don't know. There's a phrase. I can't remember who said it. They were like, write tests, not too many, mostly integration tests. That's right. And I can't remember who said it. So apologies to the luminary out there who came up with that aphorism., I'm looking at this device on my bench right now. And Luke and I have talked about this in the past.

Like I have a suite of integration tests that basically, you know, use the CLI, the, you know, over serial and tell this thing to do something and read some data back. And that's exercising a lot of functionality. And it's nice because I now have this automated suite of these things that I run and it, and it takes probably five minutes cause

it actually does a lot of different things. Um, but I can run that like, uh, I can run that every evening or whatever. I don't run it on every commit. Um, but you know, when I'm kind of done for the day and I'm kind of packing up, I run it and I, and I immediately know, or just whenever I feel not confident about something, I'm worried if this broke something, I run it and I go get a cup of coffee and I come back and it's done. Uh, and it makes me feel better. Um, the, not everything in this device is unit tested, but, but anything that I, I didn't feel comfortable, especially, you know, error injection. Like if, if a module doesn't see an error come along very often, but when it does, it needs to do the right thing. Even if that's a spy driver,

like how often do you get spy frame errors? Once a career, like not very often, but yeah, but, but you know, if, if there, if there's something where like a spy driver has to handle some situation correctly, I might put a unit test on that. Yeah. Safety critical is another aspect. Exactly. And I would have to, um, then that might drive the architecture of my spy driver to separate concerns. I'm not going to test that it writes the correct values to registers. Some people do that. I don't. I would put that behind a very low layer shim abstraction

and then unit test the layer above that to make sure it does the right thing given what it's seeing from the registers. Okay, moving on. Another listener question this one from doug g who clearly is trying to poke the bear requirements specifications and documents can agile help i'm gonna go get a so so so what i advocate with with documentation is as much as makes sense, make your documentation automatically generatable. So for medical devices, we've talked about this past requirements. The agile way to develop requirements is to have some collaborative repository so you're not like have a Word document stored in Dropbox, which I've done a hundred times and you're yelling at people across the office. You just overwrote my changes. No, you know, and Apple is flying all over the place. Um, so like the agile way to do that is to write it in an environment that is collaborative because you know, you're

going to have to collaborate on them. And then if there's a pro, if you need to then take that from that collaborative environment, maybe it's a wiki or some kind of requirements tool database. And then if you're copying and pasting that into word documents, that's error prone and you're going to have to do it a bunch of times. So you might as well take the time to write a script to automatically generate the document in the right format that the FDA is going to expect. Um, like that, that's how you apply that. That's an example of where people, the waterfall methodology assumes you only do that process once and nothing in real life works that way. So take the, like after you've done it the second or third time and you early in the project

and you realize you're going to have to do it a hundred times, take the time to automate that. And then it's just taken care of. And now that feedback cycle is very quick every time. And you're always in a shippable state. I don't know. Does that, does that help for requirements and documents? What do you think? I mean, I like the idea. You're always in a shippable state. I don't... Agile can help with documentation and specifications and requirements if you look at it from the perspective of risk management and

fast feedback loops. In some cases where you can't make a prototype to show people, when you're talking about satellites or big equipment. Why can't you make a prototype, a satellite? Oh, let me finish.

You can't make one initially to show people, so you give them the requirements. You give them the specifications, and they can okay those. Those are the documentation form of minimum viable product. I think that you can't necessarily make a satellite on Earth that functions as it would in space. I mean, the James Webb Space Telescope, I could tell you, probably was not developed without requirements and specifications. Those still can be developed with Agile because it's about communication. It's about showing people what you're going to build and getting them to understand early in the process

if that's what they want. But do you agree, Luca? It sounded like you weren't really sure about the whole satellite thing. No, no, no. I'm perfectly happy with what you're saying. I mean, this is a trope that I get quite annoyed with, really,

to say, ha-ha, we're agile. We don't really need requirements or documentation or anything of that nature. Of course you do. You just accept that they will change over time as you learn more. That's all. But just like you said,

if some aspects of the James Webb telescope can't be, I don't know, can't be tested on Earth or just need to be figured out first, you need to write them down. You need to make them understandable, shareable, communicable. Is that a word?

Although usually it's used with diseases so maybe not okay fine but i guess you know what i mean and that is really the point is it useful to somebody is you know are you transmitting valuable information to somebody um that is the point if you pull that off then you're doing it right by definition. I do really like to think about specifications and requirements as early agile. It's agile before you do the code. And they're never fixed. They're part of the conversation of, is this what you want me to build? And of course, there are times when that doesn't make sense.

You do not give your customer a requirements document if they're a consumer. But if you're working with space equipment or aviation or medical, yeah, yeah, you're doing the first step of, is this what you want me to build? And it's not fixed in stone until you give it to the FDA, in which case it is. Yeah, I've always had trouble with, there have been people I've talked to who said, well, why bother doing requirements they change? Or why bother doing a design, is this going to change? So you think about it.

Why bother writing code? Why bother doing anything? It's an illogical position, but I think my gut feeling is some people are like, I don't like writing documents. I don't like thinking about this. It's much more fun to write code. So if I can come up with a justification for not having to do this, that sounds really good. Well, the customer is always changing the requirements. So it's not worth writing those down. I'll just think that's crazy yeah and i i guess because i focus on medical devices that i have for so long requirements are just yeah it's just part of

the deal but but in in consumer products i would say like i don't know that i don't think you would have a separate requirements and then a design you would just you would have documentation of how are we building this darn thing um and like it has only the level of detail that makes sense so that the upstream people can look at it and understand it and agree with it and the downstream people can not step on each other's toes um i mean if you get too detailed from that then then it's just it's never in sync with the actual code and it starts to lose its value so change that um so yeah again because i'm in medical devices we have very like you have the requirements and then you have the design and the only thing that's that that i do about applying agile that

is just recognize that it's not this linear thing that some people even still have they they even have in their in their you, you know, Microsoft, uh, project schedules, like these gated things, like the requirements are here on this date and then they fix. And then the design document comes here and that's where I have to take them by the ear and say, no, that's not how real life works. Um, like, yes, you spend time arguing over requirements upfront, but you, you, you get that as, again, that's kind of that minimal set of requirements. Yeah. And if we have to argue over what it's going to do, like if you want prototypes or mock-ups or whatever, we can do that.

We can do all that. And let's get to the minimum set of requirements, design, and architecture for that. And there's always just going to be just tight feedback loops. Everything we get, and oh, crap, we can't build that. It's not going to, you know, I, the, the CPU is too slow. Either we have to have a faster processor or I have to do this loop slower. How does that impact the functionality? And, and all of those things, you just have to do it many, many times. So don't introduce a lot of friction into each one of those points. I have one last question for each of us to answer

from Scott Watson. What's a good elevator pitch to give for Agile to a skeptical manager, especially one who has actively resisted Agile processes even while the surrounding teams are adopting them? We're all arguing for Agile here. No, I'm just trying to think of the, yeah, that's very specific. I'm not going to come up with this. I'm just going to throw, I, I,

you were mentioning earlier that the couching it in terms of risk management is a great sell. So I don't know if we can come up with a pithy 22nd version of that. So the thing that I'm reading here is that this is, this feels like a bit like a real-life situation. And there's just somebody who... Do my homework for me, please. Please talk to my manager.

Yeah, don't get me wrong. I totally understand those people. Also the people, you know, the engineers who would really want to be more agile, but are bound by the structures within the company. You know, it's just really difficult for me to tell them, you know, there's only so much you can do without having some power at your company. So it feels like this hypothetical manager

is resisting just because they don't like agile for some reason. And so that feels like whatever factual arguments we come up with are not going to move the needle. Because what's that saying? You can't reason somebody out of a position they didn't reason themselves into. So I would love to say something pragmatic about risk management and

speed and all of that but maybe it boils down to do you really want to be the guy who who tackles this last do you really want to be a laggard in your company and instill some good old FOMO in them

maybe that is the pragmatic people management approach. I think there was something we have touched on here and that's made me feel a little bit better is, you know, there's a cafeteria approach to Agile. You pick the things that work best for your company, but the goal should be to eliminate artificial rigidity from your processes so that you're not locked into things that are harming you

and making your development process slow just because they exist as part of your process. And finding those things and eliminating them and putting in rapid iteration cycles where you can answer questions that are high risk and get them out of the way. That's always been my philosophy with development is what are the risky things?

Let's make sure we can do those in those work before, before we get too far along. And it's like, you know, it's September and we need to ship in October. And Oh, by the way, the battery only lasts two minutes because we didn't bother to do a power

analysis, stuff like that. Um, so I think where agile helps is kind of giving a framework for eliminating rigidity, increasing your ability to answer questions quickly, and to be able to adapt to new situations that come up inevitably when you do development. And if the word Agile is turning a certain person off, just don't use that word. Substitute nimble or just give the specific examples.

Don't say, hey, we need to do agile to reduce risk. Like, hey, we need to test this thing to reduce risk. Hey, we need to build this mock-up. Kind of use the situation, the specific scenarios. And if it's obvious that that's the right thing to do that but and you're not couching it in we're bringing agile in i think the person is much more likely to go for it maybe not and maybe they're just in transient if you say you're going to hire six scrum masters then maybe you know that that's that turns people and that's usually not the answer i think what you do is you get get some custom printed magnets with the words risk management, fast feedback loops, early customer, I don't want to use feedback again, early customer comments, minimal projects that can be shipped, and you just shake them up and you figure out how to say that.

You can put nimble in there, but don't put agile. Okay. It's about getting the feedback earlier. It's about not waiting until it's too late. It's getting the feedback fast enough that you can react to it in a reasonable manner to reduce the risk of the product as a whole, reduce the risk of your piece of the product and reduce the whole company's risk

as they work to find a product that can get customer feedback faster so it's all about little mini milestones and trying to get things more easily defined i like that yeah which is also always a good idea isn't it call it agile don't call it agile yeah it's what i've always done so i just never never called it Agile. I mean, I was doing it before Agile existed. Yeah, I'd like to say I wasn't there, but I guarantee that the wheel wasn't invented

in a waterfall process. Right, right. But it was used on waterfalls later. Anyway. Yeah, no, I think a lot of the trouble that managers have when people bring Agile to them is that, unfortunately, there's been an Agile industry that's built up and that turns people off. And I think when somebody says, oh, I want to do Agile, they think, okay, we're buying a product.

We're buying the Agile. We have to buy the Agile product. And that's where people get in trouble instead of just, here are these principles. We can implement these principles in different ways, but we should hew to these principles because they're very helpful for getting us past roadblocks and answering questions. It goes back to Matt. He sees anything that shiny as hiding something under the surface. Right. Oh, of course. Of course.

There's lots of things hiding under the surface of Agile. I mean, it does have its downsides. One of which is it's maybe more effective, but it's certainly less efficient. And a lot of people are distressed by this loss of efficiency, which is just a thing. But also, I think if you approach a manager and say, you know what, we should be doing agile. Um, what you're really saying is I'm going to take away your approach to managing risk, which you have so far done using contract and specifications and all of that.

And if you don't give them a good answer on how you're going to manage risk now, then they are quite reasonably concerned yeah they say well there are risks here how do we address them and if you don't give them a good answer then they are quite right to resist yeah yeah if you're just going in let's do agile you better have a good story for how that applies to your product your actual organization and how you're going to use it rather than just here's here book. Let's do it. I say don't use the word. Yeah. But even if you don't use the word, like Luke is saying, you have to have a story for how

what you're proposing is going to solve problems instead of just be, you know, a list of things on a piece of paper. Also, you don't need permission to do agile. These things about fast feedback loops, they don't actually require other people until you start working at the team and higher levels. You can do fast feedback on your own. Yeah, I think the strongest point here

is things like TDD. Look, who is going to tell you when to write your tests? That's just childish. If you feel like doing Tdd or unit tests go ahead and do it it's your own personal decision that said there is going to come a point where you really need buy-in from the rest of the organization in order to apply your ideas and your principles to more parts of the overall value stream of the overall product creation process. You can only go so far on your own.

Well, I think we are about out of time. So I'm going to say I am Elysia White. I work at Logical Elegance as a consultant. I am the author of O'Reilly's Making Embedded Systems. And I hope you come listen to us on the Embedded Show. I'm Christopher White. I also am a consultant at logical valiance and I try to do as little as possible.

I'll say I'm Jeff Gable. I am a consultant for the medical device industry and embedded software. So I develop embedded software for medical devices and do all the documentation and everything to get your product through FDA, at least from a software side. So come to me if you want to know more at jeffgable.com. Luca? Yeah, and I'm Luca Ingianni. You can find me at luca.engineer.

So luca.engineer. I promise that's a real URL. Yeah, I do lots of training, lots of consulting in this entire space of Agile, DevOps, scaled Agile, helping your teams to get faster, get better and enjoy their work more. Luca and Jeff do have a podcast of their own, the Agile Embedded Podcast.

Please check it out. Thank you very much. Yeah, thanks for reminding me. Oh, yeah, that whole thing. Thanks, guys. This was super fun. Thank you. Thanks, Alicia.

Thanks, Chris. Likewise. Thank you so much.