Embedded - 16: Democracy Is the Worst Form of Government

Episode Date: August 28, 2013

Elecia tries to get a handle on whether Agile works with embedded software.  Curtis Cole (@citizencurtis) argues in favor of user stories, scrums, and story points. Agile software development on Wikipedia Test Driven Development for Embedded C by James Grenning   "Many forms of Government have been tried and will be tried in this world of sin and woe. No one pretends that democracy is perfect or all-wise. Indeed, it has been said that democracy is the worst form of government except all those other forms that have been tried from time to time." Winston Churchill

This is Making Embedded Systems, the show for people who love gadgets. I'm Alicia White, and today we have Curtis Cole here to talk about Agile and Embedded Systems. Hi, Curtis. Welcome to the show. Hi, Alicia. Thank you for inviting me. We've known each other for a few years, since we worked together at LeapFrog. But if I didn't know you, how would you describe your career? To most people, I describe it as a string of meaningless jobs, but that's a little bit unfair to all of the great companies that I've worked for. Since I've been in Silicon Valley for

32 years or so, I've worked for quite a string of companies, three of them for about four years each, but a lot of two-year startups. Well, that's normal. Certainly, my resume is one or two, four years, and a bunch of one and two years. Thus far, it hasn't been too much of a barrier to finding new jobs. Most interesting, I think, is given that it's been 30 years since I've been out of college, it's surprising how useful and still relevant a lot of the fundamentals are as we interview new candidates all the time. We're still just checking for basic fundamentals, algorithms, data structures. O-notation.

O-notation indeed. And you've done the engineer thing, you've been a manager, you've been a director, all sorts of stuff, right? Indeed. And most of the time has been in management. Early on in my career, got to be project lead, got to be manager, a handful of startups as VP of engineering. Still working, so apparently none of that panned out financially. You'd get bored. Many, many times. I looked at the transitions of the industry over those 30 years.

I've always been kind of on consumer electronics, embedded systems, low level. I watched the internet bloom and I said, you know, there's a lot of future there. I should make the transition. And I never, never did. And surprisingly, the world has come back around again. Embedded systems systems consumer electronics is hot there's a lot of exciting things going on i decided to make the transition to go back into engineering and i'm really loving it which is you know the thing why i was started programming

30 years ago yeah i i totally understand i i did a little bit of management and it was interesting and cool and had some good things. And then I went back to engineering because that's where it's fun. So my plan for today is to spend a bit of time chatting about software process, which makes me cringe and want to groan as I say that. But the thing is, you seem to like doing agile on embedded software projects. And I've been part of about two and a half teams that did Agile on consumer electronics. And it has never gone well. So my hope is you can help me figure out why I would want to try Agile again, how to get it right, either as a manager or as a developer. That's very exciting.

I do happen to like processes. Maybe that's all that time I spent as a manager and seeing teams that are inefficient. So I've watched Agile over the last several years, some people working for me. I'll tell you a brief story of my first exposure to the Agile process. I was VP of software at a startup company, and there was a small team of two people that were close to getting fired. They were just not performing.

And those engineers themselves were very, very frustrated. It was taking them forever, months, to get things past QA and get product out. They were just basically not delivering. They wanted to learn Agile, change their processes. We said, go ahead, try. And they came back, changed the way they were working, built their own test cases, and found themselves regularly delivering, constantly improving product and became the stars of the engineering team rather than the dogs of

the team. And so right then I saw that there's an opportunity there for us to change what was always frustrating with the waterfall processes that I've used in many other times. And the next opportunity was to figure out, well, how does that apply to embedded systems, which is a different piece altogether. It is a little special. I've been working on this little project that involves a web page and I can totally see how Agile would work there because my web page will never be finished. There will always be something to add. And so I try to make, you know, a line in the sand that when I get here, I'm done enough

to show people. When I get here, I'm done enough to show people. When I get here, I'm done enough to show people the new stuff and it's better. But it's never, I mean, web changes so fast and so often. I can see how Agile with its constant updating makes sense there. But embedded systems, you ship the product and then you're done. And before you ship the product and then you're done and before you ship the product you are not done but it's not so much it's not a requirement of agile systems that you have a a lack of a completeness as you might with

the web where it's never done so you are constantly developing things and clearly you need to ship product so you must need to ship product before it's completed. And there's hardware. I mean, when you ship hardware. Right. With hardware, of course, and with embedded systems, typically you're done at some point and you ship it. But that doesn't mean you should not or cannot use an agile process getting to that point. And in fact, if you look at the embedded systems, consumer electronics, and this is more true today than it was 30 years ago, which are just part of an ecosystem because they're typically connected to other devices that are out through the web, then there's a whole, there's quite a stream of milestones between the start of a project and the completeness

when you're shipping it to customers where you want to deliver incomplete but usable versions of the product. And the agile process is one that allows you to have some control and be constructive during all of those prototyping and intermediate stages as you develop an embedded system. Okay, so we've mentioned waterfall and agile, but we haven't really defined our terms. For waterfall, I would say that those milestones are, well, they're what I learned in school.

You start with requirements, and then you have a requirements meeting, and then you go and you do design, and then you do implementation and verification and maintenance. And sometimes there are loops, you know, instead of just going straight to requirements, maybe you do a prototype to figure out what the requirements are going to be. And maybe you do some re-verification or re-implementation when you realize it's not exactly what you wanted. But generally you go requirements, design, implementation, verification, and then only then you kind of, the maintenance

phase is you start over with that. You have maintenance requirements and then you spin. But Agile, Agile's different. You define it. I was afraid you were going to ask me to do that and there are some great it's more of a philosophy it starts with the point that at least for me that really strikes home is you can't know the requirements at the beginning, waterfall always fails

because the requirements you start out with are wrong and changing and changing absolutely so in the agile process you always learn that. And changing. And changing. Absolutely. So in the agile process, you simply embrace that. You realize that you don't know all the requirements at the beginning. So trying to architect or design something and have it come out of your head fully formed is impossible. So what you're doing there is doing a continuous improvement of things with

the philosophy that you wish to ship product along the way. And I'm going to... Papers are shuffling. Isn't that a great sound? Curtis is pulling out comic strips. No, no, I'm sorry, not comic strips. The comic strips are very good. So are you going to read the Agile Manifesto?

Absolutely. Okay. We're uncovering better ways of developing software by doing it and helping others do it. Through this work, we have come to value, and here's a list of things, individuals and interactions over processes and tools, working software over comprehensive documentation, customer collaboration over contract negotiation, responding to change over following a plan.

That is, while there's value in the things on the right, we value the things on the left more. And so the things on the left are the individuals, the working software, the collaboration, and responding to change. Yes, and it's thinking more of the developers and those individuals with the customers are going through a process of discovering what the requirements are. Imagine

shipping product without ever putting it before the customer to see, well, what do you think of this prototype X, Y, and Z? Oh, I've met those products. They're interesting. Right. So the agile system takes that down to its conclusion where the developers are constantly engaged with the development of the process so that they're regularly developing something that they can put in front of their customer and observe their reactions, learn something more about the requirements for the next phase, and incorporate that in the next designs and implementations that they do. But there are good products that don't do this. I mean, I never heard of the iPhone before. There was no customer interaction there. Well, I was not working on the iPhone. I can't imagine that they didn't have customers and proxies of customers.

I guess Steve Jobs was their customer. And at some part, and Steve Jobs is... Known. Yes. He captures a certain... Yeah, he's a great... He can act as an archetype for an ideal customer. And the phone is at some level well understood. The smartphone, maybe not. At that time, I had a flip with sidekick and it was really cool but then the iphone came and i'm like this is junk yes so what tactically back away from the products but tactically agile has a bunch of words that are associated with it. Scrum and stand-up and points and test-driven development,

which is the only one that I really like out of that set. And that's not part of an Agile system necessarily. It isn't, okay. It has nothing to do with the Scrum. But you're right. So let's not talk about the abstract. I just want to again capture the recognition that requirements are unknown at the beginning.

You want to constantly refine a product by getting feedback along the faces as you go along in development. So how do you do that on a project? And the driving force of this is a sprint, which is a defined period of time, typically two to four weeks, at the end of which you're going to have a revision of a product that you could potentially ship to a customer. Certainly, it's something that you can get some feedback for

and some learnings from to guide you in the next phase of development. And like the iPad or iPhone development, the customer doesn't necessarily mean Joe Schmo on the street. Right. It could be anybody that acts as a customer. It could be an internal demonstration, a review, and feedback into the next phase. And it even can be other engineers. I mean, if I need to write a complicated I2C subsystem, to some extent, my deliverable during that sprint is to make sure I can present it to my team because they're the people going to use it. That's very, very important. In anything that you're developing, in my mind, you need to

understand who is your customer of it. And quite often, deep down in the software, your customer is another software engineer. Well, I think Agile is kind of cool because you do have to define your customer relatively early. I've worked on projects where they defined requirements and failed to define the customer. And that doesn't really work either. It's nice that agile keeps you in touch with who's going to be using this, who cares? Absolutely. So, so there is, uh, without actually, as you say, talking to Joe out in the street, uh, there's a proxy for that in the agile system. Somebody acting as a product owner who's setting some priorities and direction and trying to channel the customer for priorities for

the agile system. So is that a product manager or a program manager? Is that kind of the same role? That depends on the organization. It doesn't have to be an engineering manager. It needs to be somebody who can understand the customer requirements well enough to translate them to the engineers in the work that they need to do. Yeah, okay. That makes sense. Typically, quite often through features or actually through user stories as a description of, as a iPhone user, I want to be able to make a phone call while I'm... Surfing the web, ignoring my parents. Absolutely. Something of that nature, which the engineers can translate into, what features do I need to implement to make that happen? Right.

The phone needs to work as a phone, and it also needs to work at a web server at the same time, which they may not have realized initially. Right. Now, the engineers have to be smart enough to realize how big a task is this. They'll break down that particular user story, perhaps into a couple of other smaller stories about starting a phone call, ending one, talking, muting, etc. And then they'll provide some estimation, and classically done in something called story story points for the size of that. Story points are a way that Agile avoids talking about schedules. Is that true or just my imagination?

The goal is to avoid getting bogged down by talking about how long it's really going to take to implement something. So you pick some abstract value. Some people use t-shirt sizes, the classic size. So you can say, well, this is bigger than that, or it's smaller, it's tiny, it's giant and so on. But as engineers, we always want to quantify everything. So on the teams I've always worked for, they tend to start mapping those points down to how many days or weeks of real effort is it? Yeah, it does seem like it's sure it starts out as small, medium, large, and then it switches pretty quickly to small must be three days and medium should be seven and large should be 12.

Again, the goal is to not get bogged down in the conversation. So you let engineers be engineers and just make sure the conversation moves on to. We got a rough idea of how big it is. Story points is going to get eight points. Let's move on. And management. I mean, this is fine as a engineering level, but doesn't somebody need to know when we're going to ship this?

I mean, story points are nice, but how do you get management not to say you need it in three weeks? Some parts are greatly improved by an agile system. So we really haven't talked through some of the mechanics of the agile system. And then we can talk about some of the things that are frustrating in it. So in a sprint, there is to execute a particular sprint. And a sprint is something that you rinse and repeat multiple times. And this is two to four weeks?

Two to four weeks. So there's a sprint planning meeting at the beginning where a product owner will set a vision for this is what I'd like to accomplish at the end of the sprint. Perhaps a demonstration. I'd like to show these features. Here's a bunch of user stories that I'd like to have implemented. And he presents that to the team and says, and this is how much, you know, work roughly, I think it is.

He just has a gut level for if it's achievable or not. And then in the perfect world, he leaves the room and the engineers themselves start to break this down, make sure they understand what's needed by that, may call the product owner back in, ask some questions and so on. They allocate points to it. And based on previous experience, they say, this is how much we can sign up to accomplish during the sprint. So the product manager isn't around to say, oh, I didn't realize that feature was so long. I would rather have this feature. Because a lot of the product managers don't know what's one line of code and what's a whole new server system. Actually, in an Agile system, you will expose that information to the project manager.

Ideally, you can size tasks for the next sprint. There's a backlog you have for tasks that you're not performing this sprint, but in the future you can size them out so that the product owner can look at that backlog and start to shuffle things and prepare them for the next sprint. He has a rough idea of how much work there is. Okay, so we're in the beginning. You had a special name for this meeting. A sprint planning meeting. Sprint planning meeting.

And the goal there is for the product owner to specify the priorities and goals for what he hopes to get accomplished, and the engineering team to sign up for how much work they believe they can accomplish during the sprint planning. And then you start the sprint sprint and they do their work. So you aren't really working on schedules, you're just trying to work on, can I finish this feature in the time given? In the end, they're the same thing. What work gets done by this date? So it's a schedule. It just happens to be only looking the sprint length out into the future, two to four weeks. Okay. And then ignoring what's going to happen past that.

Indeed, focusing only on the here and now. So in completing the description about the agile process, the engineers record the amount of work as they complete it during a particular sprint. And you can see in something called a burndown chart, how much work is being accomplished versus the work intended to be completed during that sprint. At the end of the sprint, there's a retrospective where you do demonstrations of what you accomplished for each of these user stories. There's a retrospective where you discuss what you did that you liked and want to continue, what you did that you wish to stop doing,, what you did that you wish to stop doing,

and some new things that you'd like to start doing in the future. And you can, over the past several sprints, look at how much work was accomplished, something called a velocity, how many points of work the team can actually accomplish during a sprint. The multiplier, the story points today is multiplier that I'm sure some program manager is going, that's what I needed. Any engineer will do the math there, but a product owner or a product manager who's more math challenged, perhaps, will simply look at for each sprint, they can do this many points and they'll look at the backlog and say, oh, they can accomplish this much of the next chunk and this much for the next chunk of tasks.

That requires really having the points dialed in, and that requires understanding what the project is or what the features are. Well, the points dialed in part, the team does get a feel for how big tasks are, how much they can accomplish, and how big things are and assign the points correctly. So there's a feedback mechanism there, and that works out okay. But what it doesn't do is translate a two-week sprint, what you can accomplish there, to what a team can accomplish on an 18-month engineering project.

And that's where some of the failures of an agile system lie for a project manager, which is they still don't have a tool that gives them the visibility and confidence in what a team can accomplish over a very, very long period of time. Because in those 18 months, there's probably a feature that's like make the display subsystem and drilling down into that isn't defined yet. And so estimating that as anything other than extra, extra, extra large project. If you tried to define all of those tasks for the complete project lifestyle, you fall back into the entire waterfall process.

Then you're down to requirements and design. Which you know are not correct anyway. So what the agile system acknowledges is you're never really going to know because you don't even know what those requirements are and things are going to change going forward. One of the things you didn't mention was stand-ups, which I thought were a really important part of Agile. They are an important part of Agile. I think they're a very important part of Agile. Frankly, I think they're very important for an engineering team, even if you weren't doing an Agile. But they get a terrible reputation. They're supposed to be short status meetings,

but after an hour and everybody takes a turn talking and nobody is listening. Yes. Well, democracy has a terrible reputation as well. Nevertheless, it's probably the best tool we have for running civilizations or societies. So stand-ups can have a terrible reputation because they quickly get off topic. Engineers like to dive deep in things and you

have to fight some of those engineering instincts. In a stand-up, you really want to do only three things when you speak, which is what did you do yesterday? What are you doing tomorrow? And where are you blocked? When you speak, you need to think about what do other people need to hear from me? Not all the technical details. What do they need to hear from me so that they're not blocked? And when you're listening to other people and before you ask a question, you need to think of what do I need from this other person? If it's a tiny thing, you can just say, tell me this information right now.

So the meeting doesn't get long. If it's a large thing, you simply have to say, I need to learn more about that, otherwise I am blocked. And that's hard for an engineer to do. And those three things that you have to talk about yourself, the what am I doing today, what am I doing tomorrow, and how am I blocked, those you should go into the meeting with. I certainly have been one of the people that everybody's talking about whatever they're talking about, and I'm trying to think about what I'm going to say, and I'm preparing my little speechlet.

And then they say stuff and I miss it. So you're an engineer. Great. Exactly. Exactly. And I know other people do this because then I say, I'm blocked on so- so and they like wake up. What? What? I'm sorry. I missed that. I had that conversation happens a little too often in our stand ups. I mean, not you and I because you and I don't have stand ups, but in stand to change people's personalities. You're not going to change the way engineers are. But if you get nothing else out of the stand-up, at some point someone will say your name, Alicia, I said I'm waiting for such and such, and you will wake up and respond to that,

telling them what they need to know. Or at some point when you go through your own list, you'll realize I need something or other from Curtis, and even though Curtis has already stopped talking and you weren't paying attention, you can at least ask him, I'm sorry, can you tell me again about this? Because I need that. So in the end, the information gets exchanged so that people get their critical information they need from one another so that they're not blocked and they can continue to be productive. I feel like we're all tasks and this is when the scheduler runs and the messaging system runs is is during the stand-up

Yeah, that works um, so i'm actually kind of happy because I thought that I thought that this was just a symptom of the terrible stand-ups that i've been part of but it sounds like you're saying No. No, this is normal. Engineers don't pay attention to each other because We're all thinking about what we're thinking about That's really normal, huh? Well, there are at least two of us that do that. No, there's more than that because in the meetings and all that seems to be. Now, it is nice if someone in the stand-up is an adult and can actually say,

please take that conversation offline. Or Alicia, were you listening? Your name was just mentioned. So is the name for that person the scrum master or do we just call them, you know, mama? It's usually the adult, but it is nice to have a scrum master there who actually keeps people on the process so that things don't get off track. Well, so scrum master is an official title. Yes. What are their other duties? They should also manage the sprint planning and the sprint retrospective as well.

So ideally, they would be the person who would be strong enough to say to the product owner, I'm sorry, you need to leave the room right now. So is this in the waterfall methodology, would that be like the manager? Or is it a different person? I don't think you can. I wouldn't relate the one to the other. would that be like the manager or is it a different person? I don't think you can, I wouldn't relate the one to the other, the scrum master having an analogous role in the waterfall process. And is it a particular, you know, you are the scrum master for this team or does it change day to day?

No, you would be the scrum master for the team for an extended period of time. For at least the whole sprint? Yes. No, no, no, no. Probably for the entire life of the project. There's no reason to change it every two weeks. Well, you know, if the retrospective, I said, I hate Curtis, wouldn't that, you know, feed in? Yeah, well, that's a different issue. If you're raising personal issues during retrospective, that's a little different. So ideally, you'd pick a scrum master as an individual who's a little more process-oriented and was able to listen to other people speak while still thinking of their own thoughts. And they've got to be a decent communicator, because they do need to be the person who walks between other people. Right.

If people are failing to listen to each other, they're the person who detenses, that's not the right word, decreases the tensions. Yes. Perhaps somebody with some management skills who doesn't want to be a manager, yes. Oh, I see. So you've been doing scrum mastering, huh?

Now and again. But that is, is that the same as the project architect i mean how do if i have a complicated 18 month project how do i do the architecture because if i'm just constantly doing what i need to do in the next four weeks how am i going to make sure i get it designed enough that i'm not reproducing my work all the time. And in my experience, that's one of the hardest things I have with agile systems and teams have with agile systems and even managers have with agile systems is the architecture and designs.

Let me try a parallel explanation for that from test-driven development. Test-driven development is, I've seen it used within Agile system as well, so that when you have a story you're supposed to implement, one of the first things you do in implementing that is write the tests for that story, and then you start writing the code to implement those, and you go through test by test, implementing the code to pass the test.

But that's not required for Agile. Certainly, I have worked on FAA and FDA systems, and you write the requirements, you write the test, and then you write the code to pass the test. But that's not required for agile. Certainly I have worked on FAA and FDA systems and you write the requirements, you write the test and then you write the code. And so that's right. Test driven is not necessarily agile, although it's my favorite part of agile. But there's, well, there's a piece of it that I want to take over in answering your question about architecture and design within an agile system. So, and I've done only side projects with test-driven development, so I'm by no means an expert. But in the test-driven development,

one of the rules you use is when you write the code, you just write the code for the next test that's failing. And you write it to pass that test. And then you go on to the next test, and you write it to pass that test. And then you go on to the next test and you write the code to pass that. And then you get on to the next one and you realize, I need to refactor my code. Because as this one, the simple thing I did beforehand doesn't pass and I really need to architect it a little bit better and do that.

So as you write the code, you learn more about what it needs to perform as you execute these tests. And then you learn, you learn more about what it needs to perform as you execute these tests, and then you learn, you become smarter so that you refactor and re-architect your code. That kind of process where you don't try to architect the system up front has to apply in the Agile system as well, which is related to the overriding concept as you don't know the requirements well enough at the very beginning to architect the entire scene. So that means within a given sprint,

you're going to come up with some new feature, new user story, new requirement that's going to require you to rethink an architecture design of a subsystem or the entire system. Hopefully not. I like the idea that as you are building your design or as you're even building your requirement, you have to figure out how to test it. I mean, I really like that. I'm not really sure about this. Every line of code, you write a

whole test for it or you improve your test for it. I know some people are in that camp and James Grenning wrote Embedded test driven develop Oh test driven development and embedded see right and he's gonna be on the show probably a month a half, but That's awesome. Thank you for the reference I read his book and that's where I was taking his statements that you just write enough code to pass the next test and so on. And therefore, he then acknowledges that at some point during this, you're going to end up having to refactor your code. And refactoring, I think, is a really good thing. It means that you can go from having your code be all bloaty and 100 statements and case statements and state machines to having

something really trim and nice and optimized as well as working. And once you have all your tests, you actually manage to have some confidence that the next change you make gets tested automatically. I like that. I'm still a little, the whole baby steps thing. So put aside the testing part of this and look at it from a two-week or a four-week part of an agile system where you come into a new sprint, you've got some new user stories, which are some new requirements, do a redesign at that scale and at that scope so that you can implement all the existing stuff plus the new user story, the new requirement in there. And so you're doing refactoring within Sprint and redesign within Sprint, not looking at the entire

project 18 months long or six months long, whatever it is, but only looking at what you're going to need to accomplish for the short term, the part where you actually have some visibility. It sounds more organic and more of a bottom-up design instead of a top-down design. And I'm a strong believer in top-down designs. When you do that re-architecture and redesign, you can still do it and should still do it from the top-down. Who's my customer for this subsystem? What do they need to do? If you have some visibility into some future user stories, you want to pull it in, you know, you're an engineer, you're going to want to, you're going to do that anyways. But it has to be done within the scope of your sprint. You can't decide to rewrite the entire system in one sprint. You're going to have

to just do some piece of it and realize in the future, there's going to be some other re-architecting that we need to do. One of the challenges on the agile for the manager and even for the team is putting in a task into the sprint that says, I know I'm just going to need to refactor this. And at the end, my demonstration is going to be all the same features I had beforehand, plus this tiny little thing that doesn't seem like very much, but I know I've actually accomplished a lot of work in the refactoring underneath. So that's actually a good task. That's not one of those tasks that gets put in and everybody thinks that it's a waste of time.

It's worth putting in that task. No. And in fact, in my current project, and we've reached a point where we said, gosh, there's this big problem. We need to fix it. We're all excited about fixing it. Actually, the team is engaged in doing this refactoring. where we said, gosh, there's this big problem. We need to fix it. We're all excited about fixing it, actually. The team is engaged in doing this refactoring.

Engaged in doing refactoring. Yeah. Wow. I mean, refactoring means I did it wrong the first time. How do you get over that feeling? Well, as a father with two daughters, I know I'm a little different than other people. I don't get wrapped up in the feeling that, oh, I did it wrong before. Well, you've got two.

I mean, the younger one's better, right? Which one do you love more? Again, I don't provide the values. I don't go down there. It's not about which one do I love more and I did it wrong. It's nothing personal. It's simply an engineering. I know more now.

I need to do this better. And I like the opportunity of building better software. It's okay if I have to throw out code. Well, and I remember you mentioned college topics that come up still. I remember they taught us egoless programming, that you shouldn't sign your code, you shouldn't sign your comments. You should do the best you can to have it not be about you. And it sounds like that's one of the things you're suggesting makes Agile better.

Well, no, I'm suggesting that makes it better for any engineer anywhere. Okay, I'll agree. But if you can, and as the team I'm working on is learning about Agile, if the team can learn to embrace and accept the refactoring that's going to occur at some unknown times throughout the life of the product, then you can do those well and you don't think of it in terms of, oh, so-and-so implemented this improperly. I know there's a lot of people

that would like to rewrite an accelerometer driver that I wrote previously. And how do you not feel like it's wasted time like if i had done this i guess that's back to if i'd done it properly so and you've answered that it's it's about not worrying that i did it wrong it's not about doing it wrong you did it right at the that time and then things have changed except the agile part which is you're going to start this accepting that you don't understand all of the requirements at the beginning. So you're not going to try to design the system that's going to stand through the complete development. You're going to accept that you're going to learn more about the requirements and the customer as you go forward.

That's inevitably going to require redesign of some portion of the system. And it's refactoring code and architecture. I mean, you have to accept that your whole system may get flipped on its head or whatever. And so that's how you do architecture is two weeks at a time? Yeah, I still struggle with this as well. If you're talking fundamental architecture, then yeah, you may have blown it. So you can't take a team of junior engineers, teach them an agile process, and have them build a space shuttle, for example.

They are going to have to have some experience before they start. There's going to be some skeleton, some foundation, some architecture, some decisions made at the beginning that might be prohibitively expensive for you to change without resetting your schedule. So that's just like any engineering project where you realize,

I can't change some architecture thing that's too expensive, and hopefully we got it right. And in Waterfall, you definitely can go down a path where you've designed an architecture based on requirements, and one little requirement change means you're toast. Agile is just accepting that up front, accepting it will happen instead of worrying that it might happen. Yes.

It ties in with a lot of other things as well. If you've designed your system so that it's more modular, hopefully an architectural change in one portion of the system doesn't ripple through and destroy the rest of the system. Well, that's one of the things I actually like back on test-driven design is it encourages a level of modularity to me. And I like modularity. I agree.

And in embedded systems, I mean, we've been kind of talking about Agile, but we're both pretty and we make devices, whether it's children's toys or all of the things that I always say, race cars and whatever. It's things you touch and hold and move around, not necessarily computer software that you reload. And nothing we said prevents you from using it on an embedded system as I do. And I think it's very important.

Many of the products we work on go through various phases. There's prototype hardware. There's a revision of the prototype. There's some initial production. There's very important. Many of the products we work on go through various phases. There's prototype hardware. There's a revision of the prototype. There's some initial production. There's mass production. Well, that's nothing more than an agile system. They're just having deliverables. In the hardware world, it's just they're spaced very far apart.

We're doing the same thing on a smaller level with the firmware, the software, as we go along. And the modularity is nice for refactoring and for test-driven development, but also because if you're working on an embedded system, you don't want to have to emulate the hardware, at least not all of it. I mean, I don't want to write a processor simulator. Well, that would actually be kind of fun,

but I don't want to have to do it and get something else done. How are you, does that matter? I mean, that matters a lot for test-driven development, but for Agile, how much does it being embedded matter? I don't think it matters much at all. As long as you have hardware where you can run your embedded system on, or a simulator, I suppose. You can develop software with confidence. It will run on the final product to the extent you know about it now.

And then you can demonstrate it. That's all it takes. You know, here's a user story. I'd like to implement this feature. The team implements it. They demonstrate at the end. They get some feedback from the customer. That's what I wanted. Oh, now I know I need something else. And a new requirement is exposed and goes into the backlog in the future. And how do you deal with shipping? I mean, you said that an agile project can end. I mean, manufacturing starts and it's a program that you can't really update the firmware on. That is an end. Well, at some point, your manager is going to tell you this

product needs to go out to customers. What do you do differently there? Perhaps the scope of the tasks that are going to the sprint are changed. They're bug fixes, they're reliability testing, they're minor feature changes. They're not things that disrupt the entire system where you would have to go through two months of testing before you would ship the product. And so you can still ramp things down. You can run an Agile process all the way through shipping.

So one of the problems I've had is that Agile doesn't seem to value stability and robustness. It really emphasizes flexibility that in my experience with the very few number of products that I've worked on that use an Agile process, it encourages buggy products and it's easier to fix a website than it is to fix a firmware update because not everybody flashes their firmware you should see what level my iphone is on or what os it is and agile extends this we're going to work on neat new things through the whole life cycle and it's bad for embedded systems because at some point you have to say, this is it. Should we just be doing Agile until verification?

No. Agile, remember, Agile is a way for an engineering team to be productive through their work, through an acceptance, again, of unknown and discoverable requirements as they go forward, says nothing about the ability for a team to deliver a customer-ready product. If you're not going to test it, if you don't have test-driven development as part of the Agile, if you don't write any test cases as part of this, then the code the engineers

are going to come out with is going to be minimally functional and unacceptable for a customer. If you want to ship it to a customer, you need to put in the tasks that are required for any engineering team to accomplish that. You're actually hopefully going to write the unit test for this. You're going to write integration tests, system tests. Maybe you'll have a QA team if you need. Depends on how the testing needs to go. But you're going to have those tasks as part of the process as well.

Okay. So essentially, if you're going to write buggy software, you're going to write buggy software, whether it's going to be Agile or Waterfall. And it's just Agile gets a bit of bad rep sometimes because people are using it, but they would have written buggy software anyway.

Yeah. Agile is only a process. It's not actually engineering. It's not the technology. It's not the testing. You're right. So if you think about Agile as in some sense a tool, and there are a number of tools that help engineers become more productive. At the top,

if you want a successful product, you've got to hire first the right people involved. You hire people with no experience, you're not going to get anything. Once you have the right people, you need to have the right process. And I'm talking here about engineering skills, the ability to deliver quality software. And finally, at the bottom, and only at the bottom, do you start talking about changing the tools. Let's go to an agile process and use a scrum tool as opposed to a waterfall. Maybe a manager is not really up to the task, might say, well, our software is buggy. Our team is not very productive. Let's try an agile process.

That's not really addressing the problem with the people and their engineering skills. That's just changing the tool that they're using. Okay. Yeah, that makes sense. I mean, it's about a process and the process is not... The company still needs a way to deliver a quality product. Yeah. Yeah. That makes sense. So the other big problem that maybe you can help suggest some optimization. I've been working on systems that take batteries and optimization is a necessary evil because the less we run, the more, the longer the battery will last. And that's what the customer needs. To me, optimization is not necessarily something you do at the start, but it is something you think about the whole time. And you should optimize the bottlenecks and not optimize the

whole system. But even at the very beginning, you have to be thinking, how am I going to deal with this? How am I not going to leave the Bluetooth on? How am I going to make sure the Wi-Fi gets turned off? How am I going to make sure the LEDs are all on when they need to be on and off when they need to be off and the GPIOs? Is this just going to end up being a task? There's optimization tasks through the Agile process, or is it one end thing? This is one of the things that I do occasionally stay up late and worry about on the particular team we're on right now.

I heard a rule many years ago about when you think it's time to optimize, don't. And that works most of the time, but there are other cases where you find out too late that, ah, my architecture was wrong and I'm never going to achieve the goals I wanted to. So my current thinking and understanding of this is just as you suggest. Spread out through the life cycle of a product, there are some, if you wish, user stories. If your product is to have a phone that lasts 48 hours, for example, someplace in the process you might want to make sure that your phone can last at least 36 hours

and that you understand where the bottlenecks are or that it lasts 60 hours without all of the hardware draining current on it. Whatever it is, you want some intermediate milestone between where you start and where you need to finish. So on an agile process, I think you do need to insert some user stories that say, we need to look at the system, measure our battery current consumption, find out the bottlenecks are, that's going to expose tasks you need to put in the future that says, oh, I need to fix this driver, fix this component, or fix something. And board bring up is just another task, just another user story that at the end of my sprint, I am going to show all parts of the board function as intended.

Absolutely. Including manufacturing tests as well, if those are deliverables. And do you find it difficult when you're constrained by so many embedded systems constraints? I mean, you can't necessarily change your processor at the end when you realize that the requirement now is to process audio three times faster than what you started with. Well, we work with a good hardware team, so I bet they'd actually like that challenge. But as an embedded engineer, I like all of those challenges, so I don't see them as barriers. But I don't think those are a problem really any more than even your web development. Sure, you could say I could throw another virtual machine at the problem, but even that scalability starts to break down. So I don't consider the embedded system to have additional barriers that make software development fundamentally different.

Well, it is fundamentally different at some level, but at the top level, you know, you got your requirements, you build it, you test it, you show it to people, you learn more about it, and you move on. So, no, I don't see those as a barrier. Certainly not for an agile system as well. When you say that, I was thinking, well, he's back to describing waterfall. It sounds like agile is waterfall made small. I sometimes think of it that way as well. I could say test-driven development is the same thing as well. You come up with a test, this test fails, what do I need to do to make it happen? The requirement is that test needs to pass along with all the ones up to it. At some level,

we always break problems down into little tiny pieces that we ask the question, do I understand what the requirements are of this thing? What kind of design am I going to do to implement it? Let me code it. Let me test it. Let me ship it. Well, this is kind of starting to sound like the scientific method. Define the problem. Define what you know about the problem. Fix the problem. Repeat. Wash. Rinse. Repeat. Okay. Well, cool. And I liked your analogy about agile being a democracy. I, I see that a lot. And I have to admit, I'm not always a fan of pure democracy. Sometimes I wish for a benevolent

dictator, so that I don't have to see all of the system. I don't have to understand all of the issues. Now, remember, I said benevolent dictator in here. So if you're planning on ruling the world, you've got to be nice about it. I am benevolent. Every dictator thinks they are. But what's... I didn't really... And I didn't really describe Agile as a democracy. I only meant to say agile as a softer development process may get a lot of criticism as democracy gets a lot of criticism, but it happens to be the best that we have in both of those cases. Democracy is the worst form of government except for all of the others. Indeed.

I think Mr. Churchill said that one. Yes. All right. Then we won't talk about government, which is just as well. And then let's see, what else do I have problems with? I have my list here. Ship deadlines. I heard somebody say, we will release it with whatever features are done by the time it

is time to release. I've worked on projects like that. Does that work? Because it hasn't so far for me. And my sample size is limited. But I just, the stuttering that occurs when it's like, oh, we'll just ship it when it's time to ship it with whatever we have. And I'm like, no.

That's not really an agile question. That's really a software engineering question. Happy to talk about it, but it's not really agile. It was framed to me as a, we're using agile processes, so you don't need to know what it's going to be when it ships. Ooh, that's, yes, that's a little dangerous because engineers like to sometimes cut corners. Many of them do.

If you tell them, I need to demonstrate this feature, they'll write software so you can demonstrate that feature. That's different than saying, I need this feature to work for every customer out in the field every time. That's a different software engineering task. Yeah. So if you don't tell the engineer what you're doing is trying to get something ready for a customer,

they will not address the 10 little loose ends that are in their head that are still unfixed when they check in their code. But you're an engineer enough to understand why that happens. I mean, there's always something else to do. There's always a shiny object that also needs to be played with. And if you come and tell me I need X and I have 15 other things to do, X may get done. But no, I may not try to spend some time thinking about, well, what does he want with X?

Who's going to use X? What's Y and Z? I bet he's going to want those next. It depends on how busy I am. Yeah, so again, Agile doesn't suddenly make all of your software shippable at the end of every sprint. You still need management and other people and other processes to make sure that you have a shippable product. And you need to use the Agile system underneath that only to make sure that you're making continual progress on your sprint basis and the engineers understand what their priorities are.

Okay. I feel better about Agile. It's less – it was being used to explain a lot of actual bugs in the projects and clients i've been working with but it doesn't sound like it itself is evil it just is being used to hide little gross pieces that shouldn't be there anyway and i can i can fight the the evil part as long as i know where it is yeah you can fight the evil part as long as I know where it is. Yeah, you can fight the evil part as long as, say, your manager isn't the evil part. Because a manager could subvert an Agile process very easily. Of course.

Or they can subvert a waterfall by not passing along requirements or killing designs. Yes, if you're fighting your management it's not agile's fault not necessarily agile's fault yes agile's just the tool they're using to get you to do what they want but it may be the wrong thing yes well hammers are also a tool and i don't like it when they use those either but agile may be worth it what about pair programming is that part of agile or is that separate that's not a pair of, that's not a pair. That's not paired with Agile programming. Okay.

And that's when, I've used pair programming. That's when you actually sit with someone and work with them. And sometimes one person writes a test, one person writes the code. Sometimes you just both write the code. And I've used it and it was awesome and it got a lot done. And we shared screens and it was awesome and it got a lot done and we shared screens and it was neat but the other time I tried it it was bleh because one person drives and I can't see their screen or I can type faster and can't stand the pressure of not typing and and the skill level match.

I haven't I haven't participated in pair programming. I've seen some portions of it work really well. I think it boils down to a lot of chemistry and rhythm between the two individuals. I saw a pair working very well together as a team. Again, this is the very first Agile team that I have observed. And they would write test cases for each other. And that worked really well. They would then work independently, but they were cross-checking each other's work. And talking a lot.

And that actually makes, it made one job a lot of fun just to be talking and have somebody else half in my brain. We just watched Pacific Rim and they shared brains and it was cool and the giant fighting robot.

And David, sometimes I miss you. Don't tell my husband. Well, with that, I think maybe I should leave and make it up to my husband. Thank you for explaining Agile. I suspect I'll be a bit happier trying it again and looking for the parts that are cool about it. Well, Alicia, I hope we can work together again on an agile project. And I'll teach you to be Scrum Master if you wish. Scrum Master sounds a lot like being the grown-up.

And yeah, I think I can manage. I think so. Excellent. Thank you for inviting me. That's the show this week. Thank you for listening. And thank you to Christopher White for producing and not divorcing me. That's the show this week. Thank you for listening. And thank you to Christopher White for producing and not divorcing me. If you have any comments for me or Curtis,

send email to show at making embedded systems.com or hit the contact link on embedded.fm. And if you were one of those lucky few who crossed those signals and send it to show at embedded.fm, resend your email. It's working now. Until next time, may all your boards come up on the very first try.