Command Line Heroes - The Agile_Revolution

Episode Date: January 30, 2018

It's the turn of the 21st century. Open source software is changing the tech landscape. But new patterns of work have now become necessary. Developers search for a revolutionary approach that will allow open source development to flourish. A group of developers convenes at a ski resort in Utah to craft such an approach. What emerges is a manifesto that changes everything. Dave Thomas, one of the authors of the Manifesto for Agile Software Development, brings us back to that now famous retreat where the agile revolution was first organized. Not everyone was as quick to sign on to this new approach, though, and in this episode, we hear why. Read Ruha Devanesan's take on agile's ability to harness diversity here. Please let us know what you think of the show by providing a rating or review in Apple Podcasts. Drop us a line at redhat.com/commandlineheroes, we're listening...

There are certain stories that end up defining an industry. Stories we tell ourselves about where we come from, who we are, what we do. Last episode, we tracked the rise of Linux and open source technology. This story, though, this one's about what happened next. After the OS wars, developers were front and center, and that new prominence meant they were going to reinvent their jobs. This episode, we'll learn how a focus on the developer brought about an entirely new methodology

for software development, and how that new approach to workflow is having an unexpected impact far beyond the action on our screens. I'm Saran Yadbarek, and this is Command Line Heroes, an original podcast from Red Hat. Episode 3, The Agile Revolution. Today's story begins in February 2001, and it's set at a ski lodge in Utah.

We turned up at a lodge, you know, the pine beams and the fireplace in the entryway. We got there the night before and we basically just sat around and talked about what we were going to talk about. And the next day we turned up and we'd reserved a meeting room. We took the tables and moved them out to the edge and we just put the chairs in a circle or an oval. So, you know so we could basically be facing each other and somewhat more open. That was Dave Thomas. Dave and 16 others got together at Snowbird Ski Resort that winter, not to ski, but to talk about what was wrong with the developer's world in the 1990s. I say talk, but argue might be more like it. They had originally met at a conference called OOPSLA,

Object Oriented Programming Languages and Systems. And it was actually at that conference that they all agreed that creating software was messy. They just hadn't agreed on what they should do about it. So the meeting on the mountain at Snowbird, that was where they were going to try and nail down a solution to that problem. And what was that problem exactly? I asked Dave what was wrong with the way developers used to do things. So have you ever, I don't know, decorated a room? I've tried. Or had, okay.

And so if I said to you upfront, okay, I want you to sit down, please. Here's a piece of paper. And I want you to specify exactly what this room should look like when you're finished. Right? Could you do that? Actually, I did this for my own office.

I tried making up a little sketch and a little rendering and putting all the shelves where I thought they would be. It didn't really work, though. It didn't end up the way that I planned. But even if you do that, what do you do? You put the shelves up, and you go, oh, that's not going to work because the door's going to get in the way. So you move the shelves somewhere else. Or you say, you know what, I can't really put carpet there because my chair is going to get stuck on it, whatever it might be, you know. So you always are going to iterate. Whenever it involves any unknowns at all, right, the human brain is just not up to modeling the real world accurately enough to be

able to know what it wants up front. So software development is the same. People don't know up front what they want, right? And the number of times where I have gone through that process, where I've got a spec from a customer and I have implemented it pretty much identical to that spec, it gets to the end and they go, but that's not what you wanted. And you go, but that's what you asked for. And they said, yeah, but that's not what I meant. So this whole process that you can specify

and then move through some mechanical set of steps and then at the end you're done, doesn't work in software. It doesn't work in anything where there in software. It doesn't work in anything where there is ambiguity. It doesn't work in anything where there is some degree of judgment. You know, almost like any, quote, artistic endeavor. It's just not going to work. There always has to be feedback. And that was the missing step. So maybe you've heard of the software crisis that ran through the 1990s. Software

development at the time was just a mess. Companies were spending more money fixing software than they were creating it in the first place. And meanwhile, developers like you and me, we were log jammed. In some cases, we could only put new software out every few years. We were stuck with these sluggish, old, waterfall workflows, A to B to C, totally predetermined. So the search for new processes, for better ways to get software made, was sort of all-consuming at the time. In fact, every month there seemed to be somebody new who had a grand vision for how they were going to fix the software development process.

There was Kanban, Rational Unified Process, the list goes on. The methodology wars saw a slew of competing new visions, new fixes. And that was the scuffle Dave Thomas and his buddies up at Snowbird were stepping into. Their own salvo was something called the Manifesto for Agile Software Development. The speed of production was ramping up like never before. Open source had made developers more powerful, and in turn, developers needed a new, more agile mode of production. The guys who met at Snowbird argued a long time before landing on that word, by the way.

Agile. In the end, it was the descriptor that made sense. It's how you describe big cats in National Geographic. A word that describes the exact opposite of a waterfall's preordained path. As new information came up, it's a word for people willing to change course on a dime. And notice that it's not a noun, it's an adjective.

Agile would be a mode of being, not a concrete thing. So what did those Agile guys have to offer? What was their big solution? The Agile that many of us know today is a complex set of roles and systems. You've got a scrum master, a scrum team, a product owner, and they're all going through one or two weeks' prints of work.

Meanwhile, work is piling up in iceboxes and sandboxes and, well, let's say it can feel like a lot of process. But none of that was there at the beginning. The guys who wrote that manifesto aimed for simplicity and clarity. And the vision was so simple, in fact, that it had the power to define the path of almost every developer's destiny from then on. We came up with the, you know, we prefer A over B way of expressing the values. And we actually wrote down pretty much the four values that are now part of the manifesto over lunch. Four big ideas that could govern development. In case you don't know those Agile commandments by heart,

they go like this. Individuals and interactions over processes and tools. Working software over comprehensive documentation. Customer collaboration over contract negotiation. Responding to change over following a plan. I remember the first time I encountered the manifesto. I was newish to coding, and to be honest, I didn't get what the big deal was. Until I understood the tools and platforms that make Agile work, to me, it was just some fuzzy concepts.

But for developers who've been struggling with these issues for a long time, it was just some fuzzy concepts. But for developers who've been struggling with these issues for a long time, it was a call to action. The manifesto was designed to be a spark that could inspire something bigger. These four values and some supporting material were posted on a website, agilemanifesto.org. And they just straight up asked other developers

to sign their names to show support. I think we were all stunned, you know, when we got to a thousand signatures and then 10,000 and just keep growing and growing and growing. And it basically, it became a movement. It was never their plan to walk out of that ski lodge with a manifesto, per se. They were just a group of people passionate about software development who felt a kind of evangelical zeal about helping others do development better. It became clear, though, that Agile had legs. Red Hat's chief developer advocate, Burr Sutter, talks about what a relief Agile was for developers trapped under a waterfall. So the concept of Agile fundamentally resonated

with people because it basically said, look, we focus on people over processes. We focus on interactions and collaboration over tools and documentation. We value working software above all else. And we'd rather people work in small batches and be highly interactive and highly iterative. For some, this developer's revolution went too far. Agile could even be seen as legitimizing an irresponsible hacker mindset. One of the most important voices that spoke out against Agile early on was Steve Rakitin.

He's a software engineer with over 40 years' experience in the industry. When he finished college, Rakitin went to work building the first digital control system for nuclear power plants. And for decades, he's kept on working on power plant software and software for medical devices, safety-critical software. So, yeah, you can imagine that this is a guy not so interested in a

fly-by-the-seat-of-your-pants approach. And so when Agile came out at the tail end of the methodology wars, Rakuten sort of rolled his eyes. It was like, well, here we go again. Another bunch of people sat around drinking beer and coming up with, you know, other ideas for developing software, many of which, by the way, had already been developed and used in some of these earlier methods. He wasn't wrong about giving them side eyes either. You can track back agile-ish philosophies decades before the Snowbird Summit. For example, lean workflow methodologies like Kanban go all the way back to the 1940s, when Toyota got inspired by shelf-stocking techniques at supermarkets. Their philosophy for lean manufacturing

ended up being repurposed for software development. Rakuten had a whole other concern, too. I was very skeptical when this manifesto was published because it basically came across as a way to allow software engineers to spend more time writing code, less time figuring out what needs to be done, and a lot less time documenting anything. For Rakuten, this was about more than coming up with new workflow ideas. It was about the integrity of his profession. For a long time, software engineering was not viewed as a legitimate engineering discipline, like electrical engineering and all the other engineering disciplines. And in my opinion, part of the reason was because there was a general lack of accepted practices that software engineers used.

As we got through the decade of the 90s and we started identifying some of these processes, it seemed like some of them were actually starting to take hold, and many of them made a lot of sense. Then comes along the manifesto. If software engineering is ever going to become a legitimate engineering discipline, you need process. Every other engineering discipline has processes, why not software? I'm Saran Yitbarek, and you're listening to Command Line Heroes, an original podcast from Red Hat. So if we leave aside people working at nuclear power plants and focus instead on the larger corporate world, we find that Agile has, bit by bit, taken over. But not without a little corporate resistance, naturally.

I think the greatest resistance that we tend to see in Agile adoption comes from senior and middle management. This is Daryl Rigby, a partner at Bain & Company. They've been experimenting with using Agile in software development companies, but also product development, new service development, advertising programs, loyalty programs. And everywhere they go, there's a potential for the managers to get a little nervous.

It changes the very notion of how they believe they add value because they're moving away from micromanaging or micrometalling with these teams to empowering them and coaching them. Now, Agile is no guarantee against micrometallers. And I admit, the first time I saw an Agile management board, I thought it was just this never-ending to-do list. A bit overwhelming. But then I started actually using an agile project management tool, and I was blown away. I was new out of a coding boot camp, and I was trying to figure out how to prioritize features and make product decisions. That scary-looking tool forced me to organize all these ideas and give them names, order, and structure. It helped me manage

my project. So I do take Rigby's point. Some people might look at the effect of tools like that and think, if Agile empowers developers, then it must disempower their managers. But this was larger than any one job title. Agile's forward momentum was growing, and more importantly, Agile was proving itself. Agile has been used by tens of thousands of teams at this point, so we've got a lot of great data on where Agile can be used. And the answer is, any time you're thinking about doing innovation, an Agile team can probably do it better than the way you're doing it today. There are a lot of bigger, well-known companies that have transformed themselves. Amazon is a big user of agile approaches, Netflix, Facebook, Salesforce,

all of them heavy users of agile. And it has really not just redefined the way they work, but the way the industry works. When Rigby first heard about Agile, he thought it was just a weird language. He was working with the IT departments at a lot of big retailers, and he kept hearing them talk about time boxes and sprints and scrum masters. At first, he didn't have a clue what they were talking about. He told me he actually tried to ignore any mention of Agile like it was another language he didn't have to learn. After all, he wasn't a developer himself. Today, he's such a believer that he's

literally brought Agile to his home and into his church. I do not necessarily sit down with my family every morning and have a scrum meeting with them. But I have become very good at prioritizing the things that I'm going to do. Agile has gone from fringe to mainstream in just over a decade, but the corporate assimilation came at a cost. And in some cases, that assimilation even meant bastardizing the manifesto's original intentions. Dave Thomas reminded me of that. He says that when he and those 16 other snowbird guys first wrote it, there was no real

prescription at all. So even though the manifesto doesn't tell you how to apply the values, I'm assuming you had some idea of what you thought would happen or what people would do with it. Honestly, I did not. This might surprise you. Agile can seem so prescriptive today. There's a whole marketplace of books, certifications, tools, courses, and products that show you how to, quote, do Agile. But despite the thousands of manuals and professionals who want to show you the one true way, Dave Thomas says they're really missing the whole point. It's a set of values. It's like the golden rule, I guess. It's like, you know, if you're about to do something evil and vicious, you think, well, okay, how would I like it if someone did that to me? You know, the golden

rule applies. Well, it's the same with the Agile Manifesto, right? It's not telling you what to do and what not to do. It's just telling you how to assess whether or not what you do is in line with that kind of way of doing things. Yeah. And I think that just going back to that name of Manifesto for Agile Software Development, the one word that really stands out and that has been very persistent and people really latched onto is the word agile. What is wrong with the use of the word agile today? The problem with the word agile is that in the title that we came up with, it is an adjective that describes software development. But what happens then is that people say, well, how do I do this agile thing?

Suddenly out of the woodwork spring, a whole army of consultants and consultancies who look at the success of XP, look at the success of the manifesto and say, hey, there's gold in them, there are hills. And so they start telling people how to, quote, do agile, unquote. And that is a problem because you can't do agile. Agile is not what you do. It's how you do it. And yet there are companies out there who will happily sell you an Agile in a box. That, I think, is a travesty. The consultancies that go into a Fortune 1000 company and help them set up, quote, Agile, they're walking away with $5 million. Great, good for them. But the reality is that that's like telling a tiger to be agile by saying, take seven steps and then spring off your left foot and then take two more steps and spring off your right foot.

Well, that only works if the gazelle is doing the same thing. And guess what? No one's told the gazelle to be agile that way. And it's basically running off to the horizon, laughing have is agility because they have been boxed into a particular path. Management is going to be judging them based on how well they follow those principles or those procedures, not on how well they're developing software. So looking back, when you think about the role of the developer before the manifesto and then now after the manifesto, how has the role of the developer changed or expanded because of what you wrote? To their credit, I think that the majority of programmers out there get it. I think that the

manifesto has empowered a lot of developers to start following practices that to some extent they knew they should have been doing, but they never really had the authority to do. Things like testing, for example, gathering feedback, short iterations. So in many ways, the job is more interesting and more fulfilling. I think also programmers are feeling a little bit more scared because now they have responsibility. In the old days, I was just following orders. Why doesn't this program work? Well, I followed the spec, whereas nowadays it's on your shoulders.

So I think the job has grown up a bit because of the manifesto, and I think people are beginning to realize that they have an end-to-end responsibility for what they deliver. The success of Agile has been so pervasive that it's altering workflow and attitudes far beyond the development world, certainly beyond the Snowbird Lodge. We have to ask, what does it even mean to be an Agile developer today versus 2001 when the manifesto was written? Does the original spirit of Agile persist? And if it did change, is that such a bad thing?

For Ruha Devanason, a diversity business partner at Google, the Agile mindset might have evolved to the point where it's now influencing basic levels of fairness and workplace equality. Part of what makes teams inclusive is their ability to evaluate and reflect on how they work together on a very fundamental level. And most teams, when they work together, they don't get the space to stop and think about their team dynamics, about whether everyone's having a voice at the table, about whether someone's steamrolling someone else, or whether someone is silent the entire time. And if they are silent, why are they silent? And so when thinking about inclusion, I thought that some of the tools that Agile teams use could be very useful in giving teams a structure or a framework to be more inclusive. So diversity, not just in terms of gender and race and ethnicity, but also functional diversity. And functional diversity introduces complexity into

a team. But let's make a distinction here. Ruha is not saying that Agile equals diversity. She's saying Agile plus diversity equals better teams. Ruha's ideas were crystallized in an article she wrote called, Can Agile Methodology Unlock Diversity? We'll throw a link in our show notes. It's worth a read. In it, she walks you through this idea that diversity isn't just a fuzzy concept your HR department keeps talking about. It's actually a strong business case here. Diversity can dovetail with Agile. So that introduction of complexity for the end goal of having an outcome or a product that has been looked at from many

angles. That's the same fundamental point of view that we take when we say adding diversity to a team leads to a good outcome, leads to more innovation and more creativity. Because when you have multiple perspectives looking at a problem, helping to do problem-solving work, you're more likely to come out with an outcome that is more robust. Even something as simple as a daily stand-up, where everybody on your team gets to report, is going to give voice to introverts or other people who have a hard time being heard. The thing I really like about Agile is it has some built-in mechanisms to help teams

stop and reflect. And that may be because Agile is so quick. And with two-week sprints, if you don't build in those mechanisms, you're likely to go way off track and not have the consciousness to come back onto track. Reflection on how we're working together, how can we course correct, is one of the fundamental ways in which teams can be inclusive. Since we're talking about inclusivity, now might be a good time to point out that those 17 founders of the Agile Manifesto, yeah, they were all white men. There was zero diversity in that room.

And that is a very common criticism of the group and one that I have a lot of sympathy with. If the manifesto founders used those agile principles and applied it to their own meeting, they may have gotten partway done and asked themselves, hey, did you notice we didn't invite any women to this meeting? I wonder if a person of color would have a different opinion. People's friends tend to be and look like they do. So if the first person that thought about the Agile Manifesto was a white guy, it's not surprising that the people he invited to the table were also white guys. We have an opportunity to stop and say, let's take a step back. Let's expand our lens and look for people outside of the network that I currently have who can bring a different perspective

and help me elevate this methodology to something even better. It makes sense to me that because Agile is so, well, agile, we can apply it to different problems and industries. The application of Agile and what it looks like in real life is constantly shifting, expanding. I guess it's practicing what it preaches. There is no settled answer, no settled end point. That's something we sometimes forget. Hard rules are the enemy of agility. So if an Agile team ever tells you that part of being Agile means you have to produce a new release every two weeks, or you have to do this,

or you have to do that, then by definition, that's not Agile. The second you say always, you aren't being Agile Alliance. And that alliance became a non-profit. And that non-profit started hosting a conference every year. It's grown and grown, spawning whole new theories and methodologies. And here's something I think is super interesting. At the 2008 conference, Belgian developer Patrick Dubois attends and sets off down a path that leads him to inventing a whole new practice of software development.

DevOps. I'd never thought that Agile, a set of principles, and DevOps, a whole industry, were related. But now I'm thinking, how strong really is that line between the rise of Agile and the invention of DevOps? Did the one breakthrough lead to the other organically? We're going to find out together, because our next episode is all DevOps, all episode long. And it's dropping in two weeks. make sure to subscribe to the show. Just search for Command Line Heroes in Apple Podcasts, Spotify, Google Play, CastBox, or however you get your podcasts. Then hit subscribe,

so you'll be the first to know when new episodes are available. I'm Saranya Tbarek. Thanks for listening, and keep on coding. Thank you. now is foundation models. And those are important, but at Red Hat, we see them as just a piece of the larger AI infrastructure. And here's what I mean by that. Enterprises are built of hundreds or even thousands of applications. It's not hard to imagine a future in which those applications are being served by hundreds or thousands of models. Without a common platform for your data scientists and developers, without a way to simplify some really complex workflows as you

train, tune, serve, and monitor models, it can get overwhelming pretty quickly. And that's why we've built Red Hat OpenShift AI, a platform where everyone is working together on the same page to build and deploy AI models and applications with transparency and control. Find out how at redhat.com.