Screaming in the Cloud - What an “Agilist” Brings to the Engineering Table with Cliff Moon

Episode Date: August 18, 2021

About CliffCliff is an Agile Consultant and self proclaimed “computer botherer.”Links:Agile Manifesto: https://agilemanifesto.orgTwitter: https://twitter.com/moonpolysoft

Hello, and welcome to Screaming in the Cloud, with your host, Chief Cloud Economist at the Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud. This episode is sponsored in part by Cribble Logstream. Cribble Logstream is an observability pipeline that lets you collect, reduce, transform, and route machine data from anywhere to anywhere.

Simple, right? As a nice bonus, it helps you not only improve visibility into what the hell's going on, but also helps you save money almost by accident. Kind of like not putting a whole bunch of vowels and other letters that would be easier to spell into a company name. To learn more, visit Cribble.io. That's C-R-I-B-L dot I-O, and tell them Corey sent you, and wait for the wince. This episode is sponsored in part by Thinkst Canary. This might take a little bit to explain, so bear with me.

I linked against an early version of their tool, CanaryTokens.org, in the very early days of my newsletter. And what it does is relatively simple and straightforward. It winds up embedding credentials, files, or anything else like that that you can generate in various parts of your environment, wherever you want them to live. It gives you fake AWS API credentials, for example. And the only thing that these are empowered to do is alert you whenever someone attempts to use them. It's an awesome approach to detecting breaches. I've used something similar for years myself before I found them. Check them out. But wait, there's more.

Because they also have an enterprise option that you should be very much aware of. Canary.tools. Take a look at this. What it does is provides an enterprise approach to drive these things throughout your entire environment and manage them centrally. You can even get a physical device that hangs out on your network and impersonates whatever you want it to. Whenever it gets NMAP scanned or someone attempts to log into it or access files that it presents on a fake file store, you get instant alerts. It's awesome. If you don't do something like this, instead, you're likely to find out

you've gotten breached the very hard way. So check it out. It's one of those few things that I can look at and say, this is an amazing idea. I am so glad I found them. I love it. Again, those URLs are canarytokens.org and canary.tools. And the first one's free because of course it is. The second one is enterprising. You'll know which one of those you fall into. Take a look. I'm a big fan. More to come from Thinks Canary in the weeks ahead. Welcome to Screaming in the Cloud. I'm Corey Quinn. My guest today has done an awful lot of things over the course of his career. Startup engineering, software work, founded two startups, has been an engineering manager a bunch of places, has been the CTO at UpGuard, for example, and consulted at one point on HBO's Silicon Valley. Also of note, he is now these days a renowned Agile consultant. Cliff Moon, welcome to the show.

Hi, Corey. Thanks for having me. So you and I have had energetic conversations about Agile. And based upon that context, calling you an Agile consultant for enterprises is basically a deadly insult at this point. Let's get some context on that. For those who have not heard of the term because they live wonderful, blessed lives, what is Agile? A lot of people talk about it, but always the presupposition that people listening know what it is. Yeah, that's a great place to start. So let's go back into sort of prehistory, right? What we call agile today is, I guess, several generations removed from the original thoughts of agile, right? So in case folks aren't aware to kind of lay the

background, like there was a group of software developers, I think it was like in two might have been in 2000 might even have been earlier than that, who came up with what they call the Agile Manifesto. And I'm not going to go through it point by point, one, because I don't remember it, two, it's not germane. But it's called a manifesto. I mean, if you take a look at things that have been written historically that are called manifestos, very few of them are good. Like generally a manifesto sounds like something you wind up writing in a cabin somewhere right before you wind up doing some sort of horrible crime that winds up living in infamy for 30 years. Yeah, yeah.

Manifestos, they get a bad rap for good reason. Anyway, let's not go down that rabbit hole. But, you know, yeah, so Agile Manifesto, right? Basically, it was this group of people. They said, hey, you know, we don't think we're developing software the right way. This is unnecessarily painful. We're doing things in kind of a silly fashion. Like let's refocus it around, you know, the customer let's do this yada, yada, yada. And again, like, like you're saying, like it's a manifesto, it's not very prescriptive about what to do to solve the

problem. It really just kind of points out problems and then gives a bunch of vague statements about here's the things we should value or whatever. Right. And so again, like we're saying, like as a manifesto, it kind of mutates from there. And then everyone kind of agrees. They say, yep, this is the wrong way. Let's try a better way. And, you know, down through the line, what ended up happening was a lot of people figured out that they can make money doing this and make money being an agile consultant or an agilist if you're if if you like that title uh you know so so it's people who come in and i guess like try to teach you how to do agile the right way and the problem is is that the right way

usually ends up being uh scrum and so scrum if we can get that, is this idea of like having a two-week sprint. You plan out the work you're going to do for that sprint. There's a bunch of meetings you have to do that are kind of mandatory. You do a stand-up every day. You do a retrospective. You do sprint planning, et cetera, et cetera. And so it's this like cake in a box, right? So it's like a ready-made thing and like, ta-da, now we have Agile. We've implemented Agile. We've done this Scrum thing. The problem, of course, is that like most people, when they implement this, and certainly not the Agile consultant in most cases,

because they're basically there to just bake the cake and then keep soaking hours until they're forced to leave. The problem that I had whenever I wound up dealing with, I guess, Agile consultants in large enterprises, they always looked a lot like Agile trainers. And I don't know what they were charging because it doesn't matter what they were charging. They wound up gathering the entire engineering department in part of the building for two or three days to talk to them about how tickets worked and how planning worked and how to iterate forward and how to wind up planning for spikes and all the various terminology and how to

work with different tooling and the rest. And the reason it doesn't matter what these people cost was because it was absolutely dwarfed by the sheer cost of every engineer in the company sitting through this nonsense for the better part of a week. Oh, absolutely. And then it's an ongoing thing, right? Well, it's supposed to be an ongoing thing.

Yeah, it's an ongoing concern. You end up having all of these meetings that you have to do every two weeks or God forbid every one week or whatever the iteration speed that they've laid out is. And the thing that kind of gets lost in the sauce here is why are you doing this? What is the point of all of this? And I think one of my favorite things to do, you know, if I'm at a company that's like they're implementing scrum and they've got their sprints and then we have a retrospective, one of the things I love to do is I love to kind of touch the third wire during retrospective because that's when you're supposed to bring up, hey, what can we do better? You know,

what could have gone well? And that's when I like to say, hey, can we just not do this? And the response that I get is usually an indication of how hard of a job I'm going to have of, you know, sort of trying to deprogram people. Cause when it ends up being, is that, you know, and especially if you have an agile consultant, an agile teacher or whatever, they're not going to like the sound of that at all. Right. It's like, why are we doing any of this? What is the point? And when you dig into it, you know, especially when we talk about scrum, it sort of confuses a bunch of different goals that a lot of companies don't even necessarily have anymore, right? Like one of

the tenets of Scrum is that every two weeks or whatever your iteration speed is, whether it's one week, two weeks, whatever, the whole system has to be shippable, right? So that means that everything has to work together correctly. And then you can ship an entire like vertical or monolith or whatever the problem with this is that like especially related to if people are deploying to the cloud or if they're running some sort of sas service this is a meaningless statement right like the way that people develop software in that arena today is things get shipped immediately the system is always shippable because the system is always up because prod is always up. And so you ship your component, everything is backwards compatible, and then

your features are behind a feature flag. So the idea that like, oh, everything has to be like set in stone on a two-week cycle or whatever, doesn't mean anything anymore. Unless, of course, you have a physical artifact that you actually send to a customer, like a CD or an image to download or something, right? But if you're doing cloud-based software... Or a giant rocket. Yeah, or a giant rocket, yes. Oh, God. For some things, you're always using waterfall. It's like giant rockets going to space, or more realistically for most of us, and more prosaic at least, is billing systems. People don't generally tend to iterate forward on things that charge customer credit cards. It's a lot of planning, a lot of testing, and they roll it out and everyone sweats bullets for a while. Right. And I would submit

that, at least in my experience, most companies which have tried to implement a scrum type process, what they're actually doing is they're running a two-week waterfall. Because a lot of times they've got a lot of technical debt. So the idea that you can ship things immediately might be a little bit shaky. And so what you know, what you end up doing is you have this iteration speed of like two weeks, and then you have to plan everything out for that. And then you have to go through testing, QA, you know, acceptance. The whole cycle has to run in a two-week sprint. And it truly is a sprint in that case, because it's too much work, it's too much stuff, and everything just kind of falls apart. And then they wonder why they can't ship any software anymore. Well, it's because you adopted this process. Oh, I've been in environments where we'll sit

down and do quarterly or, God forbid, annual planning about what we're going to build this year via Agile. It seems a little unlike what Agile professes to be. Now, other than the sheer aspect of hypocrisy surrounding all of this, you take it a step further and say that it in many ways causes harm. Yeah, it causes harm in a couple of different ways. One of the ways that I think it is most harmful is the effect that it has on junior engineers. So people who are just starting their career, folks who are just coming out of college, and in most colleges, they don't really teach you sort of software engineering processes or software engineering practices beyond the nuts and bolts of the code or like the theory of the code or whatever. Right.

But they don't teach you like how to work in a professional environment. And so then you get a lot of folks just entering their first job and they learn sort of the way to do things at that job. And then they go on, they move to another job. And someone might have, you know, they might go through 10 different companies in their career, maybe some more, but they learn a certain way of doing things. And then all of a sudden, it's like, yeah, I know how to do Agile. We did it at company X, Y, and Z. And then they sort of cargo cult it and take it to the next place if they don't already have it.

So it's this sort of inculcation of younger engineers into this way of doing things that is completely harmful because most places, you know, they don't sit you down and tell you why we're doing this, right? Because they don't necessarily know why we're doing it either, right? Like I said, a two-week sprint with Scrum, the system is shippable every two weeks, you have to go through testing and yada, yada, yada. This may actually make sense in some cases. And professionally, like in my experience, I've designed certain processes that are similar to that longer timeframes, but they were like kind of designed towards both the product and the team and sort of the interval that they had to ship on. But in most places I've been, no one's thinking about it from that perspective. They're not

thinking, uh, we have to design our processes around the software or our customers or whatever. They just kind of do either whatever the Agile consultant tells them or whatever they learn at the last place. And so it sort of has this effect of replicating a sort of a cargo cult mentality throughout the entire industry, which is sad. I've talked to a number of relatively junior folks who have not heard of Agile or Scrum or any of these higher level concepts about software development methodologies. They just

walked into the workplace one day and everyone's doing two weeks planning sessions. I've had people ask me six months into their career, why is it called a sprint? Or what is up with the swim lane style things? It seems weird, but everyone I talk to is used to it. Is this just, is it this company thing? Or is this an industry thing? And on some level, it's no, it's just a terrible thing. It's sort of like a mind virus that wind up kind of took root in an awful lot of people's minds.

Yeah, absolutely. And so when we talk about like the damage being done, I think that's the worst, right? When you think about new people getting into the industry, having a fresh perspective and that perspective or sort of having an opportunity to forge a new way to do things, that kind of gets ground out of them immediately.

And they have to do things this set way. And this especially goes for people who end up at large companies where it's just like, you're not gonna change anything. You're gonna get in there and you're gonna do it their way and then that's it. In the rare case when someone comes out of college or comes out of a training program and then they go to a startup that doesn't have as much structure. Those are really the only sorts of areas where you even have an opportunity to innovate in terms of the process of how we develop software, because otherwise it's just set in stone by this point.

So you're given a blank slate or a blank white whiteboard as a case, maybe, or God forbid, a blank Jira board. How would you structure it instead? How would you advise companies to think about software development? Since I think it's pretty clear that an awful lot of what they're doing today either isn't working or is some weird bastardized hybrid of different methodologies that doesn't really have a name other than something cynical like scrum bot? Yeah, so that's a great question. So I think where I'll sort of take this is I can talk a little bit about, I mean, I've done this before,

right? So I've been hired into several different startups as either like an engineering manager or like a director or, you know, basically like hired management. And typically when a startup hires an engineering manager or someone on that management chain, they only really do so when the pain has gotten so bad that they want to throw money at the problem, right? So I sort of specialized in that for a little bit. Very thankless job, but it was interesting because what happens is that every team fails in its own unique and beautiful sort of way. So one of the first places where I did this, there's a person running product. He had learned his Agile methodology from being at

Booz Allen Hamilton, which, I mean, it is a nightmare factory and every metric you can sort of measure it on. But apparently they specialize in agile as well as the military industrial complex. So great. He was running things on a one-week sprint and it was a shippable system, right? So it had a cloud component, but it also had a component that was forward deployed into a customer network. So he was running this where basically everyone would work on a one-week sprint. They would then do like a bug bash every Friday. And it was very much a case of like,

you keep doing the same thing and you keep getting the same results and you keep doing it to see if you get different results. And they were very much in that kind of mode, right? So they would do this every single week. You would have a bug bash where the same bugs came up every single time. They wouldn't get fixed. No one would triage them. So the same bug was, you know, in the system and maybe like 10 or 15 different tickets. No one was triaging it. And it was just, it was just a mess. And so when I got there, like part of my job was to just kind of break apart this crazy structure that was happening. And again, try to design something that would actually work again for the product. You have to design something that has to work for like how the product gets deployed. So as I said earlier, like if you have cloud services,

they can deploy whenever, right? So like structuring them around some sort of timeframe doesn't really make a ton of sense. However, when you have something that gets deployed into a customer network, like a, like an agent or some kind of desktop software or anything that's sort of on machines that you do not have direct control over, you have to factor that into the speed at which you ship. You have to factor that into your engineering process. Because if you can only ship out that executable once every quarter, it's like, how fast can your customer actually consume these things, right? Most places, if you give them updates every two weeks, they're going to say, what are you doing? Why are you making my life hard? And a lot of places, you know, the fastest, especially if you're selling to an enterprise, the fastest they can consume a sort of a forward deployed component

is once a month at the very fastest. Usually they prefer on a quarterly or even a six-month basis. But if that's the case, you have to design your engineering process to account for that. Then the other part is that when you land at a place like this, you can't just pull the rug out from everybody immediately. It's similar to saying like, oh, we got to do a rewrite. It's like, well, you can't just do a rewrite of your engineering process either. You have to incrementally make changes to it so that people are not confused about what they're supposed to be doing, but you're making changes towards things running in a more smooth fashion. So what I typically try to do is I try to design a process and then get the team bought into it

and then hopefully get them moving faster. Now, the first time I tried this, it was a disaster. That was the company I was just talking about where they were running one week sprints. I did not know what I was doing at the time. That was a very difficult situation. Landed at another place after that, where this one was a two week scrum, similar problems around like, okay, frequency of testing, you have a component that gets deployed into a customer network. How fast can we deploy that? You know, and similar sorts of problems. And so now that I could kind of see what the pattern was, I could now develop a, I had a much better time developing a process that actually worked and help the team ship with confidence, which is, you know, really what you want the process to do is you want the process to be something that takes burden off of the engineering team,

as opposed to something that makes your off of the engineering team, as opposed to something that makes your job as the engineering manager easier, which I think a lot of engineering managers approach it from that perspective of like, oh, I can get a report out to JIRA and then I don't have to talk to everybody every day or whatever, right? If you're trying to make your job easier through the process, you are necessarily putting more burden onto your team. Your company might be stuck in the middle of a DevOps evolution without even realizing it. Lucky you. Does your company culture discourage risk? Are you willing to admit it? Does your team have clear responsibilities? Depends on who you ask.

Are you struggling to get buy-in on DevOps practices? Well, download the 2021 State of DevOps Report, brought to you annually by Puppet since 2011, to explore the trends and blockers keeping mid-evolution firms stuck in the middle of their DevOps evolution because they fail to evolve or die like dinosaurs, the significance of organizational buy-in, and oh, it is significant indeed, and why team identities and interaction models matter. Not to mention whether the use of automation and cloud translate to DevOps success. All that and more awaits you. Visit www.puppet.com to download your copy of the report now.

It feels like this is almost the early version of a similar political machination playing out, where we see it now with, there are these large companies that once upon a time had these big monorepos, and they had 5,000 developers, and everyone wound up causing problems because a group of developers collectively referred to as a merge conflict. Then they wound up building out, ah, we're going to break the monolith apart into microservices, and it solves that political problem super well. And then you wind up with a bunch of startups with five engineers working there, and they have 600 microservices running in their environment. And it feels like someone took a idea outside of the context in which it was designed

for and applied it to a bunch of inappropriate areas and just bred an awful lot of complexity while actively making everything worse. Please don't email me if people disagreed with that statement. But it feels like an echo of that, doesn't it? Oh, absolutely. Yeah, I mean, I think a lot of this is sort of a reflection of our relative infancy as an industry, right? When you think about the amount of time software engineering has been around and has been a going concern of itself, as opposed to other engineering disciplines, I think we are still very much in our infancy. Like I was saying, like, like they don't really teach this sort of stuff in school and certainly not the theory behind why you would structure

things this way versus that way. In fact, most people who get promoted as a manager, you get promoted from being an engineer, someone who codes all day, or codes and does design work, but basically someone who works as an individual contributor. Then you get promoted to being a manager, and very few places give you any training or education or anything at all about how do you even do this job. And so you either sink or swim. It's an orthogonal skill set that basically bears little relation to what you were doing before. Exactly. The thing that sort of gives you any sort of ability to swim in that type of job is, you know, having sort of the clout or like, you know, the respect of your former peers as you get

promoted into that. And the people who do well at that, they basically learn on the job and rely on that inbuilt respect to basically screw up a lot until they can get the hang of it. But yeah, most of the time you don't get an education in management or any of the other things that are not just specific to people management, but like people management for software engineering, which I do think is its own sort of discipline. On some, I guess, almost borderline ridiculous level, it feels like no one really knows what they're doing when it comes to management, especially in engineering. In other disciplines, it seems that management is treated as a distinct key skill, but very often the way that my management strategy evolved, and people think I'm kidding whenever I say this, but I assure you I'm not. It came out of looking at what my terrible managers had done in the past and what didn't work for me and what made me quit slash become demoralized slash convince others to quit, et cetera, et cetera, or, you know, steal office supplies, whatever it is that you act out. And then I just did the exact opposite of those things. And I've been told repeatedly, wow, you're a great manager. Not really. I just don't do all the things I hated.

It gets you surprisingly far. Well, yeah, absolutely. But that gets you far with your own reports. There's a whole other side to being a manager, which is dealing with the outside world. And then that's, you know, especially if you're in a large organization, even in some smaller ones as well, right? Like there's a whole dimension to the job that you as an individual engineer, you don't even see. That's the politics part of the job about like, how do you justify what you're doing? How do you advocate for your team? How do you operate as a quote unquote shit umbrella for your team? And all these sorts of other things where you provide a safe harbor within the company for your team to all these sorts of other things where you provide a safe harbor within the company for your team to operate and then try to procure resources and make sure that

the decision makers above you understand the importance of what you're doing. And no one teaches you how to do that. Oh, never. You're absolutely right on this. I was mostly focused on managing my reports. I completely failed in those roles, managing up, and in some cases, managing sideways as well, just because that was never clear to me when I was an independent contributor working on engineering problems. It's an evolution on some level of figuring out what it is that the role really is. And all this stuff is not that complicated to teach people, but for some reason, culturally, we don't do it. We take the hacker news approach to things and try and figure out complex forms of interaction

from first principles. And it really feels like there are some giants around upon whose shoulders we could stand. Yeah, I mean, I agree with that. I mean, there's definitely people in the industry who've written books and who are, you know, starting to try to put down that first layer of institutional knowledge to kind of share with other folks. You know, you got people like Camille Fournier and other folks who've written books specifically for like engineering managers who work in the software world, which I think is a really great sort of first step. But yeah, when it comes down to it, it's like, OK, we're going to implement this process. We're going to do these things. I don't know why. It's almost like, you know, no one got fired for buying IBM. No one got fired as an engineering manager for implementing Scrum. But if you try to go and do some other weird stuff, you're running the risk of getting fired if you fail.

There is the question of whether someone at IBM will get fired for buying Red Hat, but that's not the analogy that people always fall back on for the last 25 years. I think that there's also the idea that people will try and build their own thing where it makes sense for them And that can lead to a lot of, frankly, disastrous outcomes in some level. I feel like this does tie into the, I guess, almost unthinking adoption of agile and similar methodologies or perversions of those methodologies in many large enterprises. Do you see a fix for this? Or is this something that we all more or less have to live with and watch people continue to make the same mistakes for another 10 years? I think for the most part, you have to, I guess, change starts at home.

What I would advocate for is that if you have problems or qualms with the process that your particular organization is following and you have ways you think you could fix it or changes that you'd want to make to it, then, you know, start advocating for those. And you'd be surprised about how far you can get sometimes with just saying like, Hey, can we just stop doing this? Or can we do this a different way? But I would also say that like, one of the things you just said sort of knocked something loose from my mind, which is that like, even when companies kind of share like, Oh, we've done something amazing here, right? We've designed this, you know, amazing new process. It really works well for us. And they write a big blog post about it. Turns out if anyone ever follows up on that, they either never did it or it was never as described and they certainly don't do it today. Right. So I think a good example of this would be like when Spotify put out their, this was a number of years ago, Spotify put out their big creed about like, here's how Spotify develops software. And they had this whole bespoke thing about, you know, they've got these pods of people and you've got a matrix management and they reinvented a whole bunch of

stuff and then you talk to anyone who was at spotify during that time and they're like yeah we tried that it didn't work but they still put out the blog post so and i think it's still up and hasn't been taken down yet it's yeah work for other people? No, absolutely not. But it might work for us. Yeah, it's the same kind of trick that companies do with open source, which is like, you open source something to a bunch of fanfare and try to get people to adopt it when it hasn't even been adopted internally.

And anyone who tries figures out it's not the right thing, and they don't even like it. But it's like, oh, yeah, we can open source it. And then it comes with the imprimatur of whatever company it comes from. I mean, this is a pretty classic joke. You know, it's like that old movie, the gods must be crazy. You know, you throw the Coke bottle out of the plane, someone on the ground picks it up and eventually ruins their life. Even though it's just a Coke bottle. Same thing with open source, same thing with management processes. It seems like it's going to be one of those areas that continues to evolve,

whether we want it to or not, or at least I hope because the failure is it doesn't. Yeah. I mean, hopefully it evolves. And like I said, I would say, you know, change starts at home. Try to advocate for changes, you know, on your own team and think outside of the box, like try to figure out what you can kind of get away with and try to figure out, I guess, you know, on your own team and think outside of the box, like try to figure out what you can kind of get away with and try to figure out, I guess, you know, ways to sort of break down the, the walls and the rituals that the agile consultants have set up. I hope you're right. If people want to hear more about your thoughts on these and many other matters, where can they find you? Yeah. So typically I'm just usually tweeting. So my Twitter account is at Moon Polysoft.

And that's usually where I'm doing most of my stuff. Yeah. And we will, of course, include a link to that in the show notes. Thank you so much for taking the time to chat with me today. I really appreciate it. Yeah, Corey, it was great. And thanks for having me. Cliff Moon, absolutely everything except an agile consultant.

I'm cloud economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice. Whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with a comment that you're going to continue to iterate on and update every two weeks, like clockwork. If your AWS bill keeps rising and your blood pressure is doing the same, then you need the Duck Bill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duck Bill Group works for you, not AWS.

We tailor recommendations to your business, and we get to the point. Visit duckbillgroup.com to get started. this has been a humble pod production stay humble you you you you you you you you you you you you you you you you you you you you you you you you you you you you