Screaming in the Cloud - Innovations and the Changing DevOps Tides of Tech with Nigel Kersten

Episode Date: July 22, 2021

About NigelNigel Kersten’s day job is Field CTO at Puppet where he leads a group of engineers who work with Puppet’s largest customers on cultural and organizational changes necessary for large-scale DevOps implementations - among other things. He’s a co-author of the industry-leading State Of DevOps Report and likes to evenly talk about what went right with DevOps and what went wrong based on this research and his experience in the field. He’s held multiple positions at Puppet across product and engineering and came to Puppet from the Google SRE organization, where he was responsible for one of the largest Puppet deployments in the world.  Nigel is passionate about behavioral economics, electronic music, synthesizers, and Test cricket. Ask him about late-stage capitalism, and shoes.Links:Puppet: https://puppet.com2020 State of DevOps Report: https://puppet.com/resources/report/2020-state-of-devops-report/

Hello, and welcome to Screaming in the Cloud, with your host, Chief Cloud Economist at the Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud. This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production.

I'm going to just guess that it's awful because it's always awful. No one loves their deployment process. What if launching new features didn't require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren't what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you and watch for the wits. Your company might be stuck in the middle of a DevOps evolution without even realizing it. Lucky you!

Does your company culture discourage risk? Are you willing to admit it? Does your team have clear responsibilities? Depends on who you ask. Are you struggling to get buy-in on DevOps practices? Well, download the 2021 State of DevOps Report, brought to you annually by Puppet since 2011, to explore the trends and blockers keeping mid-evolution firms stuck in the middle of their DevOps evolution because they fail to evolve or die like dinosaurs, the significance of organizational buy-in, and oh,

it is significant indeed, and why team identities and interaction models matter. Not to mention whether the use of automation and cloud translate to DevOps success. All that and more awaits you. Visit www.puppet.com to download your copy of the report now. Welcome to Screaming in the Cloud. I'm Corey Quinn. This promoted episode is sponsored by a long time, I wouldn't even say friend, so much as antagonist slash protagonist slash symbiotic company with things I have done as I have staggered through the ecosystem.

There's a lot of fingers of blame that I can point throughout the course of my career at different instances, different companies, different companies, different clients, et cetera, et cetera, that have shaped me into the monstrosity that I am today. But far and away, the company that had the most impact on the way that I speak publicly is Puppet. Here to accept the recrimination for what I've become and how it's played out is Nigel Kirsten, a field CTO at Puppet, or the field CTO. I don't know how many of them they have. Nigel, welcome to the show, and how unique are you? Thank you, Corey.

Well, you know, reasonably unique. I think you get used to being, you know, one of the few Australians living in Portland who's decided to move away from the sunny beaches and live in the grey wilderness of the Pacific Northwest. So to give a little context into that ridiculous intro, I was a traveling contract trainer for the Puppet Fundamentals course for an entire summer back in, I want to say, 2014, but don't hold me to it. And it turns out that when you're teaching a whole bunch of students who have paid, in many cases, a couple thousand dollars out of pocket to learn a new software, where in some cases they feel like it's they're taking their job away because they view their job, rightly or wrongly, as running

the same script again and again. And then the demo breaks and people are angry. And if you don't get a good enough rating, you're not invited to continue. And then the company you're contracting through hits you with a stick. It teaches you to improvise super quickly. So I wasn't kidding when I said that Puppet was in many ways responsible for the way that I give talks now. So what do you have to say for yourself? Well, I have to say, you know, congratulations for surviving opinionated defensive nerds who think not only you, but your entire product, your demo, could be replaced by a shell script. It's a tough crowd. It was an experience. And some of these were community based and some of them were internal to a specific company. And if people have heard more than one episode of this show, I'm sure they can imagine how that went. I gave a training at Comcast once

and set a personal challenge for myself of how many times could I use the word Comcastic in a three-day training? And I would work it in and talk about things like the schedule parameter in Puppet, where it doesn't guarantee something's going to execute in a time window. It's the only time it may happen if it doesn't fire off and then it isn't going to happen. It's like a Comcast service appointment. And then they just all kind of stared at me for a while and credit were due. That was the best user rating I ever got from people sitting through one of my trainings. So thanks for teaching me how to improv at basically what could have been a very expensive mistake on Puppet's part. It accidentally worked out for everyone. Brilliant. Brilliant. Yes,

you would have survived teaching the spaceship operator to that sort of a crowd. Oh, I mostly avoided that thing. That was an advanced puppetism, and this was puppet fundamentals, because I just need to be topically good at things, not deep dive good at things. But let's dig into that a little bit. For those who have not had the pleasure of working with Puppet, what is it? Sure. So Puppet is a pretty simple DSL. You know, DSLs aren't necessarily in favor these days. Domain Specific Language, for those who have not caught up on that acronym. Yes. So, a programming language designed for a specific task.

And you know, instead we've decided that, you know, the world will rest on YAML and we've absorbed a fair bit of YAML into our ecosystem. But there are things that I will still stand by, are just better to do in a programming language. If X, then Y, for example, is just easier to express when you have actual syntax around you and you're not sort of forcing everything to be in a data specification language. So Puppet's pretty simple in that it's a language that lets you describe the state that infrastructure should be in. And you can do this in a modular and composable way. So I can build a little chunk of automation code, hand it to Corey.

Corey can build something slightly bigger with it, hand it to someone else. And really this sort of collaboration is one of the reasons why Puppet's sort of been at the center of the DevOps movement, which at its core is not really about tools. It's about reducing friction between different groups. Back when I was doing my traveling training shtick,

I found that I had to figure out a way to describe what puppet did to folks who were not deep in the space and the analogy that i came up with that i was particularly partial to was imagine you get a brand new laptop well what do you do with it you install your user account and go through the setup you install the programs that you use some which have licenses on it you You copy your data onto it. You make sure that certain programs always run on startup because that's the way that you work with these things. You install Firefox because that's the browser of choice that you go with, etc. Now imagine having to do that for, instead of one computer, a thousand of them, and instead

of a laptop, their servers. And that is directionally what Puppet does. Absolutely. This is the one I use for my mother as well. Like I was working around Puppet for years before. And I used, the way I explained it was, you know, when you get a new iPad, you've got to set up your Facebook account and your email. Imagine you had 10,000 of these.

And she was like, and I was like, you know, companies like Google, company like Big Banks, they all have lots and lots and lots of computers. And she was like, they run all those things on iPads? And I was like, this is not really where my analogy was going, but. Right, and increasingly though, it seems like the world has shifted in some direction where when you explain that to your mother

and she comes back with, well, wouldn't they just put the application into Docker and be done with it? Oh dear. But that seems to be in many ways that the direction that the zeitgeist has moved in, whether or not that is the reality in many environments, where when you're just deploying containers everywhere through the miracle of Kubernetes, if you'll pardon the dismissive scorn there, that you just package up your application,

shove it into a container, and then hurl it from the application team over to the operations team like a dead dog cast into your neighbor's yard for him to worry about. And then it just sort of takes up the space of you don't have to manage state anymore because everything is mostly stateless in theory. How have you seen it play out in practice in the last five years? Well, I mean, that's a real trend. And, you know, like the size of a container should be smaller than an operating system. And the reality is, you know, like I'm a sysadmin, I love operating systems. I nerded out on operating systems. They're a necessary evil. They're terrible, terrible things. Registry keys, config files, you know, like they're a pain in the neck to deal with. And if you look at, I think what a lot of operations folks missed about Docker when it started was that it didn't make their life better. It was worse. It was like

this actual sort of terrible tool chain where you sort of tied together all these different things. But really importantly, what it did is it put control into the hands of the developers. And it was the developers who were trying to do stuff, who were trying to shift applications. And I think Docker is a really great technology in the sense of, you know, developers could ship value on their own. And that was the huge, huge leveling up. You know, it wasn't the interface, it wasn't the user experience, wasn't all these things. It was just that the control got taken away from the IT trolls in their basement going, no, don't touch my servers. And instead given straight to the developers. And that's huge, because it let us ship things faster. And that's ultimately the

whole goal of things. The thing that really struck me the most from conducting the trainings that I did was meeting a whole bunch of people across the country in different technological areas of specialty, in different states of their evolution as technologists. And something that struck me was just how much people wound up identifying with the technology that they worked on. When someone is the AIX admin and the AIX machines are getting replaced with Linux boxes, there's this tendency to fight against that and rebel rather than learning Linux. And I get it. I'm as subject to this as anyone is. And in many cases, that was the actual pushback that I saw against adopting something like Puppet. If I identify my job as

being the person that runs all of these carefully curated scripts that I've spent five years building, and now that all gets replaced with something that is more of a global solution to my local problem, then it feels like the thing that made me special is eroding. And we see that with the migration to cloud as well. When you're the storage admin, and it just becomes an API call to S3, that's kind of a scary thing. And when you're one of the server hugger types, again, as guilty as anyone of this. And you start to see cloud coming in as like a rising tide that eats up what it was that you became known for. It's scary. And they'd been doing basically the same set of tasks every year for 25 years. It's one year of experience repeated 25 times. And they don't have that much time left in their career intentionally. So they want to retire, but they also don't really want to learn a whole bunch of new technologies just to get through those last few years.

I feel for them, but at the same time... No, me too, totally. But what are you going to do? But without sounding sort of too dismissive there, like I think it's a natural tendency for us to identify with the technology. If that's what you're around all the time, you know, mechanics do this. Truck drivers with brands of trucks. People like build attachments to the technology they work with because we fit them into this bigger techno-social system. But I have a lot

of empathy for the people in enterprise jobs who are being asked to change radically because the cycle of progress is speeding up faster and faster. And as you say, like they might be a few years away from retirement. I think I used to feel a lot more differently about this when I was really hot-headed and, you know, sort of much more of a tech enthusiast. And that's what I identified with in terms of, you know, it's okay for a job to just be a job for people. It's okay for someone to be doing a job because they get good healthcare and good benefits and it's feeding their family. Like that's an important thing. You can't expect everyone to always be incredibly passionate about technology choices in the same way that I think, you know, many of us who live on Twitter and hang out in this space are. Oh, I have no problem whatsoever with people who want to show up for 40 hours a week-ish,

work on their job, and then go home and have lives and not think about computers at all. There's this dark mass of developers out there that basically never show up on Twitter. They aren't on IRC. They don't go to conferences. And that's fine. I have no problem with that. And I hope I don't come across as being overly dismissive of those folks. I honestly wish I could be content like that. I just don't hold still very well. But yeah, so I think you touched on a few interesting things there. And some of those we sort of cover in the State of DevOps report, which is coming out in the next few weeks.

Indeed. And the State of DevOps report started off at Puppet. And you've now done it for, what, 10 years? This is the 10th year, which is completely crazy. So I was looking at the stats as I was writing it. And it's 10 years of State of DevOps reports. I think it's 11 years of DevOps Weekly, Gareth Rushgrove's newsletter. It's 12 or 13 years of DevOps days that have been going on.

You know, this is longer than I spent in primary and high school put together. It's kind of crazy that the DevOps movement's still kind of chugging along, even if it's not necessarily the coolest kid on the block now that, you know, GitOps, SRE, Flavor of the Month, various kinds of permutations of how we work with technology have perhaps got a little bit cooler. But it's still very, very relevant to a lot of enterprises out there. Yeah. As I frequently say, legacy is a condescending engineering term for it makes money. And there's an awful lot of that out there. Forget cloud, there are still companies wrestling with,

do we explore this virtualization thing? And that was something I was very against back in 2006. Let's be very honest. I am very bad at predicting the future of technology. And I can see it for small niche edge workload cases where you have a bunch of idle servers, but for the most part, who's really going to use this in production? Well, basically everyone, because that in turn is what the cloud runs on.

Yeah, I think we can safely say I got that one hilariously wrong. But hey, if you aren't going to make predictions, then what's the matter? But the industry pushes you in these directions. So there was this massive bank in Asia who I've been working with for a long time, and they were always resistant to adopting virtualization. And then it was only four or five years ago that I visited them. They were like, right, okay, it's time. We're rolling out VMware. And I was like, so I'm really curious, what exactly changed in the last year or two in like 2014, 2015, that you decided virtualization was the key? And I'm like, oh, there was this jack

wagon who conducted this training. Yeah, no, no, sorry. I can't thank Reddit for that one. They couldn't order one rack unit servers with CD drives anymore because their whole process was actually provisioning with CDs before that point. Welcome to the brave new world of pixie booting, which is kind of hard. So yeah, virtualization is easier. You know, sometimes people have to be dragged into various ways of technological advancement, which gets to the real thing I want to cover, since this is a promoted episode where you're talking about the state of DevOps report. I'm almost less interested in what this year's has to say specifically than what you've seen over the last decade.

What's changed? What was true 10 years ago that is very much not true now? Bonus points, you can answer that without using the word Kubernetes more than twice. So I think one of the big things was that we've definitely passed peak DevOps team. Like if you may remember, there was a lot of arguments and there's still regular,

is DevOps a job title? Is it a team title? Is it a way of working? Oh, I was very much on the no side until I saw how much more I would get paid as a DevOps engineer instead of a systems administrator for the exact same job. So, you know, I shut up and I took the money. I figured that the semantic arguments are great, but yeah. And that's exactly what we've written in the report.

And I think it's great. The sysadmins, you know, we were unloved. You know, we were in the basement. We weren think it's great. The sysadmins, we were unloved. We were in the basement. We weren't paid as much as programmers. The joke used to be, for developers, DevOps meant I don't need ops anymore. But for ops people, it was I can get paid like a developer. In many cases, oh, well, systems administrators don't want to learn how to code. Yeah, you're remembering a relatively narrow slice of time between the modern era where systems administrator types need to be able to write in the lingua franca of everything, which is, of course, YAML, as far as programming languages go.

And before that, to be a competent systems administrator, you needed to have a functional grasp of C. And there is only a limited window in which a bunch of bash scripts and maybe a smidge and a pearl would have carried you through. But the deeper understanding is absolutely necessary and I would argue always has been. And this is great

because you've just linked up with one of the things we found really interesting about the report is that, you know, when we talk about legacy, we don't actually mean the oldest shit because the oldest shit is the mainframes. It's a lot of bare metal applications. A lot of that in big enterprises. We're still waiting for an AWS 400

to replace some of that. Well, it's administered by real systems engineers, you know, like the people who wrote C, who wrote kernel extensions, who could debug things. What we actually mean by legacy is we mean late nineties to late 2000s, early 2010s, stuff that was put together by kids who, like me, happened to get a job because you grew up with a computer and then the dot-com explosion happened. You weren't necessarily particularly skilled and a lot of people were,

they didn't go through the apprenticeships that mainframe folks and systems engineers actually went through. And everyone just held this stuff together with duct tape and dental floss. folks and systems engineers actually went through and everyone just held this stuff together with, you know, duct tape and dental floss. And then now we're paying the price of it all, like way back down the track. So legacy is really just a certain slice of rapid growth in applications and infrastructure. That's sort of an unmanageable mess now.

Here in San Francisco, legacy is anything prior to last night's nightly build. It's turned into something a little ridiculous. I feel like the real power move as a developer now Here in San Francisco, legacy is anything prior to last night's nightly build. It's turned into something a little ridiculous. I feel like the real power move as a developer now is to get a job, go in on day one, rebase everything in the Git repository to a single commit with a message, legacy code, and then force push it to the main branch. And that's the power move. And that's how it works. And that's also the attitude we wind up encountering in a lot of places.

And I don't think it serves anyone particularly well to tie themselves so tightly to that particular vision. Yep, absolutely. But this is a real problem, this space. And one of the things we found in the state devils report is that, let me back up a little and give a little bit of methodology of what we actually do. We survey people about their sort of performance metrics, you know, like how quickly can you do deploys? What's your sort of mean time to recovery? Those sorts of things. And what practices do you actually employ? And we essentially go through and do statistical analysis on this. And everyone tends to end up in three cohorts. They separate pretty easily of low, medium, and high evolution. And so one of the things we found is that everyone at the low level has all

sorts of problems. They have issues with what does my team do? What does the team next to me do? How do I talk to the team next to me? How do I actually share anything? How do I even know what my goals are? Like fundamental company problems. But everyone at all levels of evolution is stuck on two big things. Not being able to find enough people with the right skills for what they need and their legacy infrastructure holding them back. The thing that I find the most compelling is the idea of not being able to find enough people with the skills that they need. And I'm going to break my own rule and mention Kubernetes as a prime example of this. If you are effective at managing Kubernetes in production, you will make a very comfortable

living in any geographical location on the planet because it is incredibly complex. And every time we've seen this in previous trends where you need to get more and more complexity and more and more expertise just to run something, it looks like a sawtooth curve where at some point that complexity gets abstracted away and compressed down into something that is basically a single line somewhere, or it happens below the surface level of awareness. My argument has been that Kubernetes is something no one's going to care about in roughly three years from now, not because we're not using it anymore, but because it's below the level of awareness that we have to think about. In the same way that there aren't a whole lot of people on the planet these days

who have to think about the Linux virtual memory management subsystem. It's there, and a few people really care about it. But for the rest of us, we don't have to think about that. That is the infrastructure underneath our infrastructure. Absolutely. I used to make a living, and it's ridiculous looking back at this for a year or two, doing high-performance custom compiled Apaches for people. Like, I was really, really good at this.

Well, yeah. Apache is a great example of this. Where back in the 90s, to get a web server up and running, you needed to have three days to spare and in-depth knowledge of GCC compiler flags and hope for the best. And then RPM came out. And then, okay, then Yum or other things like that on top of it. And then things like Puppet started showing up and we saw, oh, great, now Insure installed. Great. And then we took a step beyond that and it was, oh, now it's just a Docker run, whatever it is. And these days, yeah, it's a checkbox in S3.

So let me get your Kubernetes prediction down right. So you're predicting Kubernetes is going to go away like Apache and highly successful things. It's not an open stack failure state. It's a Apache invisibility state. Absolutely. My timeline is a bit questionable, let's be fair, but on the aggressive side.

But yeah, I think that Kubernetes is inherently too complex for most people to have to wind up thinking about it in that way. And we're not talking small companies, we're talking big ones where you're not in a position if you're a giant blue chip fortune 50 to hire 2000 people who all know Kubernetes super well, and you shouldn't have to. There needs to be some flattening of all of that high-level complexity. Without the management tools, though, with things like Puppet and the things that came before in a bunch of different ways, we would all not be able to get anything done because we'd be too busy writing an assembly. There's always going to be those abstractions on top of abstractions on top of abstractions, and very few people understand how it works all the way down. But that's, in many cases okay. That's civilization, you know?

Like, do you understand what happens when you plug in something to your electricity socket? I don't want to know. Like, I just want light. And more to the point, whenever you flip the switch, you don't have that doubt in your mind that the light is going to come on. So if it doesn't, that's notable. And your first thought is, oh, the light bulb is out,

not the utility company is down. And we talk about the cloud being utility computing. Has someone put a Kubernetes operator in this light switch that may break this process? Well, okay, IoT does throw a little bit of a crimp into those works, but yeah. So let's talk more about the state of DevOps report. What notable findings were there this year? So one of the big things that we've seen for the last couple of years has been that most companies are stuck in the middle of the evolutionary progress. And anyone who deals with large enterprises knows this is true.

Whatever they've adopted in terms of technology, in terms of working methods, agile, various different things, most companies don't tend to advance to the higher levels. Most places stay mired in mediocrity. So we wanted to dive into that and try and work out why are most companies actually stuck like this when they hit a certain size? And it turns out the problems aren't technology or DevOps. They're like really fundamental problems. Like we don't have clear goals. I don't understand what the teams next to me do. We did a bunch of qualitative interviews as well as the quantitative work in the survey with this report. And we talked to one group of folks at a pretty large financial services company who were

like, our teams have all been renamed so many times. If I need to go and ask someone for something, I literally page up and down through ServiceNow trying to find out where to put the change request and they're like, how do I know where to put a network port opening request for this particular service when there are 20 different teams that might be named the right thing and some are obsolete and I get no feedback whether I've sent it off to the right thing or to a black hole of enterprise despair. I really love installing, upgrading, and fixing security agents in my cloud estate. Why do I say that? Because I sell things for a company that deploys an agent. There's no other reason. Because let's face it, agents can be a real headache. Well, Orca Security

now gives you a single tool to detect basically every risk in your cloud environment that's as easy to install and maintain as a smartphone app. It is agentless, or my intro would have gotten me in trouble here, but it can still see deep into your AWS workloads while guaranteeing 100% coverage. With Orca Security, there are no overlooked assets, no DevOps headaches, and believe me, you will hear from those people if you cause them headaches, and no performance hits on live environment. Connect your first cloud account in minutes and see for yourself at orca.security. That's orca as in whale, dot security as in that thing your company claims to care about but doesn't until right

after it really should have. That doesn't get better with a lot of modernization. I mean, I feel like half of my job, and I'm not exaggerating, is introducing Amazonians to one another. Corporate communication between departments and different groups is very far from a solved problem. I think the tooling can help, but I've never been a big believer in solving political problems with technology.

It doesn't work. People don't work that way. Absolutely. Like one of my earliest times working at Puppet doing sort of higher level sales and services and support, huge national telco walk in there. We've got the development team, the QA team, the infrastructure team.

In the course of this conversation, one of them makes a comment about using apt-get, and the others were like, what do you mean? We're on RHEL. And it turned out production was running on RHEL, the QA team running on CentOS, and the developers were all building everything on Ubuntu. And because it was Java apps, they almost didn't have to care. But, you know, write once, debug everywhere. History doesn't repeat, but it rhymes. Before Docker, so much of development and startup land was, how do I make my MacBook Pro look a lot more like an EC2 Linux instance? And it turns out that there's an awful lot of work that goes into that that maybe isn't the best use of people's time.

And we start to see these breakthroughs and these revelations in a bunch of different ways. I have to ask, this is the 10th year that you've done the State of DevOps report. At this point, why keep doing it? Is it inertia? Are you still discovering new insights every year on top of it? Or is it one of those things where, well, someone in marketing says we have to do it, so here we are? No, actually, it's not that at all. So, definitely, we're going to take stock after this year because 10 years feels like a really good

point to sort of, it's a nice round number and, you know, a certain kind of number system. Mainly, the reason is, you know, a lot of my job is going and helping big enterprises just get better at using technology. And it's funny how often, like, I just get folks going, oh, I read this thing, like people who aren't on, you know, the bleeding edge, constantly discussing these things on Twitter or whatever, but the state DevOps support makes its way to them. And they're like, oh, I read a thing there about how much better it is if we standardized on one operating system. And that made a really huge difference to what we were actually doing

because you had all this data in there showing that that is better. And honestly, that's the biggest reason why I ended up doing it. You know, it's the fact that it seems to be a tool that has made its way through to very hard to penetrate enterprise folks. And they'll read it and managers will read things

that are like, if you set clear goals for your team and get them to focus on optimizing the legacy environment, you will see returns on it. And I'm being a little bit facetious in the tone that I'm saying, because a lot of this stuff does feel obvious if you're constantly swimming in this stuff day to day. But it's not just the practitioners who it's just a job for in a lot of big companies. It's true of a lot of the management chain as well. They're not necessarily going out and reading up on, you know, modern, agile IT management practices day to day for fun. You know, they go home and do something else. One of my favorite conferences is Gene Kim's DevOps Enterprise Summit. And the specific reason

behind that is these are very large companies that go beyond companies in some cases to institutions, where you have the US Air Force as a presenter one year, and very large banks that are 200 years old. And every other conference, it seems, more or less involves people getting on stage to deliver conferenceware and tell stories that make people at those companies feel bad about themselves. Where it's, we're Twitter for pets and this is how we deploy software. Or the ever popular, this is how Netflix does stuff. Yeah, Netflix has basically no budget constraints as far as hiring engineering folks go.

And lest we forget, their failure mode is someone can't watch a movie right now. It's not exactly the same thing as the ATM starts spitting out the wrong balance in the streets. And I think that there's an awful lot of discussion where people look at the stories people tell on conference stages and come away feeling bad from it. Very often I'll see someone from a notable tech company talk about how they do things and, wow, I wish my group did things like that. And the person next to me says, yeah, me too. And I check and they work at the same company. And it's like the stories we tell are not necessarily the stories that we live. And it's very easy to come away discouraged from these things. And that goes triply so

for large enterprises that are regulated, that have significant downside risk if the technology fails them. And I love watching people getting a chance to tell those stories. Let me jump on that really quickly. Please, by all means. One is, you know, having done four years at Google, things are a shit show internally there too. You're talking about it like it's prison. I like it. You know, like people get horrified when they turn up and they're like, oh, what? It's not all gleaming, perfect software artifacts, you know, delivered from the hand of us. But I think what Gene has done with DevOps Enterprise Summit is fantastic and how people share more openly their failure states. But even there, and this is an interesting result we found

from a few years ago, even those executives are being more optimistic because it's sort of beaten into you as a senior executive. You're putting on a public face. And even when they're trying to share the warts and all story, they can't help but put a little bit of a positive spin on it because I've had exactly the same experience there where someone's up there telling a war story. And then I look, turn to the person next to me and they work at that same 300 year old bank. And they're like, actually, it's much, much worse than this. And we didn't fix it quite as well as that. So I think, you know, the big tech companies are terrible inside, unless they're Netflix. And the big enterprises are also terrible, but they're also-

No, no, I've talked to Netflix people too. They do terrible things internally there too. No one talks about the fact their internal environments are always tire fires. And there are two stories, the stories we tell publicly and the reality. And if you don't believe me on that, look at any company in the world's billing system. As much as we all talk about Agile and various implementations thereof, when it comes to things that charge customers money, we're all doing waterfall. Absolutely.

Because mistakes show when you triple charge someone's credit card for the cost of a small country's GDP. It's a problem. I want to normalize those sorts of things more. I'm looking forward to reading this year's report just because it's interesting to see how folks who are in environments that differ from the ones that I get to see experience this stuff and how they talk about it. Yeah. And so one of the big results I think there for big companies that's really interesting is that one of the sort of anti-patterns is having lots of different types of teams. And I kind of touched on this before about, you know, having confusing team titles being a real problem and not being able to cross organizational boundaries quickly is really, really, you know, it's a huge inhibitor and

cause source of friction. But turns out the pattern that is actually really great is one that the team topologies guys have discovered. If you've been following what Matthew Skelton and Manuel Pais have been doing for a while, they've basically been documenting a pattern in software organizations of a small number of team types of a platform team, value stream teams, complicated subtest system teams, and enabling teams. And so we worked with Manuel and Matt on this year's report and asked a whole bunch of questions to try and validate the team topologies model and the results came back and they were just incredibly strong. Because I think this speaks to some of the stuff you mentioned before that no one can afford to hire an army

of Kubernetes developers and whatever the hottest technologies in five years most big companies can't hire an army of those people either. And whatever the hottest technologies in five years, most big companies can't hire an army of those people either. And so the way you get scale internally before those things become commoditized is you build a small team and create the situation where they can have outsized leverage inside their organization. Like get rid of all the blockers to fast flow and make their focus self-service to other people. Because if you're making all of your developers learn distributed systems operations, arcane knowledge, like that's not a good use of their time either. It's really not.

And I think that that's something that gets lost a lot is I've never yet seen a company beyond the very early startup stage where the AWS bill exceeded the cost of the people working on the AWS bill. Payroll is always a larger expense than infrastructure, unless you're doing something incredibly strange. And, oh, I want to save some money on the cloud bill is very often offset by the sheer amount of time that you're going to have to pay people to work on that. Because contrary to what we believe as engineering hobbyists, people's time is very far from free. And it's also the opportunity cost of if you're going to work on this thing instead of something else, well, is that really the best choice? It comes down to contextualizing what technology is

doing, as well as with what's happening over in the world of business strategy. And without having a bridge between those, it doesn't seem to work very well. Absolutely. It's insane. Like it's literally insane that as an industry, we will optimize 5%, 3% of our infrastructure bill or application workload, and yet not actually re-examine business processes that are causing your people to spend 10% of their time in synchronous meetings. You know, like you can save so much more money and achieve so much more by actually optimizing for fast flow and getting out of the way of the people who cost lots of money. So one last topic that I want to cover before we call it an episode. You talk to an awful lot of folks, and it's easy to point at the aspirational stories of

folks doing things the right way. But let's dish for a minute. What are you seeing in terms of people not using the cloud properly? I feel like you might have a story or two on that one. I do have a few stories. So in this year's report, one of the things we wanted to find out of, like, are people using the cloud sort of, you know, in the way we think of cloud, you know, elastic, consumption-based, all of these sorts of things. We use the NIST metrics, which I recognize can be a little controversial, but I think you've got to start somewhere as a certain foundation. It turns out just about everyone is using public cloud. And when I say cloud, like I'm not really talking about people's internal VMware that they've rebadged as cloud. Like I'm talking about

the public cloud providers. Everyone's using it, but almost no one is taking advantage of the functionality of the cloud. They're instead treating it like an on-premise VMware installation from the mid-2000s. They're taking six weeks to provision instances. They're importing all of their existing processes. They keep these things running for a long time. If they fall over, one person is tasked with, hey, do you know how pet number 45 is actually doing here? They're not really treating any of these things in the way that they're actually meant to. And I think we forget about this a lot of the time when we talk about cloud because we jump straight to cloud native, the sort of bleeding edge of folks and

serverless, highly orchestrated containers. I think if you look at the actual numbers, the vast majority of cloud usage, it's still things like EC2 instances on AWS. And there's a reason because it's a familiar paradigm for people. We're definitely going to progress past there, but I think it's easy to leave the people in the middle behind when we're talking about cloud and how to improve the ecosystem that they all operate in. Part of the problem too, is that whenever we look at how folks are misusing cloud, it's easy to lose sight of context. People don't generally wake up and decide I'm going to do a terrible job today unless they work in, you know, Facebook's ethics department or something.

Instead, it's very much a people are shaped by the constraints they're laboring under from a bunch of different angles, and they're trying to do the best with what they have. Very often, the reason that a practice or a policy exists is because once upon a time, there was a constraint that may or may not still be there. And going forward the way that they have seemed like the best option at the time. I found that the default assumption that people are generally smart and doing the right thing with the information they have carries you a lot further in many respects than what I did as a terrible junior consultant, which is, oh, what moron built this? Invariably to said moron. And then the rest of the engagement rapidly goes downhill from there. Try and assume good faith.

And instead of you see something that makes no sense, ask, why is it like this? Rather than, why is it like this? Tone. Counts for a lot. It's the fundamental attribution bias. You know, it's why we think all other drivers on the road are terrible, but we actually had a good reason for swerving into that lane. This isn't how I would have built it, so it's awful. Yeah, exactly. And in some cases, though, there are choices that are objectively bad, but I try to understand where they came from there. Company policy historically around things like data centers trying to map one-to-one to cloud often miss some nuances. But hey, there's a reason it's called a digital transformation, not a project that we did. And I think you've got to always have empathy for the people on

the ground. I quite often have talked to folks who've got like a terrible cloud architecture with the deployment. I'm like, well, what happened here? And they went, well, we were prepared to deploy this whole thing on AWS, but then Microsoft salespeople got to the CTO and we got told at the last minute, we're redeploying everything on Azure. And so these people were often, you know, you're given a week or two to pivot around a decision that doesn't necessarily make any sense to them. And there may have been a perfectly good reason for the CTO to do this. They got given really good kickbacks, you know, in terms of bonuses for like how much they were spending on the infrastructure. I mean, discounts. But people on the ground are generally doing the best with what

they can do. If they end up building crap, it's because, you know, our system, society, capitalism, everything else is at fault. I have to say, I'm really looking forward to seeing the observations that you wound up putting into this report as soon as it drops. I'm hoping that I get a chance to speak with you again about the findings, and then I can belligerently tell you to justify yourself. Those are my favorite follow-ups. Brilliant. If people want to get a copy of the report for themselves or learn more about you, where can they find you?

Just head straight to puppet.com and it will be on the banner on the front of the site. Excellent. And we'll, of course, put a link to that in the show notes if people can't remember puppet.com. Thank you so much for taking the time to speak with me. I really appreciate it. Awesome. No worries. It was good to catch up.

Nigel Kirsten, field CTO at Puppet. I'm cloud economist, Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice. Whereas if you've hated this podcast,

please leave a five-star review on your podcast platform of choice, as well as an insulting comment telling me that comcastic isn't a funny word, and tell me where you work, though we already know. If your AWS bill keeps rising and your blood pressure is doing the same, then you need the Duck Bill Group. We help companies fix their AWS bill by making it smaller and less horrifying.

The Duck Bill Group works for you, not AWS. We tailor recommendations to your business, and we get to the point. Visit duckbillgroup.com to get started. this has been a humble pod production stay humble