Screaming in the Cloud - Inside the Mind of a DevOps Novelist with Gene Kim

Episode Date: March 4, 2020

About Gene KimGene Kim is a multiple award-winning CTO, researcher and author, and has been studying high-performing technology organizations since 1999. He was founder and CTO of Tripwire for 13 years. He has written six books, including The Unicorn Project (2019), The Phoenix Project (2013), The DevOps Handbook (2016), the Shingo Publication Award winning Accelerate (2018), and The Visible Ops Handbook (2004-2006) series. Since 2014, he has been the founder and organizer of DevOps Enterprise Summit, studying the technology transformations of large, complex organizations.Links Referenced The Phoenix Project: https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/1942788290/The Unicorn Project: https://www.amazon.com/Unicorn-Project-Developers-Disruption-Thriving/dp/B0812C82T9The DevOps Enterprise Summit: https://events.itrevolution.com/@RealGeneKim

Hello and welcome to Screaming in the Cloud with your host, cloud economist Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud. important to me, no billing surprises. With simple, predictable pricing that's flat across 12 global data center regions and a UX developers around the world love, you can control your cloud infrastructure costs and have more time for your team to focus on growing your business. See what businesses are building on DigitalOcean and get started for free at do.co slash screaming. That's do.co

slash screaming. And my thanks to DigitalOcean for their continuing support of this ridiculous podcast. This episode is also sponsored in part by Logs.io. Logs.io hears it loud and clear from many engineers, specifically that they prefer using open source tools for observability. They get an easier onboarding experience, a built-in community ready to go and a lot less vendor lock-in. But what about the enterprise grade scale support and security? You need to improve the performance of any given cloud environment.

That's where logs. I O comes in. They offered a fully managed observability service, log management and cloud seam based on ELK, and infrastructure monitoring based on Grafana, the open source you love at the scale you need. Sign up today for a 14-day free trial at Logs.io slash Screaming, and for your chance to receive a free Logs.io t-shirt. Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined this week by a man who needs no introduction, but gets one anyway, Gene Kim,

most famously known for writing The Phoenix Project, but now the Wall Street Journal bestselling author of The Unicorn Project, six years later. Gene, welcome to the show. Corey, so great to be on. I was just mentioning before how delightful it is to be on the other side of the podcast. And it's so much smaller in here than I had thought it would be.

Excellent. It's always nice to wind up finally meeting people whose work was seminal and foundational. Once upon a time when I was a young, angry Unix systems administrator, because it's not like there's a second type of Unix administrator. The Phoenix project was one of those things that was – one of those texts that was transformational as far as changing the way I tended to view a lot of what I was working on and gave a glimpse into what could have been a realistic outcome for the world or the company I was at, but somehow was simultaneously uplifting and incredibly depressing all at the same time. Now the Unicorn Project does that exact same thing, only aimed at developers instead of traditional crusty ops leadership. So Bill Palmer, the protagonist of that book, was the VP of operations at Parts Unlimited. And the protagonist in the Unicorn Project is Maxine Chambers, senior architect and developer. And I love the fact that it's told in the same timeline as the Phoenix Project. And

in the first scene, she is unfairly blamed for causing the payroll outage and is exiled to the Phoenix Project, where she recoils in existential horror and then finds that she can't do anything herself. She can't do a build. She can't run her own tests. She can't, God forbid, do her own deploys. that many developers find themselves in where they're just caught in decades of built-up technical debt, unable to do even the simplest things independently, let alone be able to independently develop, test, or create value for customers.

So it was fun, very much fun, to revisit the Parts Unlimited universe. What I found that was fun about it, there are a few things in there I want to unpack. The first is that it really was the, shall we say, retelling of the same story in quote-unquote the same time frame, but these books were written six years apart. Yeah, and by the way, I want to first acknowledge all the help that you gave me during the editing process. I mean, your comments were just so spot on with exactly the feedback I needed

at the time and led to the most significant kind of lift to jam a whole bunch of changes in right before it got turned over to production. Yeah, so the Feast Project is told, quote, in the present day. And in the same way, the Unicorn Project is also told and takes place in the present day. In fact, they even start plus or minus on the same day. And there is a little bit of suspension of disbelief needed just because there are certain things that are in the common vernacular, kind of very much in the zeitgeist now that weren't six years ago, like digital disruption, even things like Uber and Lyft

that feature prominently in the book that were just never mentioned in the Phoenix Project. But yeah, I think the story is very much told in the same vein as like Ender's Shadow, right? Where it takes place in the same timeline but from a different perspective. So something else that – again, I understand it's an allegory, and trying to tell an allegorical story while also working it into the form of a fictional work is incredibly complicated. That's something that I don't think people can really appreciate until they've tried to do something like it. But I still found myself at various times reading through the book and wondering, asking myself questions that I guess say more about me than they do about anyone else.

But it's, wow, she's at a company that is pretty much scapegoating her and blaming her for all of this. Why isn't she quitting? Why isn't she screaming at people? Why isn't she punching the boss right in their stupid condescending face and storming out of the office? And I'm wondering how much of that is my own challenges as far as how life goes as well as how much of it is just – for, I guess, narrative devices. It needed to wind up being someone who would not storm out when push came to shove. Well, and I think she actually does the last of the third thing that you mentioned where, you know, she does slam the sheet of paper down and say,

man, you said the outage is caused by a technical failure and a human error. And are you telling me I'm the human error and just cannot believe that she's being put in that position. But yeah. So thanks to your feedback and others, right. You know,

she actually does shop her resume around and, you know, starts putting out feelers, right. Because this is no longer feeling like a great place to work that attracted her eight years prior. The reality is for most people is that sometimes it's difficult to get a new job overnight, even if you want to. But I think that Maxine stays because she believes in the mission. She takes a great deal of pride of what she's created over the years.

And I think in most great brands, they do create a sense of mission and there's a deep sense of the customers they serve. And there's something very satisfying about the work to her. And I think she is very much for a couple of weeks, very much always thinking about she won't be here for long one way or another. But by the time she stumbles into the rebellion, the crazy group of misfits, the ragtag bunch of misfits who are trying to find better ways of working and willing to break whatever rules it takes to take on the very ancient powerful order. She falls in love with the group. She found a group of kindred spirits who very much, like her, believe that developer productivity is one of the most important things that we can do as an organization. So by the time that she links up with that group, I mean, I think she's – all thoughts of leaving are gone. Right.

And the idea of if you stick around, you can theoretically change things for the better is extraordinarily compelling. The challenge I've seen is that as I navigate the world, I've met a number of very gifted employees who frankly wind up demonstrating that same level of loyalty and same kind of loyalty to companies that are absolutely not worthy of them. So my question has always been when do I stick around versus when do I leave? I'm very far on the bailout as early as humanly possible side of that spectrum. It's why I make a great consultant, but an absolutely terrible employee. Well, yeah, so we were honored to have you at the DevOps Enterprise Summit. And you've seen that the Unicorn Project book is really dedicated to the achievements of the DevOps Enterprise community, certainly inspired by and dedicated to their efforts. And I think what was so inspirational to me were all these courageous

leaders who know what the mission is. I mean, they viscerally understand what the mission is and understand that the ways of working aren't working so well and are doing whatever they can to create better ways of working that are safer, faster, and happier. And I think what is so magnificent about so many of their journeys is that the organization in response says, thank you. That's amazing. Can we put you in a position of even more authority that will allow you to even make a more material, more impactful contribution to the organization? And so it's been my observation, having run the conference for now six years, going on seven years, is that this is a population that has been, you know, has been promoted at a rate far higher than the population at large. And so for me, that's just an incredible story of grit and determination. And so, yeah, where does grit and determination become sort of blind loyalty that's ultimately, you know, self-punishing? That's kind of a deep question that I've never really studied. But I certainly do understand that there is a time when no amount of perseverance and grit will get from here to there.

And that's a fact. I think that it's a really interesting narrative just to see it, how it tends to evolve. But also, I guess, for lack of a better term, and please don't hold this against me, it seems in many ways to speak to a very academic perspective. And I don't mean that as an insult. Now, the real interesting question is why I would think, well, why would accusing someone of being academic ever be considered as an insult? But my academic career was fascinating. It feels like it aligns very well with the five ideals,

which is something that you have been talking about significantly for a long time. And in an academic setting, that seems to make sense, but I don't see it thought of or spoken of in the same way on the ground. So first, can you start off by giving us an intro to what the five ideals are, and I guess maybe disambiguate the theory from the practice? Oh, for sure. Yeah. So the five ideals are, let's go back one step. So the Phoenix Project had the three ways, which were the principles from which you can derive all the observed DevOps practices from,

and the four types of work. And so in the five ideals, I use the constructs of the five ideals. And the next version of the nine, whatever you call them at that point, I'm sure, with the geometric progression. That's right. Or actually, isn't it the prime? Oh, no, four isn't prime. Yeah, yeah.

I don't know. So, yeah, the five ideals is a nice small number. And it was just really meant to verbalize things that I thought were very important, things I just gravitate towards. One is locality and simplicity. And, you know, briefly, that's just to what degree can teams do what they need to do independently without having to coordinate, communicate, prioritize, sequence, marshal, de-conflict with other scores of other teams. The second ideal is what I think the outcomes are when you have that, which is focus, flow,

and joy. And so Dr. Mihaly Csikszentmihalyi, he describes flow as a state when we are so engrossed in the work we love that we lose track of time and even sense of self. And that's been very much my experience coding ever since I learned Clojure, this functional programming language. Third ideal is improvement of daily work, which shows up in the Phoenix project, just saying that improvement of daily work is even more important than daily work itself. Fourth ideal is psychological safety, which shows up in the state of DevOps report, but showed up prominently in Google's Project Oxygen and even in the Toyota production process, where clearly it has to be, in order for someone to pull the and-on cord that potentially stops the assembly line, you have to have an environment where it's psychologically safe to do so. And then fifth ideal is customer focus. Really focus on core competencies that create enduring, durable business value that customers are willing to pay for versus context, which is everything else. And to answer your question, where did it come from? Why do I think it's

important? Why do I focus on that? For me, it's really coming from the state of DevOps report that I did with Dr. Nicole Forsgren and Jez Humble. And so beyond all the numbers and the metrics and the technical practices and the architectural practices and the cultural norms, for me, what that really tells a story of is of the five ideals. One of them is very much the need for architecture that allows teams to work independently, having a higher predictor of even continuous delivery. I love that. And from the individual perspective, you know, the ideal being that allows us to focus on the work we want to do to help achieve the mission with a sense of flow and joy. And then really elevating the notion that greatness isn't free.

We need to improve daily work. We have to make it psychologically safe to talk about problems. And then the last one really being, you know, can we really unflinchingly look at the work we do on an everyday basis and ask, you know, whether customers care about it? And if customers don't care about it, can we question whether that work really should be done or not, right? So, yeah, that's where, for me,

it's really kind of meant to speak to some more visceral emotions that were concretized and validated through the State of DevOps report. But are these notions I am just very attracted to. I like the idea of it. The question, of course, is always how to put these into daily practice. How do you take these from an idealized, well, let's not call it a textbook, but something very similar to that,

and apply it to the, I guess, uncontrolled chaos that is the day-to-day life of an awful lot of people in their daily jobs. Yeah, right. So, yeah, so the protagonist is Maxine, and her role in the story in the beginning is just to recognize what not great looks like. She's lived and created greatness for all of her career, and then she gets exiled to this terrible Phoenix project that chews up developers and spits them out, and they leave kind of the hulks of the people they used to be, right? These husks of people they used to be.

And so, you know, she's not doing a lot of problem solving. Instead, it's just kind of recoiling from, you know, the inability for people to do builds or do their own tests or be able to do work without having to open up 20 different tickets or not being able to do their own deploys. She just recoils from this, spending five days watching people do code merges. And for me, I'm hoping that what this will do after people read the book, they'll kind of see this all around them. And hopefully they'll have a similar kind of recoiling reaction where they say, oh my gosh, this is terrible.

I should feel as bad about this as Maxine does. And then maybe even find my fellow rebels and see if we can create a pocket of greatness that can become sort of like the sublimation event in Dr. Thomas Kuhn's book, The Structure of Scientific Revolution, right? Does that create that kernel of greatness of which then greatness then, you know, surround, find itself surrounded by even more greatness. What I always found to be fascinating about your work is how you wind up tying so many different concepts together in ways you wouldn't necessarily expect.

For example, when I was reviewing your, or one of your manuscripts before this went to print, you did reject one of my suggestions, which was to just retitle the entire thing. Instead of calling it The Unicorn Project, instead call it Gene Kim's Love Letter to Functional Programming. So what is up with that? Yeah, to put that into context, for 25 years or more,

I've self-identified as an ops person, right? The Phoenix Project was really an ops book. And that was despite getting my graduate degree in compiler design and high-speed networking in 1995. And the reason why I gravitated towards ops is because that was my observation that that's where the saves were made. It was ops who saved the customer from horrendous, terrible developers who just kept on putting things into production that would then blow up and take everyone with it. It was ops protecting us from the bad adversaries who were trying to steal data, because security people were so ineffective. But four years ago, I learned a functional programming language called Clojure. And without a doubt, it reintroduced the joy of coding back into my life.

And now in a good month, I spend half the time in the ideal writing, half the time hanging out with the best in the game, of which I would consider this to be a part of, and then 20% of the time coding. And I find for the first time in my career, in over 30 years of coding, I can write something for years on end without it collapsing in on itself like a house of cards. And that's an amazing feeling to say that maybe it wasn't my inability or my lack of experience or my lack of sensibilities. But maybe it was just that I was sort of using the wrong tool to think with. That comes from the French philosopher Claude Lévi-Strauss. He said of certain things, is it a good tool to think with? And I just find functional programming is such a better tool to think with that notions like composability, like immutability,

what I find so exciting is that these things aren't just for programming languages. And so other programming languages that kind of follow the same vein are OCaml, Lisp, ML, Elixir, Haskell, right? Those are, these are all languages that are sort of popularizing functional programming. But what I find so exciting is that we see it in infrastructure and operations too.

So Docker is fundamentally immutable. So if you want to change a container, we have to make a new one. Kubernetes composes these containers together at the level of system of systems. Kafka is amazing because it usually reveals the desire to have this immutable data model where you can't change the past. Version control is immutable. So I think it's no surprise that as our systems get more and more complex and distributed, we are relying

on things like immutability just to make it so that we can reason about them. So it is something I love addressing in the book, and it's something I decided to double down on after you mentioned it. I'm just saying, all kidding aside. Oh, good, I got to make it worse. Always exciting when that happens.

Yeah, I mean, your suggestion really brought to the forefront a very kind of critical decision, which was, is this a book for technology leaders or even business leaders, or is this a book for developers? After a lot of soul-searching, I decided, no, this is a book for developers. I think the sensibilities that we need to instill and the awareness we need to create these things around are the developers. And then you just hope and pray that the book will be good enough that if enough engineers like it, then engineering leaders will like it. And if enough engineering leaders like it, then maybe some business leaders will read it as well. So that's something I'm eagerly seeing what will happen kind of as the weeks, months, and years go by. This episode is sponsored in part by

DataStax. The NoSQL event of the year is DataStax Accelerate in San Diego this May from the 11th through the 13th. I've given a talk previously called The Myth of Multicloud, and it's time for me to revisit that with a SQL, which is funny given that it's a no-sequel conference, but there you have it. To learn more, visit datastax.com. That's D-A-T-A-S-T-A-X dot com. And I hope to see you in San Diego this May. One thing that I always admired about your writing is that you can start off trying to make a point about one particular aspect of things. And along the way, you tie in so many different things, and the functional programming is just one aspect of things. And along the way, you tie in so many different things. And the functional programming is just one aspect of this.

At some point by the end of it, I half expected you to just pick a fight over VI versus Emacs, just for the sheer joy you get in effectively drawing interesting, and I guess, shall we say, the right level of conflict into it, where it seems very clear

that what you're talking about is something that has the potential to be transformative. And by throwing things like that in, you're on some level roping people in who otherwise wouldn't weigh in at all. But it's really neat to watch, once you have people's attention, just almost in spite of what they want,

you teach them something. I don't know if that's a fair accusation or not, but it's very much, I'm left with the sense that what you're doing has definite impact and reverberations throughout larger industries. Yeah, I hope so. In fact, I mean, just to reveal this kind of insecurity is, you know, there's an author I've read a lot of, and she actually read this blog post that she wrote about kind of

the worst novel to write, and she called it the Yeoman's Tour of the Starship Enterprise. And she says, the book begins like this. It's Yeoman on the Starship Enterprise. And all he does is admire the dilute crystals in the phaser and talk about the specifications of the engine room. And I sometimes worry that that's what I've done in the Unicorn Project. But hopefully, I did want to have that technical detail there and share some things that I love about technology and the things I our mutual distaste of YAML files or that we've all struggled trying to escape spaces and file names inside of Makefiles. These are things that are puzzles we have to solve, but they're so far removed from the business problem we're trying to solve that really the purpose of that was trying to show the absurdity

of the mistake of solving puzzles in our daily work instead of solving real problems. One thing that I found was really a one-two punch for me at least was, first I read and gave feedback on the book and then relatively quickly thereafter, I found myself at my first DevOps Enterprise Summit. And I feel like on some level,

I may have been misinterpreted when I was doing my live tweeting slash shit posting with style during a lot of the opening keynotes and the rest, where I was focusing on how different of a conference it was. Unlike a typical DevOps Days or big cloud event, it wasn't a whole bunch of relatively recent software startups. There were serious institutions coming out to have conversations. We're talking USAA. We're talking the US Air Force. We're talking large banks. We're talking companies that have a 200 year history where you don't get to just throw everything away and start over. These are companies

that by and large have in many ways felt excluded to some extent from the modern discussions of, well, we're going to write some stuff late at night. And the following morning it's in production. You don't get to do that when you're a 200-year-old insurance company. And I feel like that was on some level interpreted as me making fun of startups for quote unquote not being serious, which was never my intention. It's just, this was a different conversation series for a different audience who has vastly different constraints.

And I found it incredibly compelling, and I intend to go back. Well, that's wonderful. In fact, we have plans for you, Mr. Quinn. Uh-oh. Yeah, I think, when I say I admire the DevOps Enterprise community, I mean that on just so many different dimensions. The fact that these leaders – and it's not leaders just in terms of seniority on the org chart.

These are people who are leading technology efforts to survive and win in the marketplace in organizations that have been around sometimes for centuries. Barclays Bank was founded in the year 1634. That predates the invention of paper cash. HMRC, the UK's version of the IRS, was founded in the year 1200. And so there's probably no code that goes that far back. But there are certainly values. Well, you'd like to hope not. Yeah, right. You never know.

But there are certainly values and traditions and maybe even processes that go back centuries. And so that's what's helped these organizations be successful. And here are a next generation of leaders trying to make sure that these organizations see another century of greatness. So I think that's, in my mind, deeply admirable. Very much so. And my only concern was I was just hoping that people didn't misinterpret my snark and sarcasm as aimed at, oh, look at these crappy, these companies are real companies and all those crappy SaaS companies are just flashes in the pan. No, I don't believe that members of the Fortune 500 are flash in the pan companies with a couple of notable exceptions who I will not name now because I might want some of them on this podcast someday. No, it's – the concern that I have is that everyone's work is valuable. Everyone's work is important. And even – what I'm seeing historically and something that you've nailed is a certain lack of stories

that apply to some of those organizations that are, for lack of a better term, ossified into their current process model, where there's no clear path for them to break into, quote unquote, doing the DevOps. Yeah, and the business frame and the imperative for it is incredible, right? Tesla is now offering auto insurance bundled into the car. Banks are now having to compete with Apple. It is breathtaking to see how competitive the marketplace is and the need to understand the customer and deliver value to them quickly and to be able to experiment and innovate and out-innovate the competition. I don't think there's any business leader on the planet who doesn't understand that software is eating the world and that any level of investment they do involves software at some level. And so the question for them is,

how do they get educated enough to invest and manage and lead competently? So to me, it really is like the sleeping giant awakening. And it's my genuine belief is that the next 50 years, as much value as the tech giants have created, Facebook, Amazon, Netflix, Google, Microsoft, they've generated trillions of dollars of economic value. When we can get 18 million developers as productive as an engineer at a tech giant is, that will generate tens of trillions of dollars of economic value per year. And so when you generate that much economic activity, all problems become solvable.

You look at climate change. You take a look at the disparity between rich and poor. All things can be fixed when you significantly change the economy in this way. So I'm extremely hopeful, and I know that the need for things like DevOps are urgent and important. I guess that that's probably the best way of framing this. So you wrote one version that was aimed at operators back in 2013. This one was aimed at developers and effectively retells and clarifies an awful lot of the same points. As a historical ops person, I didn't feel left behind by the Unicorn project, despite not being its target market. So I guess the question on everyone's mind,

are you planning on doing a third iteration? And if so, for what demographic? Yeah, nothing at this point. But there is one thing that I'm interested in, which is kind of the role of business leaders. And, you know, Sarah is kind of an interesting villain. One of my favorite pieces of feedback during the review process was I didn't think I could ever hate Sarah more. And yet I did find myself, you know, finding her even to be more loathsome than before. She's actually based on a real person, someone that I worked with. That's the best part is these characters are relatable enough

that everyone can map people they know onto various aspects of them but can't ever disclose the entire list in public because that apparently has career consequences. That's right. Yes, I will not say who the character is based on, but there's kind of in the last scene of the book that went to print, Sarah has an interesting interaction with Maxine, or they meet for lunch. And I think the line was, and it wasn some friends and colleagues is just understand what, why does Sarah act the way she does? I think we've all worked with someone like her. And, you know, there are some that are genuinely bad actors, but I think a lot of them are doing something, you know, based on genuine, you know, real motives.

And it was just, it would be fun, I thought, to do something with Elizabeth Hendrickson, who we just had to start having a conversation. Like, what does she read? What is her background? What is she good at? What does her resume look like? And what caused her to, who in technology treated her so badly that she treats technology so badly? And why does she behave the way she does? And so I think she reads a lot of strategy books.

I think she is not a great people manager. I think she maybe have come from the mergers and acquisition route that kind of viewed people as fungible. And I think she is definitely a creature of economics. It was kind of lured by an external investor about how good it can be if you can extract value out of the company, squeeze every bit of – sweat every asset and sell the company for parts. So I would just love to have a better understanding of why people – when people say they worked with someone like a Sarah, is there a commonality to that? And can we better understand Sarah so that we can both work with her and also, you know, compete better against her, you know, in our own organizations?

I think that's probably a question best left for people to figure out on their own in a circumstance where I can't possibly be blamed for it. That can be arranged, Mr. Quinn. All right. Well, if people want to learn more about your thoughts, ideas, feelings around these things, or of course, to buy the book, where can they find you? You know, if you're interested in the ideas that are in the Unicorn Project, I would point you to all of the freely available videos on YouTube.

Just Google DevOps Enterprise Summit and anything that's on the plenary stage are specifically chosen stories that very much inform the Unicorn Project. And the best way to reach me is probably on Twitter. I'm real Gene Kim on Twitter and feel free to just at mention me or DM me. Happy to be reach out and whatever way you can find me. You know where the hate mail goes then. Gene, thank you so much for taking the time to speak with me. I appreciate it. And Corey, likewise and again, thank you so much

for your unflinching feedback on the book and I hope you see your fingerprints all over it and I'm just so delighted with the way it came out. So thanks to you, Corey. As soon as my signed copy shows up, you'll be the first to know. Consider it done. Excellent, Corey. As soon as my signed copy shows up, you'll be the first to know. Consider it done. Excellent. Excellent.

That's the trick is to ask people something for something in a scenario in which they cannot possibly say no. Gene Kim, a multiple award-winning CTO, researcher, and author, pick up his new book, the Wall Street Journal bestselling The Unicorn Project. I'm cloud economist Corey Quinn, and this is Screaming in the Cloud. pick up his new book, the Wall Street Journal bestselling The Unicorn Project. I'm cloud economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on Apple Podcasts. If you've hated this podcast,

please leave a five-star review on Apple Podcasts and leave a compelling comment. This has been this week's episode of Screaming in the Cloud. You can also find more Corey at ScreamingInTheCloud.com or wherever Fine Snark is sold. This has been a humble pod production stay humble