Screaming in the Cloud - Driving State-of-the-Art DevOps with Nathen Harvey

Episode Date: May 20, 2021

About NathenNathen Harvey, Cloud Developer Advocate at Google, helps the community understand and apply DevOps and SRE practices in the cloud.  Nathen formerly led the Chef community, co-hosted the Food Fight Show, and managed operations and infrastructure for a diverse range of web applications. Links:cloud.google.com/devops: https://cloud.google.com/devops97 Things every Cloud Engineer Should Know: https://shop.aer.io/oreilly/p/97-things-every/9781492076735-9149Twitter: https://twitter.com/nathenharvey

Hello, and welcome to Screaming in the Cloud, with your host, Chief Cloud Economist at the Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud. This episode is sponsored in part by Thinkst. This is going to take a minute to explain, so bear with me.

I linked against an early version of their tool, canarytokens.org, in the very early days of my newsletter. And what it does is relatively simple and straightforward. It winds up embedding credentials, files, that sort of thing, in various parts of your environment, wherever you want to. It gives you fake AWS API credentials, for example. And the only thing that these things do is alert you whenever someone attempts to use those things. It's an awesome approach. I've used something similar for years. Check them out. But wait, there's more. They also have an enterprise option that you should be very much aware of. Canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your

entire environment. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets NMAP scanned, or someone attempts to log into it or access files on it, you get instant alerts. It's awesome. If you don't do something like this, you're likely to find out that you've gotten breached the hard way. Take a look at this. It's one of those few things that I look at and say, wow, that is an amazing idea. I love it. That's canarytokens.org and canary.tools. The first one is free. The second one is enterprising. Take a look. I'm a big fan of this. More from them in the coming weeks. at Lumigo. If you've built anything from serverless, you know that if there's one thing that can be said universally about these applications, it's that it turns every outage into a murder mystery. Lumigo helps make sense of all of the various functions that wind up tying together to build applications. It offers one-click distributed tracing so you can effortlessly find

and fix issues in your serverless and microservices environment. You've created more problems for yourself? Make one of them go away. To learn more, visit Lumigo.io. Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined this week by Nathan Harvey, a cloud developer advocate at a small startup called Google. Nathan, thank you for joining me. Hey, Corey. It's really great to be here. We'll get to the Google bits in a little, but first, I want to start back in the beginning with your origin story. It turns out, for example, that you were at a lot of places, and the first thing going through your history that I really

recognized was way back at the end of 2009, where you were the web operations manager at Custom Ink. They're a t-shirt company and other apparel that I've been using for three years now for the charity t-shirt drive here, as well as other sundry things. Longtime listeners of the show might remember we had Ken Collins on to talk about Ruby in Lambda and other horrifying things before it was cool. Yes, indeed. I was at Custom Ink. And you talk about them being a t-shirt company, and I don't know, maybe I'm still a shill for Custom Ink,

but I really look at them as an experience company. And you've recognized it yourself. They produce and help people really encourage that group and experiences and really drive, what does it mean to connect with other humans? And how can you do that through custom apparel to me that's what custom make has always been about they're not selling t-shirts they are selling an experience in my case i view them as a t-shirt company because let's be fair here i wind up doing charity t-shirt drives and they've always been extremely supportive

of well there's really no other way to put this, my ridiculous nonsense. The year I had linked campaigns of the AMI has three syllables shirt that was on sale. And then for the Amazonians, AMI is how it's pronounced instead. And that one was $10 more because there's a price to being wrong at all proceeds, of course, went to benefit the charity of the year. And was a fun thing and i talked to a number of other folks on this and they look at me very strangely and custom ink didn't even blink right right absolutely absolutely and yes they sell lots of other apparel but for whatever reason it's it seems that sending out complicated multiple options of things that need each hit minimum order quantities to print during a fundraiser and the fact that i'd have to deal with the money because they just wind up sending it over directly.

It's just easier. It's one of those things where back when I was a single person doing this stuff, I didn't have to worry about it. Now that I've grown and my needs have multiplied, I still like doing business with them. They're great folks. Absolutely. And that's exactly what I mean by it. They've sold you on an experience. That's why you continue to do business with them. It's not just because of the t-shirts. It's the whole package that goes along with it. And then in 2012, the world didn't end, but yours kind of did because you stopped working at Custom Inc. and went to another company called Chef. You were there for a little over six years. You started off as a community director and then became the VP of community development.

And I think you did an amazing job. But first, tell me about that. Then I will give my hot take. All right, great. I'm always up for the hot takes. So listen, Chef was an amazing community of people. Oh, it was also a company. And so I really fell in love with, while I was at Custom Ink, actually, we were using Chef, and I fell in love with the community. And I was doing a lot of community support, running my own podcast or participating with some co-hosts on a podcast called The Food Fight Show back in the day. It was all about Chef, running meetups and so forth. And at one point I decided, you know, what I should do maybe is stop being on call and start supporting this community full time. And that's exactly what I did. I went to Chef, and yes, as you mentioned,

spent just over six years there, or just about six years there, and it was really, really an incredible time. Lots of hugs to be given, and just a great community in the DevOps space. I took a somewhat, I guess, agreeing but disagreeing position. I was on the puppet side of the configuration management debate and it was challenging. And then, ah, I was one of the very early developers

behind SaltStack because clearly the problem with all of these things was that no one had written it correctly and we were going to fix that. And it turns out, no, no, the problem was customers the whole time, but that's a separate debate. So I was never in the Chef ecosystem.

That was the one system I never really touched in anger. And it's easy to turn this into a, oh, you folks were the competition, despite the fact I never actually worked directly for either of those companies. But it was never like that, because our real enemies were people configuring things by hand, for one, because that's unnecessary toil, don't do it. And it was also just such an uplifting sense of community. Some of the best people I knew were in the Chef ecosystem, in the Chef orbit for a while there on some level. And this is something I'd love to get your thoughts on. It seems that a failure mode that Chef exhibited was hiring directly from its community, where if someone was a big fan of Chef, start a stopwatch, they're going to be working there before the month is out. I think that Chef, the company, definitely pulled a lot of community

members into the organization. And frankly, when the company started, that was really, really great because it was an early startup. And as the company grew, it was still wonderful, of course, to pull in people from the community to really help drive the future direction, how our customers are using it. But like you said, there is a little bit of a challenge or a concern when you start pulling too many of your most vocal supporters out of the community and putting them into the company, sometimes in places or roles where they didn't have the opportunity to be as vocal, as big a champions for the product, for the services. I think on some level it was, again, it helps to have people who are passionate about the product,

you know, working there. But on the other, it felt like over time it wound up weakening the community in some respects, just because everyone who worked there eventually found themselves in a scenario of, well, I work here, it's what we do, and now I have to say nice things. It winds up weakening the grassroots story. There's definitely some truth to that, but I think there's also some truth to just the evolution of community. As you went from a community in the early days where there were a lot of contributors

to over time, gratefully so, the community that, or sort of the proportion of the community that were consumers of Chef versus contributors to Chef, like that balance changed. And so you had a lot more customers using the product. So I don't disagree with you,

but I do think that it's part of the natural evolution of community as well. And all things must end. And of course, Chef got acquired, I believe, after you left. So, I mean, at that point you left, they were rudderless, and what else were they to do? And you went to Google. And that is always an interesting story, because Google's community interaction before the time you wound up there and after, I don't know that you were necessarily the proximate cause, but I'm going to hang that around your neck because it's all been a positive change since then, look radically different. Yeah. Well, thank you. It is definitely not something that I should wear

or carry alone, but going to Google was an interesting choice for me. I recognize that. And, you know, honestly, Corey, one of the things that drove me to Google was a good friend of mine, Seth Vargo. And just to kind of tie the complete through line here, Seth and I worked together at Custom Ink. We worked together at Chef.

He left Chef and went to Hashi and then went to Google. And the day after I knew that he was going to Google, I called him up and I said, Seth, come on. Google's so big. Why? Why? And how? I don't understand. I don't understand the move. I asked him many of the same questions back in episode three of this show. He was a very early guest when I was scared, speechless, having conversations. It's improved since then, a couple hundred in when I was scared, speechless, having conversations. It's improved since then, a couple hundred in. But yeah, very friendly, very open, very warm.

Yeah. And you know, why are you at Google was sort of the next follow-on question there in that era. Yes, indeed. And I do think that Google, and specifically Google Cloud, has really taken to heart this idea that there's a lot that we can learn from each other. And I don't mean from each other within Google, although of course we can learn a lot from each other, but we can also learn a lot from our community, from our customer base. How are they using Google Cloud? How are they using technology to drive their

business forward? These are all things that we can learn. It turns out not every company is Google, and that's a good thing. Not every company should be Google or Google-sized and certainly don't have Google customers. And I think that it's really important that we recognize that when we work with a customer, they're the experts in their customers and in their systems and so forth.

A lot has changed with Google's approach to, well, basically everything. It turns out that when you're a company that's, what, 26 years old now, 27, something like that, starting with humble beginnings and then becoming a trillion-dollar entity, things change, culture change, your community changes, what you do changes. And that becomes something that I think is not necessarily fully appreciated or fully understood in some corners. But then 2018 hit. You went to Google.

What did you do then? Because it is such a large company that it is very difficult to know what any individual is up to there. And the primary means that I engage in the DevRel community space, specifically via aggressively shitposting on Twitter, isn't really your means of interaction with the community. So from that particular point of view, it's, oh yeah, he went to Google and no one ever heard from him again. What is it you'd say it is you do there? Yeah, so for sure. What I do here as a cloud advocate is I really focus in on kind of two areas, I would say. DevOps, and I recognize that that is a terrible, terrible word because

when I say it, we all think of different things, but I definitely focus on the DevOps. And then SRE practices as well, or site reliability engineering. And specifically, what I work on is how do we bring the principles and practices of DevOps, of SRE into our customer base and into the community at large? How do we drive what is the state of the art? How do we approach these particular topics? And so that's really what I've been focused on since joining Google. Well, frankly, I was focused on that wallet chef as well, maybe without the SRE bend so much, but certainly at Google, SRE comes in. But it's always,

for the past decade for me, been about DevOps and how do we use technology to align the humans and work towards the business outcomes that we're driving for? And business outcomes become an interesting story in the world of cloud because it distills down for a cloud service provider is we would like people to use our cloud, more of it, in perpetuity. It is not a complicated business model, if I can be direct, because business models inherently are not.

Whatever it is your company does, we would like you to do it here. And that turns into a bunch of differentiated services across the spectrum, in some cases hilariously so, when it turns into basically pick an English word, and there's a 50-50 shot that that's part of a service name somewhere. But a lot of it distills down to baseline distinct primitives. You're talking about the DevOps aspect of it, which is, we talk about, is it culture? Is it tools? No, it's a means to sell conferences and books

and things like that. But what is it in the context of a cloud service provider? Specifically Google, because let's be clear here, DevOps apparently for other providers is Azure DevOps. That's right, it's a service name and DevOps Guru on the AWS side, because everything is terrible. Absolutely. Look, I think that I used to start that the only DevOps tool was the manager of DevOps. But the truth is that DevOps is, it is tooling and it is culture. And to separate the two is really a fool's errand. I think that, you know, your tooling amplifies your culture,

your culture amplifies your tooling. Together, this is how we make progress. Now, when it comes to Google, what do we mean when we say DevOps? Well, one of the good things is shortly after I joined Google Cloud, Google Cloud acquired DORA, the DevOps Research and Assessment Organization. Jez Humble and Dr. Nicole Forsgren. And then for all intents and purposes, they Googled it relatively shortly thereafter,

by which I mean we never really heard from DORA again. In 2020, the state of DevOps report didn't exist, which was what they were famous for doing. And it was, oh, yep, that's a Google acquisition. All right. Is that what happened? Did I miss some nuance there? Yeah, let's talk about that. So first, you're right. It was Dr. Nicole Forsgren who founded Dora. So when the acquisition happened, she came along to Google Cloud. Jess Humble came along through that acquisition as well. And frankly, what happened in 2020, well, Corey, I don't know if you noticed, but there was a lot happening in 2020, much of it not very good. I think when we

look at the global scale, like 2020 was not a great year for us. It was a rebuilding year. All right, fair enough, fair enough, a rebuilding year. But so here's what happened with Dora. Quite frankly, we, Google Cloud, continue to invest in that research program. And really, in a sense, 2020 was a rebuilding year in that our focus was really about how do we help our customers and our community apply the lessons of Dora. And so one of the things that we've done is we've released much of the research under cloud.google.com slash DevOps, including right there, a DevOps quick check,

where as a team, you can go in and using the metrics and the research program from Dora, you can assess, are you a low, medium, high, or elite performer? And then beyond that assessment, actually use the research to help you identify which capabilities should my team invest in improving. So those capabilities might be technical capabilities, things like continuous integration. It might be process or measurement capabilities, or in fact, cultural capabilities. So all of these

capabilities come together to help you improve your overall software delivery and operations performance. And so in 2020, the big thing that we did was release and continue to update this quick check, release the research, make it fully available. We've also spent some time internally on the program that is not super interesting to talk about on the podcast. But the other thing that we did in 2020 with the Dora Research Program was update the ROI research, the return on investment research. This is something that maybe your listeners don't care about, but their managers might care about, their CIOs, CTOs, CFOs might care about.

How do we get money back on this transformation thing? And the research paper really digs into exactly that. How do we measure that? What returns can we expect? And so forth. So that was released in 2020. I have a whole bunch of angry thoughts about a lot of takes in that space,

but this is neither the time nor the place for me to begin renting incoherently for an hour and a half. But yeah, I get that. It was a year that was off, and now you're doing it again, apparently, in 2021. And the one thing I never really saw historically, because I don't know if I'm playing in the wrong environments, or I'm certainly not the target audience now, if I ever really was. But most years I missed the release of the survey of where people can go to fill in these questions. I would be interested to know where that is now. And then I would be interested to know, how have you been socializing that in the past? In other words, where are you finding these people? Yeah, for sure. So the place you go to find the survey right now is cloud.google.com slash DevOps. You'll find a button on the page that says something like take the survey or take the 2021

survey. And what we've done in the past and really what Dora has done in the past is use a number of different ways to get out information about the survey when the survey is open and so forth. Primarily Twitter, but also we have partners and Dora historically has used partners as well to help share that the survey itself is open. So I would absolutely recommend that you go and check out the survey because I'll tell you what, one of the things that's really interesting, Corey,

over the years, I've talked to a bunch of people that have taken the survey and that have read the State of DevOps report that comes out each year. And some of the consistent feedback I've heard from folks is that simply taking the survey and considering the questions that are asked as part of the survey gives great insight immediately into how their team can improve. What things, what capabilities are they lacking? Or what capabilities are they doing really well with and they don't need to make investments on? They can immediately see that just by answering

and carefully considering the questions that are part of the survey. This episode is sponsored by ExtraHop. ExtraHop provides threat detection and response for the enterprise, not the starship. On-prem security doesn't translate well to cloud or multi-cloud environments, and that's not even counting IoT. ExtraHop automatically discovers everything inside the perimeter, including your cloud workloads and IoT devices, detects these threats up to 35% faster and helps you act immediately. Ask for a free trial of detection and response for AWS today at extrahop.com slash trial. Very often, in some cases, looking at things like maturity models and the like, the actual report is less valuable than the exercise of filling it out and going through the process. I mean, compliance reports, audit frameworks, etc. often lead to the same outcomes. The question is, are you taking it seriously or are you one of those folks who is filling out a survey because do this and you'll have the enter to win a $25 gift card somewhere,

probably Blockbuster, because it no longer exists. I get those in my email constantly of, yeah, give half an hour of your time in return for some paltry chance to win something. No, I have a job to do. And I worry if at that level of that approach, who are you actually getting that's going to sit down and fill this thing out? That said, the state of DevOps reports have been for a long time sort of the gold standard in this space. And I would encourage people listening to this to absolutely take the time to fill that out. cloud.google.com slash DevOps. I'm looking forward to seeing what comes out of it. And I love it because of the casual shade you can use to throw at other companies too. Like, are you an elite team with the implicit baked in sentiment being, no, you're not, but I want to hear you say it.

Yeah. One of the things that really sets Dora apart also, I think, is just the, well, two of the things, I guess I would say. One is the length of time that the research program has been running. It's going on seven years now that this research program has been running. And so given that, you have tens of thousands of IT professionals that have taken the survey and provided insights into sort of what's the state of our industry today and where are we heading. But it's also an academically rigorous survey. The survey and the research itself has always and continues to be completely platform and program agnostic. This is not a survey about Google Cloud. This is not a survey where we're trying to help understand exactly what products on Google Cloud should you use in order to be an elite performer. No, that's not what this is about. It is about truly the capabilities that your team needs in order to improve their software delivery and operations performance. And I think that's really, really important. Dr. Nicole Forsgren, who founded DORA,

she didn't come up with all of these ideas. Hey, I think that you get better by doing this. No, instead, she researched all of these ideas. She got this input from across organizations of all sizes, organizations in every industry. And that, I think think really sets it apart. And our ability to really stay committed to that academic rigor and the platform agnostic approach to capturing and investigating these capabilities, I think is so important to this research. And again, this is why you should participate in the survey, because you truly are going to help us move the state of the art of our industry. Now, historically, there's been challenge where the mantle of thought leadership in conjunction with Google have intersected. Because there's a common trope, historical, and I think that it is no longer accurately true.

It's an easy cheap shot, but I don't think it holds water like it once did, where, oh, Googler, that's another word for condescending. And there is an element of, oh, this is how DevOps should be. This is how we're moving things forward. How do you distance it from being, Google says you should do it like this? Yeah, this comes up a lot. And frankly, I get in conversations with customers asking, how does Google do this? How does Google do that? And my answer always is, you know, I can tell you how Google does something, and that might be interesting. But the fact is, it's not much more than that, much more than interesting. Because what really matters is how are you going to do this? How are you going to improve your outcomes? Whether that's

you're delivering faster, you're delivering more reliable, you're running more reliable services. You're the experts, as I mentioned earlier, you're the experts in your teams, in your technology, in your customers. So I'm here to learn right along with you. How are you going to do this? How are you going to improve? Knowing how Google does it, it's interesting, but it's not the path that you will follow. I think that that's one of those statements that can't ever be outright stated on a marketing website somewhere.

It's one of those shifts that you have to live. And I think that Google's done a pretty decent job of that. The condescending Googler jokes are dated at this point. And it's not because there was ever an ad campaign about we're not condescending anymore. It was a very subtle shift in the way that Google spoke to its customers, spoke about themselves.

I no longer feel the need to stand up in a blinding white rage in the Q&A portion of conference talks given by Google employees. A lot has changed, and it's not one thing that I can point to. It's a bunch of different things that all add up to dramatically shifted credibility models. Realistically, I feel like that is a microcosm of a DevOps transformation. It's not a tool. It's not a single person being hired. It's not, we're taking an Agile class for three days for all of engineering, and now things will be better. It's a whole bunch of sustained work with a whole bunch of thought and effort into

making it an actual transformation, which is such a toxically overloaded term, I dare not use it. Indeed. And there's no maturity model that shows, are you there yet? And it is something that you don't flip on or flip off like a switch, right? It doesn't happen overnight. It takes iteration and iterative change across the entire organization. And just like every change that you have across an organization, there are places where it's going better than other places. And how do you learn from that? I think that's really, really important. And to recognize and to bring some of that humility to the table is so, so important. So what's interesting about folks that I talk to on this show, well, there are many

interesting things, but one of the interesting things is that they have a higher rate than the general population of having at one point in their careers written a book of some form. And you are, of course, no exception. You and Emily Freeman co-authored recently a book entitled 97 Things Every Cloud Engineer Should Know. And it's interesting because it only has one nine in the title. Okay, that is at least an attempt at being available. I know it's available wherever most books are sold. Tell me more. Yeah, so first let's start with the 97.

Why 97? Corey, I don't know if you know this or not, but 100% is the wrong reliability target for just about everything. So 97, that feels achievable. It also feels like three people said they would do it and then backed out at the last minute, but that's my cynicism speaking. Well, for better or worse, O'Reilly has a whole 97 things series, and this is part of it.

So it is, in fact, 97 things. The other thing that I think is really important about the book, you mentioned that Emily and I wrote it, and the beauty is, for a long time, I've wanted to have written a book, and I have never wanted to be writing a book. That is what every author has ever said. It's no one wants to write a book, they want to have written one. And then you get a couple of beers into people and ask them, so I'm debating writing a book, should I? The response is no, absolutely not. No. And at some point, when you calm them down again and they stop screaming, they tell you the horrifying stories

and you realize, oh, wow, I really never want to write a book. Yes. Well, the beauty of 97 Things and this book in particular, or the whole series really is its subtitle is Collective Wisdom from the Experts. So in fact, we had over 80 different contributors sharing things that other cloud engineers should know. And I think this is also really, really important because having 80 plus contributors to this book gave us not 97 things that Emily and Nathan think every cloud engineer should know, but instead a wide variety of experience levels, a wide variety of perspectives. And so I think that that is a thing that makes the book really powerful. It also means that those 80-some folks that contributed to the book had to write a very

short article. So of course, with 80 authors and 97 things, the book is not, it doesn't weigh 27 pounds, right? It's less than 300 pages long where you get these 97 tidbits. But really the hope and the intent behind the book is to give you an idea about what should you explore deeper. And just as importantly, who are some people that you can maybe reach out to and talk to about a particular topic, a particular thing that a cloud engineer should know? Here are 80 people that are here helping you and really cheering you on as you take this journey into cloud engineering. I think there's something to be said from having the stories for this is what we do, this is how we do it, but the lessons learned stories, those are the great ones, and it's harder to get people on stage to talk about that without turning into, and that's how we snagged victory from the jaws of defeat.

No one ever gets on stage and says, and that's why the entire project was a boondoggle, and four years later, we're still struggling to recover. Especially, you know, publicly traded companies tend not to say those things, but it's true. You wind up with people getting on stage and talking instead about these high-level, amazing things that they've done, and the project went flawlessly, and you turn to the person next to you and say, yeah, I wish I could work in a place like that. And they say, yeah, me too. And you check, and they work at the same place as the presenter. Because it's conference ware. It's never a real story. I'm hoping that these stories go a bit more in depth into the nitty-gritty of what worked, what didn't work, and it's not always author as hero protagonist.

Oh, you will definitely find that in this book. These are true stories. These are stories of pain, of heartache, of victory and success, and learnings along the way. Absolutely. And frankly, in the DevOps space, we do an okay job of talking openly about our failures. We often talk about things that we tried that went wrong or epic failures in our systems and then how we recovered from them. And yes, oftentimes those stories have a great sort of storybook ending to them.

But there's a lot of truth in a lot of those stories as well, because we all know that no organization is uniformly good at everything. That may be the stories that they want to share most, but there's some truth in those stories that hopefully we can find. And certainly in this book, you will find the good, the bad, the ugly, the learnings, and all of the lessons there. Where can people find it if they want to buy it? Oh, you know, you can find it wherever you buy books. There are, of course, eBooks,

O'Reilly's website, you know. Wherever. Fine books are pirated. Yes. That's a good place to go for books. Yeah, for sure. And we'll, of course, throw a link to the book in the show notes. Thank you so much for taking the time to speak with me. If people want to learn more about the rest of what you're up to, how you're thinking about it, what wise wisdom you have for the rest of us, where can they find you other than the book? Yeah. A great place to reach out to me is on Twitter. I am at Nathan

Harvey, but I should warn you, my father misspelled my name. So it's N-A-T-H-E-N-H-A-R-V-E-Y. So you can find me on Twitter, reach out to me there. And we will, of course, include links to all of that in the show notes as well. Thank you so much for speaking to me today. I really appreciate it. Thank you, Corey. It's been a pleasure. Nathan Harvey, cloud developer advocate at Google. I'm cloud economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice. Whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment telling me that I'm completely wrong. You can instantly get DevOps in your environment if I only purchase whatever crap it is your company sells.

If your AWS bill keeps rising and your blood pressure is doing the same, then you need the Duck Bill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duck Bill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started. This has been a HumblePod production. Stay humble.