Screaming in the Cloud - Managing Humans with Charity Majors

Episode Date: September 17, 2020

Links Referenced: Honeycomb: https://www.honeycomb.io/Personal Blog: https://charity.wtf/Honeycomb Blog: https://www.honeycomb.io/blog

Hello, and welcome to Screaming in the Cloud, with your host, cloud economist Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud. well outside those boundaries. So it's difficult to understand what's actually happening. What Catchpoint does is makes it easier for enterprises to detect, identify, and of course validate how reachable their application is, and of course, how happy their users are. It helps you get visibility into reachability, availability, performance, reliability, and of course, absorbency, because

we'll throw that one in too. And it's used by a bunch of interesting companies you may have heard of, like, you know, Google, Verizon, Oracle, but don't hold that against them, and many more. To learn more, visit www.catchpoint.com and tell them Corey sent you. Wait for the wince. This episode is brought to you by Trend Micro Cloud One, a security services platform for organizations building in the cloud. I know you're thinking that that's a mouthful because it is, but what's easier to say? I'm glad we have Trend Micro Cloud One, a security services platform for organizations building in the cloud, or hey, bad news, it's gonna be a few more weeks. I kind of forgot about that security thing. I thought so. Trend Micro Cloud One is an automated,

flexible, all-in-one solution that protects your workflows and containers with cloud-native security. Identify and resolve security issues earlier in the pipeline and access your cloud environment sooner with full visibility so you can get back to what you do best, which is generally building great applications. Discover Trend Micro Cloud One, a security services platform for organizations building in the cloud, at trendmicro.com slash screaming. Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined this week by someone who needs no introduction, so I'm going to give her as little of one as possible. Charity majors, founder and CTO of Honeycomb,

but famous on the internet long before that. Welcome to the show. Thank you. It's nice to be here. We've been trying to do this for a while. We have, and it seems that scheduling never quite works out until suddenly everyone is trapped at home and eventually we both run out of excuses. Yes. Well done. So what are you up to these days? Well, I feel like I've turned into a den mother for a bunch of very anxious camp goers. Through this whole COVID thing, I have always hated repeating myself. And that's something that I feel like once you go into management, you kind of have to get over.

But during the pandemic, repeating myself honestly has come to feel like my one and only job. In what sense? Everything's going to be okay. You're doing everything we could have asked of you. You're doing all right. Everything you're doing is enough. You are enough. You are beautiful.

Please go home. Get some sleep. You know, have some soup. Take care of yourself. Take care of your loved ones. Yes, you are working hard enough. Yes, you are doing your best. Your best is enough. You know, that kind of thing. I feel like something that I kind of said off the cuff in our all hands a few weeks ago was just that, you know,

brain weasels love authority figures and they love reassurances from authority figures. And it's so true. We are adults, you know, we can be as anti-authoritarian as we want to be it's still there's something very reassuring when we're very anxious about everything there's something reassuring about somebody in a position of authority just telling you that you're good enough and everything's okay and for the first time in my life I find myself awkwardly in that position of authority and so just like trying to use it to reassure people as much as possible that what they're doing is good, they are fine, and that they can take a break. Excuse me, my main job right now. Management seems to be one of those interesting areas of expertise, because to be blunt,

a lot of the people who write these books and tell you how to be a good manager... I hope expertise was said with scare quotes. Oh, absolutely it was, because all of these people you read, oh, you're, you're giving me advice on how to manage. Great. What do you do? Oh, you haven't managed anyone in 20 years. Instead, you've been writing books on how to manage people. Yes. Or, or better yet, there are a VP of something, something, and then you hear around the grapevine, they are terrible at their job. Exactly. I can think of several people off the cuff that we are not naming because lawsuits are passe. Yes. But you're right. I see something similar to this when I was getting started in freelancer circles where people are focusing on specific

areas that they want to tackle. It's, oh, I'm going to learn how to do positioning as a freelancer. And it turns into, huh, seems I can make more money teaching other freelancers how to do positioning, for example, or how to do sales than doing those things myself. I wish that people were like legally mandated to present context with their advice. This advice is true within the narrow circumstances of these things that I've experienced before, you know, because people sound very authoritative. They're like, you should do this. And they don't bother to mention that they come from, you know, some completely different context where we're talking to completely different people. Context is everything when it comes to advice. Absolutely. I mean, one of the things I've stayed away from for a long time has been management

advice simply because my only management training was having a bunch of really terrible managers and then doing the exact opposite, which it turns out gets you further than you might expect. But then I started this thing. And when I took on a business partner, Mike, I stopped managing anyone because what I do requires me to be more than a little bit self-promotional.

And that is the recipe for a terrible manager, in my experience. You have to build other people up, but I have to be the loud, obnoxious one all the time. And you hate that, don't you? I can do it, but it's one of those things where I can't do it while simultaneously lifting up others. You do it and you're very good at it. Well, thank you.

I appreciate that. I am the loudest and most obnoxious out of the entire flock. Hearts. Exactly. So one of the questions I see is given that you have been able to walk a tighter line than I have because you still manage teams and you tend to focus on the empathy piece quite a bit, but you also have never been one to shy away from sharing your opinion online in various ways that I hold a deep personal affinity for. Some might call them bombastic. I call them enthusiastic. Sure. We only speak in grand statements.

Yeah. Well, I mean, part of it is I don't really manage a team right now. I manage three people, Liz, Emily, and Megan. They've graduated. They don't need my bullshit, you know? So I honestly feel like if we're talking to engineering managers here, context, right? I would say that your job is to craft a team, right? Your job is to craft and coach a team. Like you're not hiring individuals. You're not hiring people. You're curating a team.

And the kind of like just emotional intensity that you need to have with that team, like you're in it with them like every day. And I'm not doing that right now. And so most of the, when I'm giving, you know, my quote unquote advice, it's mostly me thinking back to the days where I was doing that and drawing on my experiences there. That plus I don't really like to talk about the people that I'm working with now. It feels kind of rude and kind of nonconsensual to show our dirty laundry. I'll talk about them someday. But right now, I honestly try not to talk very much about people at Honeycomb. Well, you also have entered one of those very interesting and rarefied positions where the people who report to you are themselves senior enough and have the context to be able to need

guidance in a very different way versus junior people or folks who are new to the workforce in one form or another often don't have that. And the way to manage those people, in my experience, has been radically different. Yeah, because I don't really have to think about the careers of the people who report to me much. I trust them for that. You know, they're in a good spot. It's much less of a, you help them grow in their skillset to get to senior engineer and then help them be effective in a larger, and it's much more just like, you know, they're grown, they're baked, they're cool. So I need to just like give them the information that they need. But managing managers is very different than managing engineers or managing ICs of any sort.

The visual that I always get is when you're doing the work, you're assembling a basket with your hands. When you're managing people, you're like walking up and down, watching people assemble baskets, and you're like inspecting and peering. When you're managing managers, you're like standing over on the other side of the pier, squinting and trying to see how the baskets are coming out, and then waving at them with flags to try and signal what they should be telling their teams to do. Like you're so far removed from the actual being able to do the thing that it's terrifying, honestly.

It's terrifying and it's a whole different skill set. And it involves a lot of letting go of the ego. You know, we spend our whole career trying to get good at this skill set, right? Being good at engineering, being good at getting things done, making things. And then we get drafted into management and it's not even like a sidestep career-wise. It's like you're starting over at zero, but you still get to sit there and enviously watch

the people doing the thing that you love to do, right? And you're trying to wall that part of yourself off. And I don't know. It's weird, man. I don't understand why people want to be managers. My honest assessment of why people want management is that they're told that that's the thing you do next. It's, well, when you've conquered the IC scale, then you get promoted to management. But it's a parallel track.

It's not something in most shops that should be considered a promotion. Or even more perniciously, they feel disempowered. You know, what did Daniel Pink say that we want out of life? You know, we want autonomy, mastery, and meaning. That's what we want from our work. So many people show up to work every day and maybe mastery, maybe they're getting mastery, but they may or may not have meaning, but autonomy, like they don't have it. And they want it.

They crave it because we all do. And they're told or they intuit for themselves that the way that you get more control over what you're doing is you go into management. And this is so, I mean, this is why I went into management. I wanted a bigger say over what I was doing. No shame. I'm not sharing the shame anyone who's done any of these things. I think that even like wanting more power, I think that it's better to be honest about these impulses within ourselves and deal with them in a straightforward way than it is to, you know, learn to puppet that.

Well, you know, I just want to like help other people or, you know, it's better to be honest with yourself about why you want something because I don't think there's a wrong reason. I would like more money and the ability to tell people what to do. You know, that's fine. That said, you're going to be in for a rude awakening when you realize that managers don't actually get to tell people what to do, at least not in healthy organizations. Yes. But what made you think that most organizations are healthy ones? Oh, there's the rub, right? And so there's always this back and forth between wanting to give people advice for how to be a manager,

but it very quickly segues into how healthy is your organization? How healthy can you make it? How many other people are there who feel like you? Because here's the thing, your company doesn't exist. You show up every day and create it with your colleagues, right? There's nothing that exists other than that. We have all of the power that we need in our hands to change things radically if you just convince people that it's worth doing and enough of the people show up and create it this different way tomorrow, right? Absolutely. Which is a terrifying and awesome thing to realize.

It's quite the thought to wrap my head around, but you're right. And so many organizations where the only way to proceed is to go into management. Well, what if there could be parallel tracks, right? What if there could be, you know, technical tracks that go hand in hand with the managerial tracks? But the way that you need to sustain this is it can't just be managers giving power to engineers. It has to be engineers embodying that power and taking it for themselves too, right? Have you ever been on teams where, you know, managers want to give engineers more power and they won't take it? Because I have. It's fascinating to watch that dynamic play out and how the organization responds to that tells you an awful lot about that company. It really does. Now, I have this in my mind's eye,

I have this ideal, the way that engineering organizations should operate. This is a creative profession. Yes, there's a high basic skill level for entry, but we create things. And managing a large code base is fundamentally a creative act, which means that we're going to do our best work when we're empowered and inspired and motivated. limited and enumerated, less King George and more the articles of the constitution, right? Where it's like, okay, we're not supposed to like bow and scrape to our leaders anymore. They serve us and we are giving them these articulated powers and they have authority and power and responsibility to do these limited things insofar as they are doing them on behalf of the company as a whole. I think that's the appropriate healthy attitude towards management because there is this drift. This power will tend to drift towards management over time unless you're consciously pushing it back out to the ICs because so much of power is information flow. And managers, they are absolutely

privy to private information and they spend their days talking to people instead of heads down in code. So power is going to drift towards them unless you actively push it out. And something you said just there does resonate in that we have a high basic skill level required for these jobs. But if you look at a lot of historical companies, by the time that someone is coming in making, say, six figures at a manufacturing plant, they've been there for 20 years. They've worked their way up. Now a new grad gets that, and a lot of assumptions in a lot of these cultures is baked in that by the time you're paying someone a certain amount of money, they, for example, have the insight, wisdom, and political savvy not to call their boss an asshole in the all-hands meeting. That does not always play out the way that one would hope. No, it doesn't.

Something I've been thinking about a lot lately is the way we humans were hierarchical beings. If there's a ladder there, we instantly want to climb it. And there's this woman, Molly, who I work with, who she was an engineer. She graduated. She got offered a manager's job pretty quickly. She climbed the ladder. She had a series of high-profile executive jobs. 20 years she worked in this industry before she suddenly realized she hated it. She was miserable. All she wanted to do was write code, and she was jealous every day of the people that she worked with who were working as software engineers. She's like, why did I climb this ladder for 20 years? I hate everything about this. And I find that so sad. And there's so much pathos to that, you know, because we have this assumption, just like you said, that if there's a hierarchy, then the person

at the top of the hierarchy is superior to all who are below them in knowledge and judgment and experience and all these things. When in fact, I can now see from experience, it's just a different kind of work, right? Yes, when you have an organization of see from experience, it's just a different kind of work, right? Yes, when you have an organization of a certain size, it's like a human data structure, basically, that you need for an org chart. Because you need some people to have their heads upscaling the horizon and thinking about the one to two year, five year journey. And you need people down in the mid range with their heads at the six to 12 month range.

And you need people down in the weeds, but they're just different kinds of work. They're not better or worse than each other. And some of us are made happy by some types of this work and not other types. And if we could just strip it of all of the baggage of power and hierarchy, you know, if Molly Molly could just work on the things that brought her joy, this is the amazing thing that we take for granted in this industry. We get to work on things that bring us joy every day. This in and of itself sets us above 99% of all humans who have ever lived. And yet we throw away this joy and meaning that we found to chase the like being better than the people around us. And I feel like happiness should be given a greater weight.

In what you might be forgiven for mistaking for a blast from the past, today I want to talk about New Relic. They seem to be a relatively legacy monitoring company, and I would have agreed with that assessment up until relatively recently. But they did something a little out there. They reworked everything. They went open source, they made it so you can monitor your whole stack in one place, and most notably, from my perspective, they simplified their pricing into something that is much more affordable for almost everyone. There's even a free tier with one user and 100 gigs per month, totally free. Check it out at newrelic.com. You make excellent points. And I think it's time for a second rant on a different topic, if you'll indulge me yet further. Namely, this for a lot of people

relatively new to the workforce has been their first downturn. The last one was 10 years ago and change, and there have been a lot of people entering the workforce since then. What are you seeing as far as the downturn's impact on cloud and cloud-like services? Yeah. So a little bit of background. I love running mail servers. I started my career that way and I'm right there with you. And then it, post-fix was great. Right. At Linden Lab, I was the only one who could get IMAP working. We ran our own, trained our own spam assassin and clam AV like Antivirus. Dovecot or one of the older ones like Courier? Yeah. All that. Excellent. The first job I had

in industry was actually writing Qmail spam filters, the first spam filters in Qmail. You were on that side of the world. Yeah. So I love that shit. And when it was, how many years ago? Long years ago. I was still a teenager. And they wanted to move our mail to Google.

And I was like, no. And I wrote this long and passionate email about why it was terrible to outsource our email to Google. And fundamentally, I still wanted to be able to grab my mail spool. Anyway, TLDR, we moved to Google. I quickly saw the light and the world has been outsourcing more and more ever since. Right. I did it personally. I finally shut off my mail server in my rack and moved it to Google. And suddenly I had hours a week back that I was spending playing whack-a-mole with spammers. I know. It's amazing when you free up time from the optional things to work on the things that actually matter to you. It's a good trend. I'm all for it.

So the last time we had a downturn 10 years ago, we saw a lot more of this. And the way I feel about it is, God knows, I have tried. I see people out there going, I'm going to hire an observability team and they're going to build it from scratch for us. And I'm just like, oh, you sweet summer child. Right. But because we are not very good at quantifying the costs in human terms, this keeps happening. Right. It is kind of like a vanity project almost where people will just be like, you know, they want to have more teams underneath them or they want to feel like their workload is so special and so, you know, one of a kind and so bespoke. And so they need to do all these things themselves and they don't think critically about the

cost involved. Well, during downturns, all of that changes. And all of a sudden people are looking at the fact that they have a hiring freeze for the foreseeable future, if not letting some percentage of their team go. And they're still going to be responsible for the same amount of work, if not more. And suddenly, the value of focusing on their core business initiatives, the things that are differentiators for them as a business, becomes manifestly clear. And the argument for outsourcing major parts of their business, like email is still very core to everyone's business, right? Becomes very, very compelling. And so these great waves happen, you know, of retrenchment where people move things

to the cloud, move things to other providers. And then when the money comes back, they don't get reversed because it was always a stronger argument all along. It's just that we don't really know how to assign actual dollars and numbers to it. So it's still a very manager-driven, instinct, that like a lot of companies were kind of in the like large mid-sized tech companies, large enough to maybe start to preen and go, oh, we're so special, but actually not. Those people were planning on going and hiring teams to do observability.

And instead they're coming to us. And they're especially come to us because we're very much associated with the next way of doing things. And new things get a lot of heat, sometimes deservedly. God knows I've sh** over serverless enough. But one thing they do tend to be is cheaper. Buy a lot. Buy a lot. Orders of magnitude. Oh yeah, the most expensive thing at any company never shows up on the AWS bill, and that is the engineering time gone into wrangling

the services that show up on your AWS bill. Yes. Thank you. Absolutely true. And the newer ways of doing things are cheaper from a financial perspective. And it's almost like we don't have to think about that while times are good, right? We think about all the other things because think about all of the factors that go into making a technical or architectural decision. There's so many. What does your team like? What languages do you know? What stuff do you have provisioned? What other stuff do you already have written? You know, all this stuff. And it can become very muddled.

And in downturns, the pure cost of doing business and writing code and shipping value to customers stands out. And that's what it comes down to is value to customers. On some level, most of us have this insane thing beaten into our skulls where we talked about learning these things and how we start out going down these paths. We were generally hobbyists playing around and our time is free. Computers cost money. That's no longer true at all in any professional context. No, but because we love what we do, it still feels free to us. Exactly. So, oh, why would I pay extra for that managed service? I can build it myself on top of EC2 and then people set out to do it. And I enjoy it and I know all the ins and outs and blah, blah, blah, blah, blah. Yeah, exactly. Right. And half the time I come in and I

see an environment like that and I understand intrinsically why it is the way that it is and how it came to be that way. But it's not a good idea. And trying to convince folks of this is sort of an uphill battle because people have to learn this on their own. I mean, yes and no. I'll give the old guard a little bit of credit in that the costs of learning a new system can be quite high. I do think these need to be kind of step functions. I think it's fine to invest in technology and architecture model and a set of technologies for a while until they start to like, you know, you kind of want to standardize on a golden path,

right? Until they start to show their age, which could be three years, right? But every three to five years, you're going to need to like renew and refresh it radically from scratch. And that's an awful lot of... It's kind of the optimized locally versus optimized globally tension. Yeah. And the problem of course is I look at this, how in the world am I going to be able to intelligently advise companies to do this on a one-off basis when I see this pattern happening again and again and again everywhere? It feels in some cases like I'm shouting into the wind. Yeah, well, I think you have been because they've had more money than sense.

And now for a time they won't. And we've always been waiting for the next correction. I don't think we expect this to look quite like this one. But now it's a question of, okay, if we have to reduce headcount or stall on hiring, which things can we transition to a managed provider that, yes, will cost us more in raw infrastructure terms, but reduce the expense of engineering time by 10x? Yes. Some reporter was asking me, like, are people going to be backing away from the cloud? And I'm like, no, no. The sticker cost, it is like a tenth of the actual cost. Most of the cost is in engineering time and in opportunity time.

You're not doing the things that could have made your business succeed because you were doing the infrastructure. Exactly. Will people be backing away from the cloud? Well, not smart people. Not smart people. Exactly. And like serverless is like an order of magnitude or more cheaper than, you know, even containers. Now you can't use that for everything. And maybe it didn't make sense for a long time to rewrite parts of your infrared and

that, but we rewrote our database in serverless, basically. It's S3 files and Lambda functions, and it fell in cost by an order of magnitude. And that is something I'm finding. It's just people look at cost through the wrong lens. They're worried about lock-in. Well, great. So you're going to build this thing. It's easy to get higher. Oh, yeah. It's easy to get headcount. And finally, we're starting to see that change a bit. I wish it were happier circumstances.

But we see Kubernetes, too, where it feels like all these people, like that's been sort of the refuge for engineers who want to build their own cloud provider, but don't actually work at a cloud provider. So here we go. I have been calling like Kubernetes the resume driven development because nobody needs it, but all the engineers think they need it on their resume in order to get jobs. And if you're working on Kubernetes all the time, like who's minding the store? I know. What the are you even doing? You know, funny story. I actually got leaked a couple of slide decks from startups. Founders who are out there pitching basically Honeycomb for Kubernetes or Observability for Kubernetes, which is missing the point in such an enormous way because

the point is supposed to be that it's boring. The thing that serves the code is supposed to be as boring as possible. It's inside your code and inside the systems that it's supposed to be interesting. Exactly. If you get excited about the things that Honeycomb does, for example, it passed a certain point, you probably should apply to work there. Yeah, well, lots of people do, but anyway. You should not try and roll it yourself from first principles inside of your own company.

There's like five companies in the world that might want to do something like that intelligently and the rest are, again, trying to reinvent things from first principles that they best should not. Sadly, all the people who are out there hiring quote-unquote observability teams are hiring to staff them with the last generation of time series database folks who are just like, oh, this is a hammer.

I have a hammer. This is a nail. I have a hammer. And I think they're going to be not too happy with the results. Yeah, I don't know. I think that some downward pressure is, you know, creativity loves constraints. And I feel like for too long in engineering, we haven't had constraints in terms of like, you know, the resources.

And I don't want to minimize the amount of pain that's out there right now, or, you know, people losing their jobs. That sucks a lot. And I feel for you. Engineers, I think you're going to be okay. I mean, I think part of the other, like part of this downturn is folks are going to distributed teams in a real way, which is exciting.

Engineers are going to be able to get jobs for the most part. And I think it's good to have some downward pressure on costs. But every crisis is an opportunity, right? And I think that the companies that have learned these lessons well and have not forgotten them between downturns have been the companies that you really look to who are doing things well, who are doing things in ways that are sustainable, who don't have to make many corrections when these things hit. And I hope that that winds up ideally breeding a culture that's a bit more aligned around this going forward. I would hope so. I don't want to go back to the way things were and seeing these ridiculous things that are pitched with absolutely no business model. I know. I know. At some point, engineers, we have to come into the fold as a part of the business.

Engineering exists to serve the business. We exist to build things for users. We don't exist for our own entertainment. You know, like there are plenty of ways you can write code for fun if that's what you're trying to do. But at work, we have a responsibility to not just take these things into account,

but to do better at learning how to verbalize them, how to spread them as our values, how to make them something that is not just passed along as an oral tradition, the way I feel like it is right now, but like actually professional standard. Wouldn't that be the dream for all of us? We're moving in that direction, you know? It's just a question of how quickly. We certainly seem to be. So thank you so much for taking the time to chat with me about this. If people care more about what you have to say, where can they find you? Oh gosh, I have no idea. Right? I'm all over the internets, but you can find me. My personal blog is at charity.wtf. Honeycomb is at honeycomb.io. Our blog on the Honeycomb site has a lot of observability stuff.

And basically, I'm talking all over. Which is delightful and entertaining. If you enjoy the theme of my nonsense, I definitely recommend you check out Charity, should you not be fortunate enough to be already aware of it. We have similar nonsense themes. It is true. Well, thank you once again. I really do appreciate it.

Thanks, Corey. It's been fun. Of course. Charity Majors, CTO and co-founder of Honeycomb. I am cloud economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on Apple Podcasts, whereas if you've hated it, please leave a five-star review on Apple Podcasts, along with a comment telling me which system you're building yourself from first principles. This has been this week's episode of Screaming in the Cloud.

You can also find more Corey at screaminginthecloud.com or wherever fine snark is sold. This has been a humble pod production