The Changelog: Software Development, Open Source - Amazon's silent sacking (Interview)

Episode Date: January 11, 2024

Justin Garrison joins us to talk about Amazon's silent sacking, from his perspective. He should know. He works there. Well, as of yesterday he quit. We discuss how the cloud and Kubernetes have transformed the way software is developed and deployed, the impact silent layoffs have on employees and their careers, speaking out about workplace issues (the right way), how changes in organizational structure can lead to gaps in expertise and responsibility which can lead to potential outages and slower response times. By the way, we officially let the cat off out of the bag in this episode. Justin has joined the ranks here at Changelog and is taking over as the host of Ship It! Expect new episodes soon.

This week on The Change Law, we're joined by Justin Garrison talking about Amazon's silent sacking. And this is from his perspective. He should know he works there. Well, not actually anymore as of today. And this is not part of this episode, unfortunately, but I learned on LinkedIn that he quit. There you go. On this episode, we discuss how the cloud and Kubernetes have transformed the way software is developed and deployed, the impact silent layoffs have on employees and their careers, speaking out about workplace issues the right way, how changes in organizational structure can lead to gaps in the expertise and responsibility, which can lead to potential outages and slow response times. And by the way, we officially let the cat out of the bag in this episode. Justin has joined the ranks here at ChangeLog and is taking over as the host of Ship It.

Expect new episodes real soon. A big thank you to our friends over at Fly, the home of changelog.com. Launch your apps near your users. Fly transforms containers into micro VMs that run on their hardware in 30 plus regions on six continents. Check them out at fly.io. What's up, friends? This episode is brought to you by our friends at Neon. Serverless Postgres is exciting, and we're excited. And I'm here with Nikita Shamganov, co-founder and CEO of Neon. So, Nikita, one thing I'm a firm believer in is when you make a product, give them what they want.

And one thing I know is developers want Postgres, they want it managed, and they want it serverless. So you're on the front lines. Tell me what you're hearing from developers. What are you hearing from developers about Postgres managed and being serverless? So what we hear from developers is the first part resonates. Absolutely. They want Postgres, they want it managed. The serverless bit is 100% resonating with what people want. They sometimes are skeptical. Like, is my workload going to run well on your serverless offering? Are you going to charge me 10 times as much for serverless that I'm getting for provision? Those are like the skepticism that we're seeing and that people are

trying and that they're seeing that the bill arriving at the end of the month and like, well, this is strictly better. The other thing that is resonating incredibly well is participating in the software development lifecycle. What that means is you use databases in two modes. One mode is you're running your app and the other mode is you're building your app. And then you go and switch between the two all the time, because you're deploying all the time. And there is a specific part when you're just building out your application from zero to one, and then you push the application into production, and then you keep iterating on the application.

What databases on Amazon, such as RDS and Aurora and other hyperscalers are pretty good at is running the app. They've been at it for a while. They learned how to be reliable over time. And they run massive fleets right now, like Aurora and RDS run massive fleets of databases. So they're pretty good at it. Now, they're not serverless.

At least they're not serverless by default. Aurora has a serverless offering. It doesn't scale to zero. Neon does. But that's really the difference. But they have no say in the software development lifecycle. So when you think about what a modern deploy to production looks like, it's typically some sort of tie-in into GitHub,

right? You're creating a branch, and then you're developing your feature, and then you're sending a PR. And then that goes through a pipeline, and then you run GitHub actions, or you're running GitLab for CICD. And eventually, this whole thing drops into a deploy into production. So databases are terrible at this today. And Nian is charging full speed into participating in the software development lifecycle world. What that looks like is Nian supports branches. So that's the enabling feature. Git supports branches. Nian supports branches.

Internally, because we built Non, we built our own proprietary. And what I mean by proprietary is built in-house. You know, the technology is actually open source, but it's built in-house to support copy and write branching for the Postgres database. And we run and manage that storage subsystem ourselves in the cloud. Anybody can read it.

You know, it's all in GitHub under Neon Database repo, and it's quite popular. There are over 10,000 stars on it and stuff like that. This is the enabling technology. It supports branches. The moment it supports branches, it's trivial to take your production environment and clone it, and now you have a developer environment. And because it's serverless, you're not cloning something that costs you a lot of money. And imagining for a second that every developer cloned something that costs you a lot of money in a large team, that is unthinkable, right? Because you will have 100 copies of a very expensive production database. But because it is copy and write and compute is scalable. So now 100 copies that you're not using,

you're only using them for development, they actually don't cost you that much. And so now you can arrive into the world where your database participates in the software development lifecycle. And every developer can have a copy of your production environments for their testing for their feature development. We're getting a lot of feature requests, by the way, there, people want to merge this data, or at least schema back in into production, people want to merge this data or at least schema back into production. People want to mask PII data. People want to reset branches to a particular point in time of the parent branch or the production branch or the current point in time, like against the head of that branch. And we're super excited about this. We're super excited.

We're super optimistic. All our top customers use branches every day. I think it's what makes Neon modern. It turns a database into a URL and it turns that URL to a similar URL to that of GitHub. You can send this URL to a friend, you can branch it, you can create a preview environment, you can have dev test staging, and you're living this iterative mode of building applications. Okay, go to neon.tech to learn more and get started. Get on-demand scalability, bottomless storage, and data branching. One more time, that's neon.tech. Well, we are here with Justin Garrison, the new host of the Ship It podcast. Maybe you've heard of it.

Justin, welcome to the show. I guess welcome to the ChangeLog Network. Yeah, thanks so much. Fun times ahead. It's going to be a good adventure. The shipping game is fun, and the Ship It show has been a lot of fun for us to produce and a lot of rave around it. Certainly have Gerhard behind the scenes.

Still doing infra here, but not involved in the show directly for the moment. But Justin, how do you feel about this? How do you feel about this podcast and some of the plans that you've started to put in place? Yeah, I mean, I feel great like as a fan of the show and what Gearheart was doing with the show, like that's the area of technology and of software that I have traditionally worked in, in the areas that excite me or is that that moment when code becomes software. Yeah. As soon as, as soon as we take the bits we wrote on some storage medium and add electricity to them and they like run through a CPU, like that's the software side of it. And that's the, like, that's the part that excites

me. It's like the, the code is, is fun. It's challenging. There's a lot of things there. But I really enjoy the, like, what happens when we actually like make this thing run. And I'm really excited about those things. And so it's like everything after that moment of like, you commit the code. Now what happens? That's where I love talking about those stuff. So this will be your first time with us, but not your first rodeo. You've been making Opsy, Cloudy.

I'm not sure how you, maybe you talk about the cloud the way that you frame it and talk about it. Is it DevOps? Is it Infra? Is it SRE? Whatever it is. You've been making content in this world for a while and working in this world for a long while.

You are currently working at AWS on the Kubernetes team, but we'll talk about that here in a few minutes. That whole scenario is, it's a debacle, I think, but it's an interesting one. But talk a little bit about your history of content creation, the stuff that you've been doing, and it'll help people see why we think this is a great opportunity for Shibit to come back working with you. Yeah. I mean, my first, the first podcast I ever did was in 2007, which is, it was just now ancient history for the podcast ecosystem. I ran a podcast called Linux Mint or Mintcast.

It was for the Linux Mint community. I was a fan of Linux Mint. I was using it. I had discovered it while I was in college. And I was like, this is amazing. It's free software. As a broke college student, I wanted to give back. And the only way I knew how to give back

as a non-developer, as someone that at the time, I was just too embarrassed to write any code whatsoever. And I was just like, I can tell people about it. I can just tell them the news and I can tell them where to find things. And I love that community building aspect of it. The Mintcast podcast is still going on, which is amazing to me. Like we, we handed it off a long time ago, but they've kept doing it. And it's awesome to see that sort of flourish for new people, every generation. But ever since then, like I, I first had a conference talk based on the Mintcast podcast that I did. I showed up at a local Los Angeles Linux community group and they're like, Hey, we want someone to talk, you know, lightning talks. I was like, I don't know what that is. Like, well, just go and talk

for five minutes. I was like, Oh, cool. Well, I run a podcast using all open source software and we publish it and it's about open source. And this was an open source conference. And they're like, Oh, great. Someone in the audience, I'm in the room of like 15 people. Someone in the audience came up to me afterwards. Like I listened to your show. Like, are you kidding? Like this is that's, that's when like the, it became real to me. I'm like, wait, wait, the thing I did in my living room on the weekends and editing this thing like that actually became someone in real life. And, and then having that full cycle kind of happened, which is amazing to me. And then ever since then I've been blogging, I've been doing conference talks. I was doing all that stuff as an engineer. I worked at Disney for six

years on feature animated films on Disney plus I've been at Amazon now for three and a half years working on the EKS ecosystem and products. And so I've been around different environments, mostly on-prem in my past and then in the cloud beyond that. But I've been just involved in open source communities, trying to give back in those ways. And sometimes making content and training people, whatever we want content to be, right?

It could be a blog post, a video, a podcast. I wrote a book a few years ago with a great friend. And, uh, and those all, those are all content pieces that people can use and they can make themselves better asynchronously. And I love that aspect of like, I can spend some time now that will have echoes for a long time. Yeah. That's still cool for us as well. When people say, Hey, I was just listening to that show you did about X. And I'm like, that was four years ago. And they're like, yeah, but it was listening to that show you did about X. And I'm like, that was four years ago. And they're like, yeah, but it's very interesting for this reason. And you're like, oh, wow, that was something that happened.

And it's so asynchronous. We're talking years of time, and it's still out there for people to consume. Of course, books, blog posts, they all kind of can get stale in the same way, but they all have that exact same long tail of value. That's just so cool. And I love having those conversations with people that, you know, I made something, you know, four years ago, like the book I wrote was 2017, six years or seven years ago now. And people come up to me, ask me about like, Hey, cloud native infrastructure. I read the book. What do you think about this? I was like, Hey, let me tell you about everything I've learned

in the last seven years. And that's the great place because we can start at the same moment of, Hey, that thing I did, here's everything I believed at the time. Here's why I believed it. You know, the context, but now what is it like today?

And we can start from the same point. And that's a great way just to talk to anyone, right? Like we small chat at in line and like, Hey, how's the weather? Like we all are experiencing that at the same time. And content in a lot of ways is similar where you get the starting point to have a better, deeper conversation.

Yeah, podcasting has some legs. It's interesting, even 2007, I began in 2005. It's just crazy to think that we've been podcasting for so long, basically. That's insane. And everything old is new again. I see a lot of those patterns from what I was talking about way back then is like coming back again. And it's just like, hey, we have new branding around it or there's some new tools and maybe

it's better than it was 15 years ago. But the ideas, a lot of the ideas are still the same. And a lot of the people that are trying to get into it and learn it are coming for the same motivations. And so a lot of the people haven't really changed as much as it's, it's the same sort of desires and they want to learn and they want to have better lives for themselves and they want to be better people in a lot of ways. Like, Hey, if I learned this thing, I'm reaching this goal. And they push themselves and they subscribe to podcasts and read books so

that they can push themselves further. So you've learned a lot in the last three and a half years working at Amazon on the Kubernetes team. I want to talk about, I mean, we all want to talk about the silence hacking team that's going on because, wow. But before we do that, let's just talk Kubernetes for a few minutes. You've been deep into it. Here you are. It's 2024. What's Kubernetes looking like?

Is it still sucking all the air out of the room of the cloud and DevOps? Is it kind of where people are over it but building on top of it? Are they going around it? Like what's your perspective on the cloud in light of Kubernetes right now? I think cloud changed a lot of things for a lot of people. Obviously, like I was working on-prem in a data center and we had to, you know, we need a new server and I have to go send an email. And that was like the first way I was like, Hey, I need to, I basically need a prompt. I have to wait three months and go through procurement to like get that prompt so I can do some work. And that distance between I need to do something and actually starting to do something was

so vast that it was just, we couldn't keep doing that because I had to like keep projects going every three months. And cloud obviously like shrank that time dramatically and just said like, Hey, it's just on demand and you only pay for it on demand. And when I was first learning Kubernetes, I would go at my lunch break and I would spin up a GKE cluster because I could get it in five minutes. I would poke at the API. I would deploy some services. I would understand how some of it worked and I would turn it all down. And over a month of doing that, my cloud bill for poking at it at lunchtime was under a dollar and i

was just like wait a minute like i i easily can just go learn this stuff and it makes it so accessible to be able to figure out how this stuff works and not spend basically any money it's not even a starbucks drink right it's like the threshold for like is this too much money it's like no like this is just i lose this money by dropping you know coins or something like this was a great investment in my time. And the cloud still does that. The cloud still pushes people forward to give them access to those technologies, especially new technologies.

It may not be just the VMs and networking anymore, but someone says, hey, I want to go run a serverless website. Let me go figure out how Lambda works. The free tier for that is amazing. And you can just run that for a while and you can learn a lot of things through doing that that's still just the cloud is expanding more and more and making a lot more new technology accessible to more people around the world and that's great kubernetes on top of that is one of those features it's just a hey there's a standard api that you can kind of interface with my first involvement with kubernetes was basically

deploying it on-prem. And it was, hey, we're doing this from bare metal. We're spinning it up. I was one of the SIG co-chairs for SIG on-prem, which doesn't exist anymore inside of the Kubernetes community. But it was just, we were trying to do it on-prem and there was a handful of us that were like, this is really hard when I don't have an API and when not everything looks the same. So what does that look like? In my time at Amazon, I actually helped with a product for EKS Anywhere, which was helping other people do Kubernetes on-prem. It was just like this recurring thing for me where it's like, hey, this was hard when I was at Disney. It's hard now when I'm at Amazon. Let's go ahead and see if we can help people do this a little easier.

Well, you're talking about your time at Amazon in the present tense. You're talking about it in the past tense. We know why. Let's get everybody in on the story. So we've been talking with you over the last three or four months. So we've known about this situation with your work at Amazon. I think as it was happening and you're kind of waiting, you've been waiting, you've been waiting, waiting, waiting. Tell everybody the story with Amazon and these silent layoffs and like what's been going on. You recently went public with a blog post about it. You also recently went public with your involvement in ShipIt. So good timing all around, you know, January 24. It's time to start making some announcements. Tell that story in brief and

we'll dive into the details after you laid out. Yeah, this is my first time in a professional role doing DevRel, doing content creation and doing this style of work. I've always just been an engineer and I love this job. I've always just been an engineer and I love this job. I actually, I really found that I was, I felt like I was good at it. I enjoyed engaging with people and, and learning something deep technically, and then telling people how it works and that like cycle of back and forth. And then being part of a product that I know a lot of people used was also just wonderful. Like that was something that I joined Amazon for. And I really like a lot of things at Amazon with how they were working on things and that

involvement to get some outside voices or non-engineer voices necessarily. I'm not writing the production code. I'm helping guide the product and then testing it and saying, hey, this is how customers should or shouldn't use this thing. And I've had experience across the board with, I helped launch AppRunner. I did a lot of work with ECS. I had some talks I've done about Lambda. Like this is really broad because no one uses any of these services in a vacuum. And I like that breadth that I was able to just, Hey, I can pick out any of these things. This year in, well, I get last year now it's 2024,

2023, Amazon really pushed for return to office. And I was hired as a remote employee before the pandemic. I started during the pandemic, but I had my contracts and negotiations and everything before as a fully remote employee. I was remote at Disney before. So I was used to that. I wanted, I needed a remote job. And in 2023, they were really saying, Hey, we're going to start returning to office. I was like, I don't, I'm a virtual employee. I don't have an office. Like the closest office to me is, you know, maybe an hour away, maybe two hours away with LA traffic. And, uh, and so I was like, I, you know, I've gone there a few times to meet up with people and to have some meetings, but it's not like a regular occurrence. And I was always told that my

role would, would stay remote and I didn't have to worry about it. And then things started changing more and more noise was happening during the summer. It was like, actually more people are going to to start coming back to an office part time. And I kept being told over and over again by managements and my leaders like, no, no, no, you're fine. You're a remote employee. My entire team is not in a location. The DevRel team is all across the United States. And when I joined, we were all over the world. I was like, we don't have a time zone, let alone location. And so as things started progressing, it was becoming more and more clear that this was going to affect me at some point. And we filled out forms to get a remote exception, which was for like a one year thing. I had to renew that every year. And I got my approval for remote exception just days before I was told that our team was actually going to be disbanded and our team would not exist anymore under the kubernetes org as part of the kubernetes product team and and they were they wanted to get rid of the sort of dev rel product space under that product under the service team and so that was okay well like what happens there was you know

a handful of us on this team what do we do and they said we want to give you time to find another job internally and if you find a job great go, go ahead and take it. You can shift internally. Amazon's also been mostly on a hiring freeze for over a year. And so that was like, well, there's a lot that's not going to be available. And most of those other teams are also requiring me to go into an office. And so this wasn't a matter of like, oh, let me just shift and start doing new work. I had to find a team that was local in LA because I couldn't just go to any office. I had to go to my team office. I had to go to return to team is what they actually deemed it. And so if I wanted to go to the local office, I had to find out what teams worked from that office and then see if

they had openings and then work with them. And that was just, there was all these, a lot of barriers. And I just kept getting more and more frustrated with some of that, like, Hey, this isn't actually just an easy switch. There's not a lot of teams hiring. If they are, I'm probably going to be in a space that I have to leave anyway. And the more people I talked to, I found the more and more people across different areas, different services and different divisions were hitting some of these same limitations and same frustrations where they were saying like, actually I have to find something else or, or move. And in many cases it was just like, Hey, well, how do you, how do you get out of this situation? And every, they're actually like official email from, you know, Amazon was like, if you don't return to

an office, you will like voluntarily resign. Like that's not, that's not like a thing. Yeah. That's not how that usually works with with this sort of like employment contract because this is a contract of just like i give you some time and some value and and you give me money and that's how any job works and and when the rules change when the contract change it became more of a frustration i found that a lot of times it was just people were just silent and they just said like i don't know what to do because i i couldn't do anything so they would just sit around and that was just people were just silent and they just said, like, I don't know what to do because I can't do anything. So they would just sit around.

And that was just like, I don't I'm going to answer some questions as they come. And I finished out my work for what I was already scheduled to do. And I asked for severance and I said, hey, like, this isn't working. I've talked to a bunch of teams. I've looked around, but this situation is just not working. And that was where that frustration was really coming from, I was like, I, I can't do anything and I'm not going to find another job and I'm not going to voluntarily resign because you got rid of the job I loved. It was like in any other case, when, when someone gets rid of the position that you really enjoy

doing, like there's some monetary or some situation where it's like, Hey, we're going to close out this contract. I know it's, it's an at will sort of thing. Like you can be let go at any time, but also like, I know there are some labor laws that protect some people in situations. And so, um, but I, I wanted to write that blog post mainly to give voice to all the people I talked to that were like, I can't say anything and I can't do anything. And I have no network of external people.

And a lot of them are fresh into the technology from the past, you know, two, three years, like they're junior engineers that are just being forced out without any sort of compensation or connections or anything else. And I was like, I really wanted to give them a voice in a lot of ways, just by sharing my own experience and saying like, this is what I've gone through. is what i've seen and and i don't think that's right and in this post you lay out amazon's obviously it's hard to give personification to a blob but like kind of the amazon incentive structure on why this might be the case like this not doing layoffs but just i, removing positions and not people. I'm not sure exactly how you say it, but just taking your role away.

And I guess they would rather pay you to do nothing than lay you off. Can you explain like your thoughts on why that is? Yeah. I mean, looking back at the last year of, you know, earnings calls and stock prices and all that stuff, I was like, actually the times that there's a problem is, is when you have to say like, we have to lay off some percentage of our company and most companies that lots of companies did that. This wasn't an Amazon specific thing, like lots and lots of technology.

People were laid off, unfortunately. And many of those public companies, they, they take a dip for a little while and then things start to settle out again because they were the overhead of creating the products got less because just people were gone whether they were doing more work or not they were able to recoup some of that money that they were spending on on the people that were hired and at some point people forget about it they just say like oh i forgot you laid off 10% of your staff. And it doesn't matter because I'm as a customer, I'm still fine. I'm still getting value from the thing.

And my value hasn't changed whether you had those people there or not. And I look at this a lot of ways, like when Netflix raises their prices, everyone feels it. Everyone's like, oh, not another, you know, another $2 a month for that streaming service. Am I getting $2 more value out of this thing? And the other, like they needed to make more money to, you know, projection wise, be able to sustain. Right. The other way you can do that is to cut things out, which I feel like is a lot of ways that people, like if I'm, if I look at my budget and I'm putting it out six months, I say, you know what, I'm going to run out of money in a year.

What am I going to do? Am I going to go get a second job or am I going to go stop having Starbucks every day? But like, those are the choices that people make a lot of times. And companies can make very similar decisions. A lot of times they'll make both decisions. They'll look for new revenue areas, but also they're going to save money in various ways. And the people at a company are almost always the most expensive. But when people leaving affects the stock price, that's way more expensive. And the stock price value is way more valuable than hundreds or thousands of people. And so if people are quietly leaving and no one really connects all of the dots and say, hey, actually, I saw 100 people on Twitter leave, that doesn't affect the stock price.

But if they say we laid off these divisions, that does. And so there's different monetary. So I can let a hundred people leave on their own over three or six months, and I will still lose less money than announcing they're all gone, giving them three months severance, and then having the stock price kind of do a rollercoaster. And so I don't know for sure. I'm not running Amazon, but these are trends that I started seeing, not just Amazon, but other companies and people I've talked to in the community. And that sort of like silently just like coasting people out and saying, Hey, you're not going to have any career progression.

Hey, I'm not going to give you any new interesting work. Hey, you can't switch teams. I, what are people supposed to do? Like these are their lives. And there's like, well, now my projections for six months are like, I don't know when my job's going to end at that point. Right. You're just like, I know that in some point I will not have a job. What do I do? And my contract says I can't go get another job. So it's like, that's the limits. What I'm going to do is, and I can't just spend no money. So it's just, I need to start saving now and then figure out what's going to come next and start making those

connections. And, and so we're doing the same projections at an individual level to say, how do I make sure I can provide for my family and live in six months? Especially if you're in the United States and you need healthcare and you need these other things that are just, you have to pay for them. And the cost of living, uh, not only like generally goes up, but like, as I get older, I'm like, wow, my, my subscription to life costs more money because I have more medication. I have more, more aches and pains. I have all these things that have to happen that it's just like my baseline when I was 20 is not near what it is now that I'm 40. And in those sorts of things, you know, pay out and I have to like, I have to figure out like,

okay, what does it cost to raise my family? And where is the end of this job or this paycheck or these things that I've just come accustomed to? what's up friends i'm here with one of our good friends for ross of luca dj for ross is the founder and ceo of socket you can find them at socket.dev. Secure your supply chain, ship with confidence. But Farras, I have a question for you. What's the problem? What security concerns do developers face when consuming open source dependencies?

What does Socket do to solve these problems? So the problem that Socket solves is when a developer is choosing a package, there's so much potential information they could look at, right? I mean, at the end of the day, they're trying to get a job done, right? There's a feature they want to implement, they want to solve a problem. So they go and find a package that looks like it might be a promising solution. Maybe they check to see that it has an open source license that has good docs, maybe they check the number of downloads or GitHub stars. But most developers don't really go beyond that. And if you think about what it means to

use a good package, to find it, to use a good open source dependency, we care about a lot of other things too, right? We care about who is the maintainer? Is this thing well maintained? From a security perspective, we care about does this thing have known vulnerabilities? Does it do weird things? Maybe it takes your environment variables and it sends them off to the network, you know, meaning it's going to take your API keys, your tokens, like that would be bad. The unfortunate thing is that today, most developers who are choosing packages and going about their day, they're not looking for that type of stuff. It's not really reasonable to expect a developer to go and open up every single one of their dependencies and read every line of code, not to mention that the average

NPM package has 79 additional dependencies that it brings in. So you're talking about just, you know, thousands and thousands of lines of code. And so we do that work for the developer. So we go out and we fully analyze every piece of their dependencies, you know, every one of those lines of code. And we look for strange things, we look for those risks that they're not going to have time to look for. So we'll find, you know, we detect all kinds of attacks and kinds of malware and vulnerabilities in those dependencies. And we bring them to the developer and help them when they're at that moment of choosing a package. Okay, that's good. So what's the install process? What's the getting started? Socket's super easy to get started with.

So we're, you know, our whole team is made up of developers. And so it's super developer friendly. We got tired of using security tools that send a ton of alerts and were hard to configure and just kind of noisy. And so we built Socket to fix all those problems. So we have all the typical integrations you'd expect, a CLI, a GitHub app, an API, all that good stuff. But most of our users use Socket through the GitHub app. And it's a really fast install. a couple clicks, you get it going, and it monitors all your pull requests. And you can get an accurate and kind of in depth analysis of all your dependencies, really high signal to noise, you know, it doesn't just cover vulnerabilities, it's actually about the full picture of dependency risk and quality, right? So we help you make

better decisions about dependencies that you're using directly in the pull request workflow directly, directly where you're spending your time as a developer. Whether you're managing a small project or a large application with thousands of dependencies, Socket has you covered and it's pretty simple to use. It's really not a complicated tool. Very cool. The next step is to go to socket.dev, install the GitHub app or book a demo. Either works for us. Again, socket.dev. That's s-o-c-k-e-t.dev do you feel like a whistleblower in a way so i feel like there's like a whistleblower moment in a way because it's you're revealing what no one else sees because you have a certain purview of the scenario and you're still employed there right and so you wrote

not a scathing thing but a very factual and i suppose lots of opinion in there of what you assume because like you said you don't run amazon you don't run Amazon and you're not a financial advisor so this is not financial advice either at the very top of it, but it's kind of whistleblower-y, if that's a word, because it feels like not everybody really realizes this silent sacking as you've brought out, but you're still there. Right, and I do feel like I collected a lot of that. I took what I was seeing everywhere because when you're on the outside, you don't pay attention as much to a company or you don't know, have all the connections with one company. Yeah. The inside,

when you at least work there, I have all of the connections of people that like, I know that left and all these people, I'm like, I loved working with these people and they're all gone. And now what do I do? And so I'm, I'm basically just collecting all of those LinkedIn posts and all those Twitter posts and all those things. And just saying like, Hey, why'd you leave? What was the reason that you're not here anymore? And everyone has their own reasons. Everyone had their own situation, but a common thread was no career progression or, or my, my boss said I had to return to office or move or resign. And those were my options. And so there is a little bit of a whistleblower only from the, like, it's going to hit customers at some point, you can't keep doing everything you were doing before

with that few of people. And at some point, the trust, the hard earned trust that Amazon has built up over the decade of running a great cloud, having awesome operational excellence, and then getting rid of a lot of the people, like you can't do the same things. And as a customer, I was a customer at Disney and as a personal customer, I very much am like, I don't know that this is going to be the same thing in 2024 or 2025. Um, there's going to be some consequences to losing people that run these services because any, any software that we're running, like the ship at podcast is all about like, how do you run it? And how does it, how does it maintain itself? As much as we want to say things are self-healing, they only are to a degree.

And at some point you need someone to be able to troubleshoot and fix those things. But those people also need to take vacations and go to sleep and have rotations for on-call, that sort of stuff too. Right. So you published that post five days ago as we record this, what has been the response? Have you had any response from anybody at Amazon or has it resonated with other folks? Like it seems like usually when you blow a whistle, people, you know, perk up and listen. Has it, has it made a splash at all? Or is it just kind of like, I will say more people saw it than I expected.

I mean, I wrote it, it was like December 30th. It was just like, this wasn't like a great news cycle time. It was just like, hey, I was at the point where I was like, you know what? Like, I don't know when this is ending and I'm okay with that, but I need to tell people about what I think is going on and why I think it's not a good situation for a lot of people. I've had so many DMs from people currently at Amazon or previously at Amazon that I have yet to have anyone disagree with me. No one has come up and said,

you're wrong for this reason. I would love to learn more. If someone has a story or situation that I pointed out something wrong, please let me know. But just from my own experience, I tried to just share my experience because that's what's protected under labor law in California is sharing my experience about working conditions.

Right. And, and so that was where I was, I was trying to stay very much in the lines of, I'm not sharing confidential internal information. I'm sharing my experience and what I've seen as a pattern and the amount of people that have reached out and made new connections to agree with me has been very surprising because I only knew of a small sphere of people that I directly worked with, or I knew their names or followed them online. And those, that was the sphere of pattern I was seeing, but seeing this play out in other countries in completely different divisions, uh, that has been really fascinating. It's just like, Oh, this, this is actually a much bigger deal, um, than even that I knew from my, from what I could see. Hmm. What about internally at Amazon? Like your higher ups, has anybody said,

all right, Justin, you're fired or like, we'll hook you up with some severance. Sorry about that. Please take the post down. I don't know. Like, has anybody said anything? No. So I know that the post was escalated and reviewed by HR and legal teams at Amazon. And in both situations, they said, I didn't break any policies. There was nothing that I went outside of. I didn't, you know, if it was a whistleblower, like I'm not breaking any rules for having a personal opinion on the internet. And that was, that was the only communication I was given back was there would be no discipline and there would, there wouldn't, I wasn't breaking a policy. And you're still employed there. You just don't have a role. Yeah.

How long can this go on? That's, I don't know. And I asked for, I asked my leadership team, my VP and my skip level for Severance in October. And I said, hey, you, when, when this all happened, I said, hey, what if we don't find a job? What if we don't find something else internally? And they said, well, you should leave. I'm like, well, no, no, no. Like, well, what are you going to do? Like, this is, you just got rid of that job that

I really, really liked. I really enjoyed this job and I enjoyed working here. Is there any severance? And they told me yes. And I said, yeah, that's a, it's a possibility. Once we, we go through all the other options, I'm like, cool. So I'm going to try the other options first. And so in October I asked for it. And every week since then, I would send a message to my leadership. I'd say, hey, by the way, I'm still here. I emailed you for severance to be able to let me go. I know I could leave at any moment. I know I could just say, I'm done. I'm out. I feel like that's not the right thing to do from a company perspective when you're forcing people out this way. And so I'm like, I can stay here. I'm not in a rush. Obviously I'm getting paid. I'm learning a lot of great things. I started my own podcast last

year that I'm not going to continue with ship it. But like my job was to learn things, create content and help guide customers. I didn't stop doing that. I frequently was always able to like, I went to KubeCon. I had great conversations with people. I'm around like learning things. I'm doing YouTube streams. I'm doing podcasts. I'm doing blog posts. I'm around like learning things. I'm doing YouTube streams. I'm doing podcasts. I'm doing blog posts. I'm doing all the things that I was doing before. I'm just not getting official work from the product team, which I wasn't always.

This wasn't like, oh, I had to wait for them to tell me something. I'm like, no. One of the great things about a DevRel position is like I can take the initiative and say, hey, you know what? I really want to learn about this new thing. Let me spend a week on it. Let me figure out how that works and then go tell people about it. And I did a lot of, I've done a lot of videos over the last couple of years on, on Tik TOK and YouTube shorts and running my own YouTube channel. And those sorts of things are just, I'm still doing them. I didn't stop. Um,

so it was like my, what I'm doing is still the same stuff. Um, maybe not as like to the, you know, every hour I'm like, I have to do this thing right now. It's because I don't have assigned tasks ever since, you know, mid-October when I finished out what was assigned to me. So I'm here, I'm still doing stuff. I'm still learning stuff.

I still love the Kubernetes community and open source and running infrastructure and cloud. I'm like, I'm still doing that stuff. I'm just not getting new work. You mentioned the RTO thing, having to go back to your team's office. Do you have to do that then? So is that a requirement for you to go

into the office? And has that been a burden if you have? I've still had a year remote exception. So I'm like under my current role, like I'm fine until August. I don't have to go into an office. Well, you know what Milton did on Office Space, right? You know what Milton did? Yeah. Mr. Lumberg told me to talk to payroll, and then payroll told me to talk to Mr. Lumberg. And I still haven't received my paycheck, and he took my stapler, and he never brought it back. And then they moved my desk to storage room B, and there was garbage on it, and I don't appreciate it.

Why don't you go back down and sit at your desk? Mr. Lumberg should be here any minute. Mr. Lumberg. Just go sit at your desk, okay? Okay. He just kept showing up. You know, he just grabbed his red stapler and he just went to work every day. And he had his cake, you know, at the office parties.

But suddenly they fixed the bug and they quit paying it. Well, and that's like, I feel writing that blog post, I felt very much like I'm in such a privileged position that who wouldn't want this couple months that I've had of I get paid very well. I get to learn whatever I want. I have resources that I can run things in cloud environment. I can have all the access to learn and do the things that I know some people would just absolutely love to have. They would, they would love to have a couple of months to go learn new technologies. And at some point I'm just like, I shouldn't say anything.

I should just stay quiet and just keep doing this. But I knew everyone else that I talked to that had no voice whatsoever and they were being forced out and they didn't get this reprieve of a month or two to, to learn some things. Like they were the ones that I wrote the blog, but like that wasn't necessarily for me. It was to give them a voice and to say, Hey, this is something that I hope Amazon and other big companies that are doing this stuff hesitate to do, whether they stop or not, at least they know, Hey, someone could talk about this. And this is not a good decision for our employees. This is not something that benefits our employees any well. This is just benefits us.

And so how do we make sure that we take care of our employees? And I'm okay because, again, I don't have a role already. The moment I'm let go is I was already planning that. This isn't something that is new to me. Right. Let's talk about the ramifications of this you talk about the no more pizza teams and how teams were lean before and now they're being emancipated or emaciated sorry not my bad different word same e-letter start

and then you predict outages ahead help us understand where since this is not you breaking the rules and this is flown by Amazon's legal and HR, and they're like, Hey, you haven't broken any rules. This is not inside information. This is opinion. What makes you think that the outages ahead for this next year? If we go back and look at DevOps, when DevOps started, like DevOps was this thing that like, originally it was your team that ships a product is full stack, right? You don't have any external dependencies. I never worked, I worked at Disney for six years and that was very much not that.

There's, once you have a DevOps team, you've lost a DevOps in many ways, right? Like that's like the centralization of, of what should have been, you know, split out around the company. And, and Disney, at least for my experience, was very centralized. And we had a database team

and we had a network admin team and we had a, you know, compute team. And you, you central for my experience, was very centralized. And we had a database team and we had a network admin team and we had a compute team. And you centralize the expertise and then every service picks and chooses from those things. Amazon was the opposite. Amazon was exactly the opposite, where every service team was a two-piece team, or they call them service teams now. Two-piece team is the old word. But in general, you have everyone you need to do the job. You don't have an SRE team

that checks things that are live. The developers who wrote the code are the ones that are on call and you don't have a DBA team. You have services that you rely on, right? We use RDS and all these other things internally because we just create those services, but it's a full stack team through and through. It's what I considered a true DevOps team and seeing how much duplication there was across the board. It was like, wow, actually having a DBA on every team is really expensive. Like that would be great because they don't need it all the time, but you have to be kind of a generalist at some point. You can't go as deep in some of those areas and seeing if the shift is really, we're going to lose a lot of people and we want to maybe run things a little

leaner. the leanest way to do that is to centralize the expertise and you centralize where you, who knows how to do something deeply. And then everyone picks off of their queue, right? You'll add another ticket to their queue, wait for it to come back. But things slow down because that queuing system just takes a little while. That also causes a lot of gaps. There's a lot of areas that get kind of fallen.

Oh, I didn't know we were doing that before. As I saw a lot of teams in other places at Disney and other companies try to move to a DevOps model, they didn't realize the gaps. They didn't realize, oh, we actually need someone that's an expert in Terraform or in this other thing. And so we had to keep relying on like, hey, can we borrow that person from that team for a little longer? But then when it breaks, they're not on call for your service and you have to find someone that's an expert. And, and there's services, there's all these things. So it's like these shifts in organizational structure cause gaps. And as those gaps show up, there's things that slip through and you don't

know who was responsible for it. And you always kind of, you don't know about it or you assume someone else is going to do it. And those are the things that cause a lot of that risk is once there's a gap there of expertise or responsibility, you really have to figure out, hey, when this isn't working in the ideal way we think it's working or when that API gets changed and we have to upgrade something or libraries roll out, whatever it is, who's testing this? Who's making sure that this is validated? There's automation you can do, but for the actual running of services and making sure that APIs are running like that organizational structure really impacts how the services run. And I see Amazon moving more towards a centralized expertise situation as service teams become smaller and they don't necessarily have all of the experts they need

to run a service with as much breadth as it has in the past. Like Kubernetes is one of those services that touches a lot of AWS. There's a lot of things involved behind the scenes on EKS. It's not just a bunch of VMs, which is like a full stack VM. We just stamp them out. No, like there's, we use a lot of the internal services. There's a lot of stuff that is reliant on, EKS relies on those other AWS services. And so you have to make sure that none of those gaps get missed. And you can be as careful as you want to, and you can checkbox everything and make sure it's carefully migrated. But at some point, there's a handoff of on-call or responsibility. And those are the things that really cause problems at organizations for everywhere, once you're running software in these environments.

Well, specifically you said, I suspect there'll be a major outage in 2024. No amount of multi-region redundancy will project you. And then you said it's because of an increase in large-scale events. These are things that I'm not even aware of, like these large-scale events. You're already teaching me.

You're not even podcasting with us yet. So I love it. This LSE thing, you know, is that these large-scale events are not something they have to, they're not incentivized, as you say, to announce these things. It's the things that hit the customers that they have to report on, and they are quickly swept into the all-greens tab, essentially. But then you go on to say that Amazon is operationally strong. You say they're much stronger than any company. It's not like you're sitting here pooing on them.

You're just predicting like, hey, the gap there of people is an issue and the centralizing is an issue. And then at some point it's going to bite us potentially. But then you say they're pretty strong. But that strength requires people. And when you reduce your headcount and they're eliminated, things are going to suffer. Practice is going to suffer. Operational practice is going to suffer. Yeah, LSEs is a term internally that we use, but it's also been in plenty of books about

AWS and things. It's just, it's a way that we measure things internally, right? Before a dashboard gets updated, we need to make sure internally, like, is this actually down? Like, we're not going to tell someone, hey, something's down before we know for sure it is. And all the graphs and dashboards might say, Hey, yeah, this is down, but someone's going to verify it. Like there's at some point where say like, Hey, is this

actually down? And, and you can do that variety of ways, but a lot of times it's just like, Hey, this dashboard says it's red. Now internally, uh, this person said it's red. Let's me verify where and how that's down. Right. Cause Amazon's AWS is giant, right? Like it's however many regions across the globe, like this might be a certain sliver of customer in a certain region. This might only be one AZ. And like one AZ out of a hundred, like, am I going to,

where are you going to update that? I'm like, oh yeah, 1% is down. What does that actually mean? And so those sorts of things, large scale events though, usually affect multiple services or multiple AZs so that things are happening at a larger scale of like oh this one customer might have a problem right now

and those sorts of things just happen as things progress you know we push out new code we make changes when you talk about a rollout right like i talk about like if you want a hundred you want to ship code a hundred times a day i was like well at amazon that's like one commit change because it has to go out a hundred different ways like Like that's not, this isn't a hundred different shipping. This is like the one thing got shipped a hundred times. And every single one of those might have some variable that isn't the same. There might be a service that's different or an API that's different or whatever it might be.

You have to be aware of those things that it's not just the get commit works on my computer. It works in this in pre-prod. Now we're good for the rest of prod because prod is so big. And so you have to be aware of some of that stuff. The operationally strong part of it, there's a thing at Amazon that's the weekly ops meeting, which is every Wednesday. And it's one of my favorite things about Amazon. I mean, you can read about it.

They have a service wheel that they spin to get an update from one of the, or a couple of the 200 services, but they go over the wins for the week. They say, Hey, here's what we did great this week. And there's a, there was an ops wins, uh, an email list, which was wonderful to read. Cause like, Hey, we changed this flag on this load balancer and we saved this percentage of, you know, latency or, or money or storage or errors, whatever it was. It was like, Hey, we celebrated those wins over and over again. I'd never seen that anywhere else where it was like, Hey, actually I'm writing up. This is one thing that one team did. Here's

everyone celebrating it. And that's fantastic because that operational challenge of running software should be more visible in the work you do that you think, Oh, all I did was like, I cut some logs. Like who cares? Like, no, no, no. You're going to cut some logs. And then you're going to actually like project that out and say, how much did that save us over the year? And then you're gonna say, Oh, well actually like, that's a big deal, right? Like that's, that just saved all my, you know, my, my pay or something for the year, whatever it is. Like there's something in there that even at a semi smaller scale where you can say like, Hey, this is great. And on those ops calls, like distinguished engineers are on the call you can say like, hey, this is great. And on those ops calls,

like distinguished engineers are on the call. Like they're running the call. They've been at Amazon for 20 years and they're talking to these people saying like, hey, we had an outage with, like they'll go through wins and they go through COEs or things that could be improved. And then they go updates from service teams. And that cycle has been really great

just to see how ops can be done really well. And everyone can be on board for it. Because as they talk about the wins, they always do that first. They talk about where it can be improved and you'll get those distinguished engineers that'll really talk about like, hey, you coupled identity to this load balancing in some way that may cause problems in the future. And they can kind of predict and see those things that I never thought people would be able to see. And it's just like, you've experienced this before. This has bit you in the past. And now you understand when things should be coupled and when they shouldn't be. And that operational, like that knowledge of just experience of running things at certain scales is so valuable. And they

try to spread that out through all the service teams. Everyone's welcome to come to the ops call. It's an internal stream and like, you can just see how they're helping everyone improve and predict what's going to happen in the future. And those sorts of things have been great to learn from and just say, like, actually I wish more companies elevated their operations, elevated their, Hey, we are running the software. It's development and writing the code and all this stuff we give for people to write code and solve the problems is fantastic, but we need it on the other side, too. We need it for running the software. What's up, friends? This episode is brought to you by our friends at Sentry.

Check them out at Sentry.io. S-E-N-T-R-Y.io. Code breaks. Fix it faster with Sentry. Don't just observe. Take action. The only app monitoring platform out there built for developers that gets to the root cause for every issue. 90,000 plus growing teams use Sentry to find their problems fast. GitHub,

Disney, VMware, Airtable, Autodesk, monday.com, Miro, Tunnel. I could just go on literally 90,000 plus teams trust Sentry to fix their problems fast. But how they do that? They have error monitoring, session replay, performance monitoring, code coverage, all the things it takes to make an application run well in production. Sentry has got you covered. Again, check them out at sentry.io. That's S-E-N-T-R-Y.io. And make sure you use our coupon code changelog to get $100 for your error monitoring with Sentry. Once again, S-E-N-T-R-Y dot I-O, Sentry dot I-O. Brings me back to my offensive lineman metaphor for ops teams you know i don't know if you're a american football guy justin but you know the o-line they just don't get any respect outside

of the team because they're supposed to do their job when they do their job you don't notice them you only notice them when they fail and so when there's an outage or a sack right not a not so silent sacking you have their big face on the television. Like this guy missed his block and therefore the quarterback got sacked. But the nine times out of 10 that he made his block, we're not talking about him. And that can be very difficult on an offensive lineman unless that lineman has the respect and praise of his quarterback and his peers on his team. And his wins are celebrated by them. And that's how you get real camaraderie.

And you have people who are willing to do the quiet things, right? To do the things that no one notices when it goes well. Otherwise, there's no glory there. There's no praise. And so I think that is really cool. Because outside of AWS, I mean, we all assume everything just honky dory because that's the way things work as customers. We only, we only are mad when it's not working right. When we have that outage, otherwise we're not praising our, our ops people.

So it comes down to that trust, right? Like if you trust your defensive end to always get the, get the blitz, like I'm going to roll to his side every time, right? Like it's not even like a question of like, Hey, this person I'm going to roll to his side every time. It's not even a question of like, hey, this person's always going to make that block. And Amazon is always going to keep that service up. That's something that you build that trust over time. But if they get injured or if they have an off week or something like that, hey, guess what? My trust just dramatically goes down. This isn't like a, hey, I understood you were off for a little while. Like, I don't know how long until you get back to where you were.

Like that trust is really, really hard to regain once you lose it. Once you get blindsided from the back, like that's a, that's a problem. And, and those are the things that you really need to be careful of as, as someone who is works in sort of this high trust environments. Same thing with security, right? Like security is a super high trust. If I have a security scanning tool and I get a breach at my company, like I'm going to be looking somewhere else. Like this is just like, Hey, guess what? Like that trust is gone. I'm

not even going to give you a second chance. Like this is, I, I can't have that news, right? Like this is those sorts of things. Like they, there's some things that like when you have, you need up time, you need up time. And if you're relying and you've moved everything into the cloud, and you say, actually, I pick US East 1, and it's not the best. Let me go over to US West 2. Maybe that one's better. Maybe go somewhere else. And at some point, you're just going to keep moving things around to find where you have more trust.

Yeah, well said. That's why I think that some of these activities are so short-sighted. And maybe it's the structure of having like public quarterly financial reports that you must show up into the right you know unless your stock you crash or whatever but eroding that trust for short-term gains is just like long-term not smart move right well and i don't even pretend to know all of the long-term investments that a company at Amazon size has had. restaurants and other second order effects of small companies that rely on them being there. And Seattle as like a tax revenue or like that affects like, like the traffic in Seattle, you can tell when the Amazon in office days are in Seattle because like the traffic is bad

because just Amazon's coming back to work. And in the days that they're off, like I've had so many friends that work at other companies and like, Oh no, I don't go into office when it's Amazon's return to office days because I'm going to be stuck in traffic for double the time. Wow. And those sorts of second order effects of like, you're affecting other companies ability to do work in an office just because you are so large and you had so many of these investments. Like I, I can't project or even predict what all of the longterm incentives are for them

to force people back into an office and to silently get rid of people and to do mass layoffs. Like, I don't, I don't know. I just know my own experience and I don't know what it's been like for people that I've been close with and worked with that have had their lives turned upside down. We're like, Oh, I don't have a job this week. I don't know what to do because hiring market's kind of down and things are rough out there. Yeah. That's why the, I guess, is it a theory with regards to it being about layoffs causing stock to drop, you know, stock price to drop? Seems like, I don't know, maybe not like 100% sound because I've seen layoffs where the stock price immediately popped because it's like, hey, this company finally got a handle on their operational costs or something. And investors like that, like if they finally woke up, I think the latest Spotify one, the stock went up following the layoff. And so that's not

always the case. And also the market is fickle and short-sighted and like there, maybe your stock will drop for the next six weeks and maybe it's that quarterly financials. That's the problem. But over the next two years, it's going to be just fine. Like it's going to be a roller coaster, but as long as you're actually providing value in the market and not getting bloated operationally, the stock's going to rebound. And so it seems like there has to be more to it than just that, but I'm not saying it can't be that. It just seems like it's a, a simplistic explanation for maybe a nuanced and weird phenomenon. For sure. And yeah, I don't want to assume that like the stock price is the only reason for this stuff to happen. Right, right, right. Yeah, because I forget business, for the website,

someone had a report on like, if you make a thing and it costs $10, you sell it for $10 and it costs you $7 to make it. If you instead raise the price by a dollar or you lower your operational overhead down to $6, you will gain more stock market value by lowering your overhead to $6, you will gain more stock market value

by lowering your overhead to $6 than you will by raising your price by a dollar, even though it's a dollar change either way. But your overhead to create the thing is much more valuable if you can lower your overhead to actually create it. And that is just like a common thing that a lot of stock market,

like a lot of companies do for the stock market, say, hey, we're lowering operational overhead for this thing. And the other fascinating thing I read this last year was the book, both jobs. If you ever read the book, it's all about like how a lot of jobs are meaningless. I don't know if I'm allowed to say, but I said twice,

sorry, I was going to market. You can say it. Go ahead. But it was a fascinating read about how a lot of these jobs, especially at large companies are not about actually doing the work. You can say it. Go ahead. they're having all those people, they're organizing it in a way that they say, hey, we can go in this direction now because I have enough people that are doing that in this direction. In a lot of ways, I feel like my role in DevRel does fall under that, where I'm not

doing the work. I'm enabling someone else, a customer, to kind of come through and then use the thing that someone else made. And I had to come to grips with that where I'm like, am I actually adding value? I need to read this book because I'm not sure I agree with it. I mean, I think organizing folks and DevRel and that in particular, like there's a certain, you know,

amount of mind margin that you have as a value worker, let's just say, to use the language you're using. And if I can put people in place to organize those people better and drain them less than I get more value from their actual work. I still feel like those are value. Those are not BS jobs personally. And ideally you could have a hundred people that all just work in the same direction and they wouldn't need coordination, right? Like, like the reason we have managers that we have to be able to coordinate information and timing and all these things. Like ideally in a wonderful world that doesn't exist, people could just work in the same

direction, do the same thing without needing the overhead. And so at some point we have to add that overhead to be able to align people in ways that make sense for a larger group to do more work. Because me doing work on my own isn't as good as five people maybe doing work in the same general direction. It's not five times better, but it's more than two times better. And so at some point I can add a manager to help us guide that way. And one of my favorite quotes from the book is like, the one thing that I took away from it was at some point we got so good at manufacturing things that there's not really a shortage of being able to create things, especially in technology.

These bits are free, essentially. At some point, we have enough bandwidth and magnets. We're not manufacturing. It's not a manufacturing limitation. What we have to manufacture is desire. People need to buy things. We need to manufacture a way for them to actually buy stuff. And that's like marketing, in a sense.

Marketing is just that. We're going to tell you, you need something. And then maybe 1% of those people actually come back and buy it. And in your manufacturing and need rather than manufacturing a product. And that's where like the marketing space exists. And that's where a lot of people that say like, Hey, I need you to go do this is we need you to go do something else, even though we have the capabilities just to do it on our own. But I can't have enough people to do it together to make a bigger impact. It's difficult to find the sweet spot when it comes to support roles,

when it comes to management and leadership roles, because some of that isn't necessary, especially as an org grows and useful and valuable and worth more than you're paying them, of course. But there's also an opportunity for you to have too much of that. And at that point, you do have people whose roles aren't bringing as much value as that person could bring in a different circumstance.

I'm not saying it's necessarily the person who's not bringing value, but we have too many managers, for instance. We have too many dev rel. And it makes sense that that's a harder thing to measure. What's the right number of DevRel for AWS, Justin? I mean, who knows what the answer to that is, right? We could go to get PhDs on coming up with an equation. And it always changes.

Yeah. Whatever you decide on isn't going to be the same next year. Right. Because the market changes and customers change and needs and products change. And so as the world shifts, I don't know how many managers we need. I don't know how many the same next year. Right. Because the market changes and customers change and needs and products change. And so as the world shifts, I don't know how many managers we need. I don't know how many people in DevRel, I don't know how many engineers we need.

We just know at some point, we don't have enough of something because that area of the product or the service is suffering. So how do we get more of it? Well, we add more people because we assume that Mythical Man month is going to take some effect of adding more people

to it. But like if I had enough people, I am going to get more out of it. And so what is that balance? You just keep having to adjust it over time. Let me slightly change the subject, but keep it on point. I think this will be interesting because I would love to have your take on this. Your prediction of a major AWS outage in 2024 reminds me a lot of the predictions that were made late 2023 when Elon Musk bought Twitter and laid off something like 70% of the company. pretty much like makes sense to me was like there will be a major twitter failure operational technical operating at scale like twitter will be down and gone soon sometime this year and i mean this hasn't been hiccup free by any means there have been outages and stuff but on the whole it seems like it's going okay over there in terms of twitter.com and the apis or x.com now i don't know mine's still twitter.com but it says x it redirects they haven't like fully you don't

know the redirects but yes the platform is called x curious like and then a lot of people that made the predictions like well these things take time and so eventually it will fail i don't know i have no idea i have no insider knowledge i'm curious just from a guy who understands systems of scale better than I do. What's your take on that? Like were they just way over bloated in terms of head count people? They didn't need all those people. You could run it on a skeleton crew and that's what they're doing. Have there been outages that I don't know about,

like major ones that would make these predictions true? Is it just, is a matter of time? I don't know. What are your thoughts? I have no insider knowledge but i had friends that that joined teams at twitter and were eventually let go and i do know that they got rid of a lot of features and products as part of this take i ran a community at twitter when it was twitter.com they had communities and i had one for the job tech jobs and i was like hey let's just start a community on Twitter for people to find tech jobs. I don't run it anymore.

It doesn't really, I think it still might exist. I don't know. They're not in the menu anymore, but they got rid of that. Spaces is still around, but pretty much everything else that I know they were working on just went away, including all of their safety and moderation teams. And so by removing some of those things, that's a lot of people to do new products, to build new things. I also know the operations side of things did take a hit.

A lot of times, things do run for a long, long time. I have servers that I used to maintain that were on for years. We never had to touch them. The website worked. Once you get it working, the port's open. I'm okay. And I can't change it. I can't scale up. I can't do new things, but it works and it's there. And I know that through a lot of the API changes of like, you have to pay for access.

A lot of the like public visibility for tweets were turned off for a while. There's a lot of things that changed that reduced the amount of just API calls and tools that were using it and reliance on it. Certainly public API would be a huge reduction in overhead. Right. I just saw on New Year's Eve, right, the safety Twitter account or emergency response Twitter

account lost its API access. It was like, oh, we used all of our credits. We're done. You can reduce your load by a lot and just say, hey, we're just going to cut back on a lot of this credits. Like we're done. Like you can reduce your load by a lot and just say like, Hey, we're just going to come back on a lot of this stuff. And maybe that was just something that you could turn off some servers. Maybe you could scale down because you're,

if your volume of calls goes from, you know, a hundred million to a hundred thousand, like guess what? Like you can turn off a good portion of your servers or just not really worry about it anymore. It's just like, Oh,

we'll just scale down. We're fine. We don't need to make changes to scale things. And we're not going to hit new scaling limits because Twitter already was hitting those limits over and over again. Because there are these stages as you run software where it's like, hey, our Postgres database was fine

with just a single replica. And now at some point we need three because we hit a scaling limit. And if you've made it to that scaling limit, you're like, okay, we already have the three replicas. Like this is going to last for a long time. we don't need to worry about storage because our volume isn't increasing as much you just keep hitting those stages over and over again and at a like op side of things like when you're running the services you can project out maybe a year if like

if the growth is stable here we can go until here and then at some point we need to switch to something else and you're constantly in that like re-platforming or or just redoing your infrastructure to make sure you can hit the next level of scale because you can't predict what areas are going to necessarily be the next bottleneck. But if you remove a lot of those scaling requirements, like just things run for a long time, like Nginx is really powerful. Like Disney Plus, the front end of Disney Plus, when I there, was just like a few engine Xboxes. I was amazed at how much you could do with a really small amount of compute. And I was like, oh, actually, no, that just works. And that's just scales. And that's kind of amazing. If you just set up with some of those small things as best practice, you don't need everything

for everyone. It's just like, oh, figure out what scale you're at and just run it. Yeah. And so I think that Twitter will last for a very long time based on the scales they were already at and the reduction

that they've had since then. And I don't think that that's going to necessarily change. I do think that there's, you know, again, there's a lot of cracks that have shown up

over the time of people not knowing who was responsible for something. I uninstalled the app, but like the Twitter mobile site wouldn't load on my phone for months. It was just like, I couldn't go to Twitter. I was like, actually app, but the Twitter mobile site wouldn't load on my phone for months. It was just like, I couldn't go to Twitter. I was like, actually, no, this is

kind of nice. I'm okay. I'm okay if I only do it from my computer, but it wouldn't load on my phone anymore. So I'm like, you know what? I'm okay. I won't go. And that's okay. So there are those weird edge cases that just no one probably caught because there was a gap there. But for sure, they were at such a large scale that notching down on the requirements allowed them to kind of figure out where they needed to fill up you know or or reach in back to where they were i think that's a fair point i think that's on point the only thing that i've noticed recurring where i'm like this is just a failure in having enough people to actually fix this. The redirect service, t.co, all the links get redirected to t.co.

Specifically inside of the phone app, it just doesn't always work. A lot of times it just fails to load page and you click back and you click it again and it works the second time. Every single time. Yeah, it's not every single time for me. It's probably like 80%.

100%. But yeah, so there's a situation where it's like, this clearly could just get fixed by somebody because it worked previously, but's probably like 80%, 100%. But yeah, so there's a situation where it's like, this clearly could just get fixed by somebody because it worked previously, but they probably haven't noticed because they're just on a skeleton crew. And I do think that that's one thing that as those small issues show up, they're going to take longer to fix.

When that was a problem before and you're like, oh, you know what, this is probably fine in the next app release. Get the new app release, something was going to change, someone's going to scale something up. But with fewer people, you just, you can't pay attention to everything at the same time. And so you really have to focus a lot more, which I think is exactly what, you know, Elon has been trying to do with X is focus it in almost a different direction. But you can't focus on all of those little things that were just like, Oh, this is a bug that's been bothering someone

for so long. When I started at Amazon, one of my main goals was to in inside your AWS account, if you click down on your account, there's account number there, but you could never copy that easily. You couldn't, there was no copy button next to your account number. And I wanted to do that all the time. I managed dozens of accounts and I was just like, I always wanted to click that button to like copy it. But if you copied it, it would get dashes or get the next line. One of the first things I did was I opened a ticket internally. I said, I need this to be a copy button, please. Like just add copy buttons to my account number, my region, whatever it is. And sure enough, that ticket got solved. That was like my main, like I wanted that to exist as a customer. Like I can, I am happy now because if I'm using this

again, I can drop that down. I can click the copy button on my account number and it copies it without dashes, which is exactly what I wanted. And in some of those little things like that fix happened pretty quick because as a customer, I asked my TAM for it, who asked someone else, but at some point they got lost in the, like, who does, where does he ask? Like, he didn't know where to go as an internal employee. Like, I know I just spent like an hour to find like, who's responsible for this. I need this widget to have this thing and find their queue of tickets. And I don't care if it's done now,

you know, but if it gets done, that would be awesome. And sure enough, like I was able to find that ticket queue, put in that ticket and I, you know, it worked. And I was like, that's, that is amazing to be able to

be an internal employee and fix a bug that was affecting me as a customer. And, and then see that, like just roll out. And I'm like, cool. That's, we're good. That's awesome. Let's talk ship it. Let's close it off with a quick ship it conversation. Obviously we're,

we're back. We are so back as the kids say, but we are going to bring ship it back. We have some stuff going on recordings happening, maybe just in your perspective on ship it. Obviously it's going to be the old show, but it's going to be a new show. It's going to be your spin on what it is. Of course, Adam and I still highly involved and excited about the reboot, so to speak, but what can folks expect from ship it from you, from the people involved and what's going to happen over the next few months of, uh, of this

old new show? Yeah. I mean, like I said, I loved what Gerhard was doing with the show. I love the topics that he was already covering and some of the guests he had on. And I want to continue that as well. I want to focus on that topic space, everything after Git push. What do we do? CICD pipeline, security scanning,

system scaling, whatever it is, all the way from observability, SRE, all of that stuff's involved in the not writing code side of things. Like how do I debug a Linux server? Like those are things that not a lot of places focus on. And I want to keep that focus of the topic of just shipping the code, getting some great guests on,

focus on areas that are, you know, running code. If you run production code in any sort of environment, I want to hear from people because it's not just a web service. And in my time at Disney animation, we had almost no web services. Even the Disney animation website wasn't run by us. It was run by another company. We just did rendering. We would render stuff and we have some internal services. How does that look? Why is that different? People still wrote code and we still did stuff and i want to know what those different environments look like for people because running software is not the same thing it's not always just an nginx with a back-end app right like a lot of

places look really different and there's so many variables in that that i want to talk to a lot more people and give people more exposure to what it actually looks like in a different way if you're in a hospital how is that different than a streaming service? Those things are very different environments and have different concerns and different needs for what they're doing with their software and infrastructure. So that's the first thing. I want to keep some of those people coming in. I also wanted to have some things that I love listening to podcasts in general and the things

that I want to hear. I don't want just a news show, but I want a couple news topics. I want to know some things that are relative or something that the hosts, whenever I'm listening to show, I want to learn what they learned this week. Some of my favorite shows that I've listened to in the past always have like something that is personal. That's like, Hey, I did this thing. I solved this problem.

And now I, whatever. It's like a small thing. It's not like, Oh, everything's groundbreaking every week. It's like, no, like I learned how to make a dashboard on my Raspberry Pi. Here's the thing I use. Here's an open source tool, whatever it is. And so I have some recurring segments that I have ideas for to make that fun. I'm bringing on an awesome host autumn with me because she has such a great different perspective and different experience than what I have from running services in a different sort of environment and with different constraints. So I'm really excited about that.

And then just having those guests come on and learn from them about what products exist, whether they're open source or SaaS products or just different ways of thinking about scaling things. I used to also run a Twitter space for reading white papers on infrastructure. I called it Paper Club. And it was a monthly, let's read a white paper and then just talk about it. It was like a book club for technical white papers. And that sort of like deep dive into technology and where technology comes from has always been fascinating to me because I can learn a lot about how or when I should use something based on what problem it

solved when someone created it, right? Like, do you want to use raft consensus? Maybe, maybe not. Like what problem did they solve when they created Raft that something else didn't solve? And then you can maybe make a better decision about which tool is the right one for you. So those sorts of deep technical topics are something I also would love to bring to the show and have people come on and talk to. I remember one of my spaces, Eric Brewer, writer of the Cap Theorem. We were literally reviewing one of his papers, not about Cap Theorem but about scaling like AOL services. And he joined the space and I was blown away. I'm like, I can just like have access to someone like Eric Brewer on a Twitter space.

Like, are you kidding me? Like that, that amount of like shrinking of what the internet is, is fascinating to me where it's just like, Oh, that was what was great about Twitter in the heyday of like, everyone was just there a lot of times and, and he showed up and people were discussing it. And I learned a lot. Like I read the paper, we were talking about it. I said, Hey, why did you do this this way? He's like, oh, because this other constraint, we didn't even talk about in the paper here. Let me tell you. I'm like, oh, that's great to know. Like, I wish I, you know, I love those conversations. I'm looking forward to having more conversations on ship it around those things about like, Hey, this is what we said in the, in the blog post about the outage.

But here's the thing that we didn't say or the constraint that we didn't know about at the time, whatever it might be. Those are all areas that I would love to talk about. Well, we got a link to a LinkedIn post that I think is the only place you mentioned it thus far on the internet. I looked on Twitter. I didn't see it there. In case I missed it.

I posted it on Blue Sky. I do a lot more on Blue Sky now. Ah, the Blue Sky guy. Right. We'll hook you up with our Macedon account for Shippit. Anyways, go ahead, Ed. I was going to say, we'll link it up in the show notes just because there's an invitation there. There's an email address there, all that good stuff.

You can pile in the comments. Just come there and celebrate bringing back this podcast and encouraging Justin and co-hosts to do great jobs with this podcast. And obviously Jared and I will be here along the way as well. But we want to hear from the community. What can we cover on this podcast that is interesting?

What should we really talk about around Git Push and applications being in production and keeping them up and what we're learning, that kind of thing. So pile on that post, share your comments, share your thoughts, email us if you got topics. I've already seen a couple emails come in.

Jared, I know you got a DM or two. So definitely action happening already in terms of what we can talk about with ShipIt. So that's awesome. Yeah, we're planning on starting recording as soon as possible. And I would love to hear more ideas for topics around. If you're running software, I want to hear about it because it's fascinating how similar and different a lot

of these environments are i guess one more plug because i just have to that's what i do is we have a couple sponsors already for this podcast century is thinking about bringing it onto their 2024 plan and i know cinedia already has committed to some episodes of Ship It. And so if you are at a company that can benefit from reaching more developers that Ship It reaches, we want to sponsor this podcast. So reach out and say hello.

Too easy. There you go. If you are a Changelog++ listener, don't worry about it. You're going to just start getting fresh, good Ship It episodes right into your feed. If you are a Master Feed subscriber, don't worry about it. You're going to just start getting fresh, good Ship It episodes right into your feed. If you are a master feed subscriber, don't worry about it. You're also going to just get Ship It episodes. If you aren't either of those,

there's no better time to subscribe to Ship It. We're at shipit.show. There you'll find an email, subscribe, of course, links to all of the popular platforms, as well as a direct RSS feed link for you to pop into your favorite podcast app. So do that. If you love the old show, definitely give us a listen. Hopefully you'll love it as well.

If you didn't like Gerhard's British accent, well, we have a non-Brit here. So maybe it's a good time to give it another go. Of course, Gerhard will be coming to a Kaizen near you very soon. Very soon, yeah. He is definitely still very much involved. He just does not have the bandwidth for Ship It right now.

Thankfully, Justin has the bandwidth. He also has the expertise and the desire to bring this awesome show back with us. So we couldn't be more excited, Justin, and looking forward to what you come up with. Yeah, and thanks to you for the opportunity. When I reached out, it was just I was catching up on episodes, and I was like, this needs to exist. There should be a show about this. I really appreciate both of you, you know, replying to the email and letting me know like, Hey, this is, this is where we're at and this is where we want to go with it. And what are your ideas? Cause that is having that flexibility and being able to rely on the audience and network you've

already built for it. Like I already know people are out there that want this content and being able to continue on the great work that Gearheart has been doing is just, I'm blown away being able to do that not starting from scratch because that is so hard to just do that for so long, starting from scratch and you all have built such a great network here that I wanted to be able to lean

on all the community and people that are involved already with Change.com. That's awesome. Let's go from one to two all right thanks justin really appreciate you telling your story with us and like we said looking forward to what you come up with yeah thank you well there you have it ship it is coming back ship it dot show if you have not subscribed yet and uh shipping to a podcast near you everything that happens after get push and a big thank you to justin for coming on the show sharing his thoughts and more

importantly joining the ranks here at changelog as one of our hosts he is going to do an amazing job with this podcast and we're excited about it hope you are too and coming up this friday we have our next edition of kaizen kaizen kaizen 13 is coming at you, and I think you're going to love it. We get into it. We get into it. I'm telling you, we get into it. That's coming up this Friday. Stay tuned for that. Once again, a big thank you to our friends at Fly, our friends at TypeSense, and to the Beat Freak residents here at Changelog. Thank you. Thank you for those beats. That's it. The show's done we'll

see you on friday Thank you. you