The Pragmatic Engineer - Shipping projects at Big Tech with Sean Goedecke

Episode Date: December 18, 2024

Supported by Our PartnerDX⁠ → DX is an engineering intelligence platform designed by leading researchers—In today’s episode of The Pragmatic Engineer, I’m joined by Sean Goedecke, Staff Software Engineer at GitHub. Sean is widely known for his viral blog post, “How I ship projects at big tech companies.” In our conversation, he shares how to successfully deliver projects in large tech companies.Drawing from his experiences at GitHub and Zendesk, Sean reflects on key lessons learned, and we discuss the following topics: • Why shipping cannot exclude keeping management happy• How to work on stuff the company actually values• Why you should take on extra responsibility to get projects done• Why technical skills are still more important than soft skills• Soft skills you should learn: including learning the “management lingo”• First-hand remote work learnings: advantages, disadvantages, and how to thrive in this setup• … and much more!—Timestamps(00:00) Intro(01:50) An explanation of shipping(05:35) Reasons management may choose to ship something customers don’t love(09:20) A humbling learning from Sean’s time at Zendesk(13:27) The importance of learning which rules need to be broken for good business outcomes(15:28) Common obstacles to shipping(18:13) DRI: Directly responsible individual(23:06) The value of strong technical skills and why moving fast is imperative(28:44) How to leverage your technical skills the right way(32:16) Advice on earning the trust of leadership(36:10) A time Gergely shipped a product for a political reason (38:30) What GenAI helps software engineers do more easily (41:08) Sean’s thoughts on GenAI making engineers more ambitious (43:20) The difficulty of building AI tools(46:10) Advantages of working remotely and strategies for making it work(52:34) Who is best suited to remote work(54:48) How the pandemic provided a remote work trial for Sean(56:45) Rapid fire round—The Pragmatic Engineer deepdives relevant for this episode:• Software Engineers Leading Projects ⁠https://newsletter.pragmaticengineer.com/p/engineers-leading-projects⁠• Shipping to production ⁠https://newsletter.pragmaticengineer.com/p/shipping-to-production⁠• Paying down tech debt ⁠https://newsletter.pragmaticengineer.com/p/paying-down-tech-debt⁠—See the transcript and other references from the episode at ⁠⁠https://newsletter.pragmaticengineer.com/podcast⁠⁠—Production and marketing by ⁠⁠⁠⁠⁠⁠⁠⁠https://penname.co/⁠⁠⁠⁠⁠⁠⁠⁠. For inquiries about sponsoring the podcast, email [e-mail]. Get full access to The Pragmatic Engineer at newsletter.pragmaticengineer.com/subscribe

Successful projects, there's a narrow path to success and there's a million paths to failure. A narrow path to success. The default state of a project is to fail. Projects only succeed because people drag them kicking and screaming to the finish line. If you leave a project alone and nobody's worrying about whether it's going to ship, nine times out of ten, that project will not ship. The ability to kind of come up with a technical demo kind of moving mountains. One demo of like, hey, look at this thing happening, can cut through weeks of conversation about whether it's possible.

And if you just go off and produce that, you can really kind of. blow people out of the water. In the context of like a large company, being technically strong if you work on software is really a superpower because it makes you connected to reality. It makes you connected to the real world to the actual product that you're building. It's sort of like a lot of the work these other people are doing around strategy and marketing and figuring out what needs to be built. That's a huge multiplier. How do you ship projects at big tech companies? Sean Goddiki has worked as a staff engineer in two companies that employ thousands of software engineers. He spent five years at Zendesk and now works at GitHub. Sean wrote a blog post about this.

topic titled How I Ship Projects at Big Tech Companies. The post went viral online and sparked a lot of heated discussion. In this episode, with Sean, we dived into, what does shipping projects actually mean when you're working inside a larger company? What is more important in order to be seen as someone who ships projects, your technical skills and engineer, or your soft skills to navigate the organization? The story of why Sean was considering looking for a new job when his manager told him to not worry too much about test failing in the staging environment.

How do Gen. AI tools change the ability for us, engineers to ship projects. Sean is a great person to talk about this as he's worked on the GitHub co-pilot team and now works on GitHub models. Advice Sean has to succeed in a full remote team when working across time zones and a lot more. If you enjoy the show, please subscribe to the podcast on any podcast platform and on YouTube. Sean, welcome to the podcast. Thanks. It's great to be a big fan of all your stuff going back many years. So really exciting to get to talk to you. So to start off, what is shipping? In your post, you wrote, and I'm going to quote you, I know it sounds extreme, but I think

many engineers do not understand what shipping is even inside a tech company. So what is it? My post kind of came from this experience of seeing a lot of engineers around me and myself in the past getting into this frustrating position where they were delivering code. They were feeling happy about their output, particularly the volume of their output, but they weren't getting the kind of recognition they felt they deserved. and the sort of management chain seemed happy or seemed unhappy or even worse just seemed totally indifferent to what they were doing.

So that's kind of the context in which I talk about shipping and the context in which I wrote my blog post. So in that context, I say that shipping is kind of a socially constructed fact, which sounds like this postmodern way of talking about things, but it actually caches out really concretely into the proposition that a project is shipped when your management chain believes that it's shipped and talks about it being shipped. So shipping kind of means doing things that your management chain is happy with. And I say management chain because like for small projects, this can be your manager and just your manager. But for larger projects, it's usually a layer or two above

your manager, particularly for like staff roles. And sometimes it's, it goes all the way up to the CEO. And in those cases, the metric of whether a project ships is, is the CEO happy with what with what you've done. That can be a kind of frustrating way to think about things, I think, for a lot of engineers. Certainly the commenters on Hacker News found it that way. I think partially because it relinquishes a lot of control. You know, it's not necessarily about the quality of your work. It's kind of about how it's received.

But I do think you have a bit of a happier experience going into these projects with open eyes. And I think you tend to be more successful if you're actually driving for this kind of like organizational definition of shipping rather than just saying, I closed all the Gero tickets, so the project's shipped. I deployed the code to production, so the projects shipped. That definitely does not mean that it's shipped. You can deploy something to production, and it can be a complete disaster. This episode was brought to by DX, a platform that helps organizations measure and improve developer productivity.

The DX team includes notable researchers behind the frameworks like DevX and Space, so they're constantly asked by leaders, what metrics should we use to measure developer productivity. To help simplify the landscape, DX just recently published the DX Core 4, a unified framework for measuring developer productivity that encapsulates Dora, space, and DevX. The DX score 4 includes four dimensions, including speed, effectiveness, and quality that provide a focused set of metrics that help track productivity at all levels of the organization. You can learn more about the DX score 4 and how it's being using companies like Dropbox, Etsy, Versal, and Pfizer at getDX.com.

core 4. Again, that's get dx.com slash core for something that feels wrong about this definition is, is it kind of sounds like, you know, we should successfully, if the management chain all the way up to CEO is happy and they're like, all right, great job. But it kind of goes really against like, I think what a lot of us feel is shipping, which is does it work for customers and users that people care about it? This goes a little bit back to, you know, like agile, like the original agile. manifesto, which talks about customers, doing stuff for customers. How do you think that is connected? Because clearly, I can see on one and, you know, if the customers are happy, the CEO is happy. But you might have things where, like, I don't know, the customers are not happy and the CEO is not happy.

And like, you've been doing this for quite a while. But surely you needed to reconcile this and also think about this fact. I want to say a couple of things on this. First, kind of you touched on this in the question, but I do want to stress that in any, like, remotely healthy company, your management chain wants customers to be happy. They want customers to be happy because they want customers to be paying money. They want more people to be using the service. You should be aligned with your management chain's goals here. If your management chain want customers to be unhappy and want the company to make no money, then yeah, you might ship and make them happy by deploying code, well, by doing whatever, but you probably shouldn't be at that company. You're definitely not going to have a fulfilling

experience. So I think, like, this idea that you're making your bosses happy instead of making customers happy, kind of presupposes that you're working at a company where those things are in tension. Certainly in my experience, I've been lucky enough that at Zendesk and now at GitHub, people tend to want customers to be happy. The other thing is, sometimes your management chain has good reason to make decisions that are going to make customers unhappy, and sometimes you don't have the context for that. So there's lots of reasons that are very important to the company. We could spend the entire rest of the podcast listing them off, and that might be a really interesting podcast. But just to give you an example, sometimes a company is in trouble and needs

to position itself as being in a particular space. Recently, AI is a good example of this. So you need to ship some kind of box ticking feature in order to, like, have a really positive impact to like the success of the company. Users might not like it. That's a shame. It would be better if they did, but sometimes you need to ship it anyway. I want to be clear that GitHub is not an example of this, like, our AI features are actually top-notch. That's part of the reason I'm here. Another really common reason why companies might want to justifiably ship something that users don't like is security and regulatory compliance.

If you've ever been at a company that's gone for, say, FedRamp, you'll have to ship features that kind of satisfy these, like... What is FedRAM? Oh, I'm sorry. It's a U.S. government kind of regulatory set of requirements. I guess that make it so, as I understand it, the entities within the US government can use your software. Yeah, it's one of the, I remember as well, it's a series of regulations if you want to do business. And there's, I guess, examples, not just this, but a lot of other regulations.

Yep, exactly. And some of these things, particularly in the realm of security controls, mean that you're compromising the kind of customer experience a little bit and doing things that customers won't like, because, for instance, you have to be more careful about the way you manage API keys. You might have to make them expire over time. Users really like having their API keys live forever. It's very convenient, but you can't always do that. Those are two reasons. There's a million others, but just to tie it back to the original question, like the fact that you're writing code that doesn't make customers happy might make you feel bad as an engineer, but it's not necessarily counter to the interests of your company. And there's lots of cases where I think it's

quite literally your job, particularly as a very senior engineer, to kind of suck it up and to prioritize the company's goals over like your own goals and how you might be building the project if it was just your personal project and it didn't have the kind of stakes that working at a large company has. You touched on on how as a senior engineer, a very senior and your staff, level engineering, it is your job to help the business to understand the business as well. What was the point where you recognize this? You learned this because this is typically a learning that, I mean, ideally it comes as you grow, but what was the point where you realize that that is my job now that I'm at this staff or above level? Yeah, I learned this one the hard way.

sort of the story of mistakes. So I was a very enthusiastic and very motivated engineer early on. I was just happy to be there. I was really excited to not be in academia anymore. So I was absolutely absorbing things. I cared a lot about what I was doing, which was good, but it kind of pushed me into a couple of pathways that I think weren't super productive. So one of them was back at Zendesk, you know, at this point, like six years ago,

we had this suite of staging tests for the staging environment that were pretty unreliable for a whole bunch of reasons, not worth going into. But it was my like holy mission for six months to like keep them green and keep them like up to date. And I shipped fixes, I shipped improvements, I shipped reliability improvements to the to the tests. I spent a lot of time and effort kind of like catching up to other engineers. years work where they shipped some feature that changed the UI, which changed the test selector. You know, this was something I cared a lot about. And I felt like I wasn't getting a lot of rewards for it.

But, you know, people seemed kind of happy about it. And then after about six months, I said something in a meeting where I complained about, you know, some feature we did that kind of broke the test again. And my manager said, you know, something like, oh, well, you know, you just don't have to fix them. and I was so mad. I was so angry. I left the meeting. I think I actually left the meeting midway,

which I've never done before or since. I just got up and left. And I went into another room. I went, this was in the office. I went into another office. And I updated my resume. That was what I did.

Was this feeling of like, engineering craft is not like, I don't know, the number one? What was it? It just once against like your core beliefs at that thing. time. I think I had just put in so much work that honestly a lot of it was like a sunk cost

thing. I felt that it had to be important because I'd spent so much time on it in hindsight. I did think it sort of offended my sensibilities to have tests that were constantly failing. That felt bad as an engineer. Yeah, I was so mad. And then I think a couple of days later, like when I recovered, when I kind of felt good again, it's sort of like instilled in me this sense that you can care a lot about something and you can have good engineering reasons for caring about it and the company might not care at all. And if you're in that position, that's your fault. That's not the company's fault. You either have to deal with that and like accept that you're not going to get rewards for doing this work that you're doing and not expect them, or you need to

like reprioritize so you're more in line with what the company wants. That's sort of what I took from that experience. And it took a couple more instances of me being invested in things and kind of, you know, suffering this misalignment between what I wanted and what the company wanted. But that was definitely like the first kind of impulse. And I think I was a mid-level engineer at that point. I don't think I was even senior. But that started like the part. pathway for me to be thinking about these things in the way I do today. But how did you reconcile this, right? Because like I can feel your frustration on why, you know, if you're a test suite, like why I have a test suite, which is not green all the time, et cetera. But in any and how did

you come to understand why either it was not important or not as important as you thought, like, for these staging tests specifically, or what was important? What did the company care about that maybe you missed at the time? I think I made the mistake that a lot of more junior engineers make of seeing something written down as like a procedure or a rule and assuming that it was true. So I had thought that like this social fact that you couldn't deploy with red tests. So we had to get the test green every time we deployed, which was a lot of work. I thought that was like an immutable fact. And then sometime after I got knocked back from this and I stopped maintaining the tests and they were red more often of the time, we had this critical deployment that had to go out

and the tests were red, the tests weren't green. And I remember we were just like, oh, whatever, we'll manually test the red cases. We don't actually need to get the suite passing. We'll just deploy the code. And I was like, oh, yeah, when the company needs code to go out, the company will find a way to get that code out. And these like rules that are in place to like keep code quality up and stuff, those aren't hard rules. those are like soft rules. Those are rules that will bend if the company has like a clear and present need

to get something done around them. I think that was kind of like, that was the end of me worrying too much about those tests, I think. Yeah, and I assume like, I'm just going to assume that these tests were more flaky. They were more less reliable, closer to end-turned test, which enter-in testing suites are always a huge headache for every single company that does them and you know there's yeah i always see companies investing a lot and then scaling back and all those things so there there's also that far right we're not talking about like unit

test or test that that that signal if they break there's a 100% chance of regression or like 99% chance of regression no no and end to end tests operating on a small set of accounts such that one test could change the state of the account and fail a subsequent test ruby test capybara and selenium for anyone listening who's in a similar painful situation so jumping back to shipping projects and production. In your experience, what are things that get into the way of shipping to production, emphasis being both on shipping and production? Yeah, I mean, this is another list of things we could go through for the rest of the podcast. Because successful projects, there's a narrow path to success and there's a million paths to failure in my experience.

A narrow path to success. Yeah. Projects want to fail. Like the default state of a project is to fail. Like projects only succeed because people drag them kicking and screaming to the finish line. If you leave a project alone and nobody's worrying about whether it's going to ship, nine times out of ten, that project will not ship. So yeah, many, many things. To get concrete, one like common category of things is that it's some detail nobody thought of. So just to do a rapid fire of like examples from my experience,

hey we got to like a week before ship and we realized we've forgotten to do the security review or we've gotten to like before ship and we thought team x was going to build this thing but actually they they thought we were going to build this thing and this thing doesn't exist or we were relying on this service but this service is actually you know in beta it's not actually deployed yet you know it's slated to go out Q2 next year sorry or hey this service is in production but you want to run a thousand requests a second through it and it can only support 10 and it's just not designed for your stuff. So it's just endless, endless, endless, like,

you're coordinating with some other team and there's a misunderstanding about who's delivering what. And in big companies, like, projects need to go out with, on the order of, like, five to 25 teams coordinating sometimes. Like, there's a lot of cooks that go into, like, shipping anything in a large company. And it only takes, like, two of them to be really badly misaligned, to sync the whole project. So one thing I've observed at companies like this where it is really complex to ship,

as you said, five to 25 teams collaborating together, which is a lot. But again, if you worked at these larger companies, the ones that do you have or similar size, it does happen. But there's two ways I've seen people respond to this who are tech leads specifically. One type of tech lead goes like, you know, puts up their hand, like, look, this is like too much for one person to do. Our TPM was not on top of it. Technical program manager who typically coordinated. It's like, it's not my fault, right? Like, you know, what can I do? And then there's also the techlies who like, who don't have this response and they actually, one way or the other,

go through it. But what is your take on this? Because when you do have such complexity, it can feel like a full-time job of just coordinating. And, you know, there is some pushback that I hear from engineers warranted or not saying like, look, this is not software engineering. This is project management and my main job is software engineering. I would need someone else to do it. How, when you were in the situation, what was your kind of response and also, especially keeping the mind that you do want to ship in the end of the day? I think there's two coordination tasks that you're sort of talking about there and they can, luckily they can be done in parallel or I don't think anything would ship at big companies no matter what. There's the task of making sure every team is like actively working on

the project and coordinating like in communication and agreeing that they're going to ship by a certain date. That's what I see as more the the TPM or program manager role of coordination. But there's also in parallel, I think, a purely technical sense in which these teams have to coordinate. And maybe you've worked with different TPMs than I have. But in my experience, you need an actual engineer on the team who is a full-time engineer, usually a senior or staff engineer, to have the entire technical structure of the project in their head in order for something to ship. And this is something I said in the blog post, and this is something that actually was not controversial. Everyone seemed to agree, which is that projects ship because one single person

has the technical context in their head. They don't ship because teams agree and manage to coordinate. teams agree and coordinate because one person among those teams has the whole flow in their head and is able to kind of anticipate problems and answer questions because of it. And I think a lot of companies kind of formalize that into having a DRI engineer as well as a TPM and as well as all these other things. There's a single engineer that's responsible for like the technical success of a project. And that's the person I'm talking to when I write my blog post because that's the position I've been in a lot of times and sort of you learn you learn hard lessons.

And DRI being directly responsible individual? Yeah, that's right. So directly responsible individual. That's what it stands for. So I like how you're phrasing this, but is it, am I understanding it correctly that you're saying that if you're the tech lead on a project or you're an engineer on your project aiming to be this operating at the staff plus level, regardless of your, title like you're like okay i want to know what's going on then it is your job to understand

the whole project understand your team's work how it ties with everything else and understand all the parts how it can go wrong and you know make sure that these are addressed at least and then if that's done you're part of the project should be fine you know there's a lot of coordination communication that you cannot control but you can control making sure do i have this whole thing in my head understanding it and having clarity and you know like feeling good about it is that roughly Yeah, that's right. That I think is what I'm saying. The question of whether it's your job is an interesting one. I don't actually know if people typically do this work because it's their job. I think they typically do this work because they care about the project succeeding and they recognize that it's not going to succeed unless somebody steps up and kind of thinks through the whole thing end to end. But whether it's your job or not, if you are in this position of shipping these projects, it can feel like thankless work. But in my experience, it is really not thankless work. People do notice and people do talk about it

and your name will come up eventually as somebody who can be relied on which will lead you to getting more impactful work, which will lead you to promo packets and so on. At least that's what's, I've been lucky enough that that's happened to me. So I think whether it's your job or not, you know, it can sort of be like, perform for the job you want, right?

Not the job you have, maybe. I'm going to plus one that having me the manager on my on my team i always remembered people who help either they bring up they bring up risk that are on projects but they also bring solutions saying oh here's a risk here's what i think i should do and when they do it enough times i'm like okay this person they're actually really good at whatever they're doing whatever their level is and the lower the level right the more ammunition that is for the next promotion and also on on uh calibration meetings for promotions or performance reviews.

It was a very powerful argument when an injury manager brought up, well, this person, they are derisking the project. The project shipped on time because of them, because they stepped in, even though it wasn't their job. So it does take, I think, a while to catch up. But managers are usually, they want to boast about their direct reports at all these occasions, which is invisible to engineers. I agree with you.

It's hard to go wrong with this. So talking about kind of technology skills or hard skills versus soft skills or politics, a lot of what we talked about shipping to production seemed like it's a bit of a soft skill, understand the business, know the priorities, you know, make your management chain happy and these things. But how do you think the technology skills fit into shipping? You mentioned they're important. But can you give examples on how important? important they are or, you know, what comes first, right? Because there's, I'm trying to figure out

if, in your opinion or your experience is one of them more important than the other. I wrote a lot about political skill because I think engineers typically understand that it's important to be technically strong. And they typically don't understand that it's important to kind of be aligned with the politics. That's why my focus was there. But yeah, if I had to pick one, I would say the technical skills are more important. Because shipping projects only succeeds, you can only ship a project when you understand an end-to-end. And it takes a lot of technical skill

to be able to understand a project end-to-end and to be able to pivot and to be able to answer questions that come in from various angles. And one thing that I didn't write about a lot but I think is really underrated is speed in shipping projects. Not just speed of implementation,

but speed of reaction. speed of communication. When something comes in out of left field, you need to have contingency plans like right away. You need to be communicating it right away. When there's a blocker, you need to be addressing it right away because all these things add up. Because there's so many things that can kill a project, like the one like, you know, grand theme that can kill a project is being too slow. Because if you take an extra week or an extra two weeks, that's an extra two weeks that something can happen. you really need to seize your opportunities and try and ship as quickly as possible

sort of before anything can go can go too badly wrong and you just don't have like the fire power to do that without enough technical skill to kind of you know be able to like see throw it end to end to end which is a lot yeah and reflecting on this I'm just thinking back of the best tech leads I remember especially when I was a manager on my team and you know there they weren't the ones who so whenever I think you solved with the problem oh here's a problem this team says they needed another like two weeks to build this stuff. The tech lead that was kind of okay was like, okay, let me talk to them, see if we can, you know, get it down to week and a half for a week, if we can reprioritize.

The awesome tech lead was like, why? What is keeping them up? Let me, I just looked at their code base and I don't really see a reason. I'm actually talking with their engineer and they said like they just need this like small ticket done. I already wrote the diff. It's ready to land for them. We'll get unblock yourself for tomorrow.

And like it's this, as you said, the person with the technical skill who can actually go there and understand and challenge. And also sometimes, especially if you're working at a company which has a internal open source policy, meaning you can make changes to their code base. Like, yeah, just make the change. And then the team, it's incredible how technical, strong technical skill, with the ability of getting things done can really just move things. Have you observed similar things? Yeah, no, I've seen that too. I've seen that so often.

One thing I've seen is like the ability to kind of come up with a technical demo kind of moving mountains. Like one demo of like, hey, look at this thing happening, could cut through weeks of conversation about whether it's possible. And if you just go off and produce that, you can really kind of blow people out of the water. One thing I say a lot is that in the context, in the context of like a large company, which of necessity has many non-technical managers and

non-technical stakeholders, being technically strong if you work on software is really a superpower because it makes you connected to reality. It makes you connected to kind of the real world, to the actual product that you're building. And it's sort of like a lot of the work these other people are doing around strategy and marketing and figuring out what needs to be built. That only, like that's a huge multiplier on like a tech company. But it's a technology. But it it only kind of starts multiplying when it locks into somebody with the technical skill to actually make it happen. So if you're that person, just by like having a lot of technical surface area, you can like unlock the sort of multiplicative factor of like all these other people in your

organization just by being willing to answer technical questions, to make technical demos and to implement things quickly that are bothering them that they think are important. You can really move an organization just by being technically strong if you're willing to put that technical strength in the service of people who have thought hard about what needs doing but don't have the technical skill to do it. I see a lot of people who are so technically strong but are pouring that technical strength into making the code a little bit cleaner or writing tests that are a little better. That can be really useful, but I think you could be doing so much more. Can we zone in on to this specific thing? Because I hear what you're saying is if you are technically strong and you can surface in a way that is a lot more helpful to the business, visible to others. But what I'm seeing is a lot of most software engineers are technically strong.

And as you say, they might be pouring it into less visible things. What might be some ways that I can, you know, like surface this a little bit more? I think the most impactful piece of advice I can give is just to go out and talk to people. And that's a little spicy sometimes because going out and talking to people sort of circumvents some of the process that these large tech companies have in place by which communication goes through from project managers to designers to engineers. But if you're an engineer and you think you have a lot of horsepower and you're looking for a place to apply it and you feel like the backlog on your team

is not the most impactful place, sometimes the best thing you can do is to go and talk to your skip level, go and talk to your project manager, your product manager, your designer, and just say, what do you think needs doing? What do you think, like, I can work on? One thing that I do see engineers do this, and this is maybe not the most politically savvy way to use this superpower, but one way I had seen engineers use this superpower

is to go to support, to their support counterparts, who raise tickets, and to kind of invest that horsepower into making things better for support, fixing bugs that, like, waste time for support. This is not the kind of work that I think is, going to be super impactful on a promo packet. But it is worth doing.

And I think a lot of engineers who already do that for support could be pretty well served by taking that and also doing it for design and product instead of waiting for these design and product plans to percolate through to epics, to issues, to scoping. Because a lot of a time, there are quick wins that you can just knock out quickly. I like this. It might make your manager unhappy. be in it. It might not be a sensible move depending on the climate of your of your team to kind of go around the process like that. Yeah, I like this, but I do think that if you're unable to go to

customer support and fix a persistent issue, you know, like on the side at all, like over a year or two years, then like it is a good question of like how effective can you be. Like I've actually noticed that some of the most effective techniques did do, not on a regular basis, but every now, or they were able to do this. So, yeah, I guess if you're trapped and looking for inspiration, this could be one way to try and also just figure out, like, what the response is, right? If your leadership team is like, oh, you should have not done that. Like, well, do you really want to be working there when you're like fixing, doing something

delightful on top of the job that you have is taken like that? Probably not. Again, it depends on the external climate and all that. but it can be a I like the suggestions. I think it's worth trying to understand why if your leadership team comes back and says you should not be doing that. Because yeah, maybe they're wrong, but maybe they're right. Maybe there's some existential problem facing the company that you ought to be working on instead. A lot of a time, like engineers can spin their wheels.

It doesn't matter like a problem affecting the company can be as serious as it gets and there can still be engineers spinning their wheels because their team's doesn't empower them to act. I've seen that happen. Yeah, it's a good question on who's, who is it down to obviously as an engineer, especially when you're senior above, you should understand that. But also, if you're not as experienced in the management team or your manager is not efficient in telling you like, this is really important. The company is on the line or our product is on the line. Well, yeah, that's a little bit on them as well. Speaking of trust with your leadership team, as a tech lead, it's really important that you do have trust with your leadership team. And this sounds pretty vague, but it is important. What are good ways

you've seen of doing this? Yeah. So by far the best way to like maintain trust with your leadership team is to ship, is to deliver and is to deliver consistently. So if you're that engineer we talked about earlier who's like saving projects and derisking projects and making sure things get across the line, that builds trust. At minimum, it, it builds trust in leadership. that you're technically strong. And that goes a long way. That gets you put on higher profile projects. That gets you kind of called in when something's important

and kind of starts this like snowball. But in order to do that, in order to ship on things that like your management team cares about, I think you do have to like maintain the capacity to do that. So what I mean by that is if you're at 120% on your team's Jero tickets and you're just burning down and burning down and burning down, you're not going to have the capacity you need when something comes in that is more visible

and is more kind of higher profile. So I do think if you want to maintain this trust with your leadership team, one big part of that is to kind of keep your regular working cadence at closer to 80% or 60% and then kick it up to 120, 130 when something does come across. And that way you can kind of save your powder for when you really need it. I think that's really important. The other thing is to try and communicate like them. So I think a lot of engineers kind of get the sense that talking like senior management is this kind of slimy thing.

Is this like non-engineering value? You're not a hacker if you use words like circle back or synergize. And I think, yeah, maybe that's true. But if you want to build trust with your management, like you need to kind of play the game a little bit. And you need to like learn how they communicate. and sort of communicate back to them a little bit like that. Just to make that really concrete, if you're shipping a project,

the most important thing, almost like more important than technically delivering, is understanding what the project is for. So a project can be for box ticking for an audit. It can be for attracting new users. It can be for extracting more money from existing users. It can be for satisfying some VP or CEOs like dream of what a future product could be.

There's so many reasons. And all of those mean you have to work differently. You know, products for new users have to be really polished. Products for like large enterprise users don't have to be polished at all, but the requirements are usually really inflexible. Stuff like that. But you have to know what the project is for. But management won't tell you that unless you can communicate in a way that they know how to communicate back to. So if you go to your manager and you say, hey, can I build a really crappy UI for this that is terrible to use?

every single manager in the world, particularly in writing, is going to say, no, of course not. You have to build something really nice because they're afraid that you're going to go and build something awful and then point at them and say, hey, they told me to do it. So you really do have to pick up this crucial information. You have to be able to communicate this, like, is it okay to make tradeoffs with user experience instead of saying, is it okay to build a crappy UI? You have to be able to talk like that. It's a hard lesson for some engineers to learn, I think.

think, but it's really important. Yeah, this is very interesting because it triggers memories for me where I was the tech lead on a project, which it didn't make sense for us to build it. And I talk with my product manager and, you know, she confirmed like, no, it doesn't make sense. In fact, like, we're only doing this because there's an executive who struck some sort of deal with this company. We needed to integrate something.

And she said, like, look, like, it makes no business sense. tried to talk them out of it. We got details, but it came down to something political, honestly, that they were saying, oh, we have a deal with this company. We cannot not do it, et cetera. So it became a little bit of ego. And she said, like, look, here's what's going to happen. We need to build it. There's no way out of it. We need to do it as fast as we can. We're going to retire it in a year. We're going to have very, very few users of this thing relatively. So, and the PM wasn't telling me. She's like, these are the constraints. So, the tech lead, I kind of turned to the team and told them like, look, here's the deal. We need to build this. We can do it quickly. We can skip some of the test. It kind of needs to borderline work, but we don't need to pour much complexity. And this was a complex project. So in the end, actually, it helped us deprioritized a lot of the edge cases. It was like, don't care. Just add an error message. Don't do this. And in the end, actually, what happened is we shipped it. We had like a ridiculously low number of users as we predicted. And we retired it. 12 months later, the code was deleted.

On one end, it was this box-ticking exercise, which didn't make sense. But we had to do it. And the shorter we did it, the better it was. It was the first time I realized that you could do stuff that doesn't really make sense, but it was a necessity. And we could have argued a lot more or dragged our feet about it. But in the end, we're like, you know what, let's do it quickly. Shep it.

Thumbs up. I'm really impressed by the trust relationship that you had with your team and with your product manager that you could just be upfront about it and say, hey, we're doing this to check a box. We can trade off user experience, no problem. That sounds like it made things easier. This was a rarely, like, it was a really, like, close relationship with the product manager. We were like really kind of one unit.

It also held that we were in a different office location from HQ. I'm not sure if at HQ, it would have been said this clearly just because then people talk and you don't really, you know, I think it could have undermines, you know, like leadership morale when you say like basically the leadership like we don't agree with leadership and we're right to do so and actually the data showed that we're right to do so but either way so one thing that is changing software engineering a lot is a gen AI it's it's now in a day-to-day tool chain you're clearly you're working on some tools that I know you cannot talk about at GitHub but a lot of software engineers are using tools like GitHub co-pilot or some competitors to make their work easier

but I'm wondering, because you're working on both this tooling, you're using it, you're also seeing what developers use. What parts of the software engine job do you think are becoming easier with Gen AI tools, specifically with getting things done, right? Yeah. Well, one thing I said earlier was that the sort of cardinal virtue of shipping projects is, is speed is getting things done quickly because so many things can go wrong when you, you delay. And one thing Gen AI is really good at is speed. It's getting you like 80% of the way

and it's getting you like volume kind of for free. And one, like I'm a huge believer in like Gen AI for programming. I use it every day. I use it all the time. I use GitHub co-pilot. I chat to co-pilot in the editor like when I'm on personal projects. I use chat GPT. And it's just the level with which you can like go from zero to something is so good. I do think it's the kind of thing where I think engineers that tend to

work on one code base and tend to not work across a sort of broad spectrum can sort of underrate how powerful it is because it's not going to be better than you at writing like Ruby and Ruby on Rails in the code base that you've worked on for five years. You're going to be better

at it than that for sure. But it will be better at you probably at working on like GoLang in your company's load balance of repo that you haven't touched and are unfamiliar with. Like it'll be better at you like in all these domains where where you don't have experience. And because so much of shipping projects requires like understanding code and writing, usually small bits of code in repos that you're not familiar with, I think like Gen AI can kind of be this real difference maker there where it can,

you know, you need to build like a page in your company's internal admin tool in order to ship a feature. That probably doesn't have to be super polished. That probably doesn't have to be like 100% there. Gen AI can probably do most of that for you. That's the kind of thing where you can save a day or two. That kind of compounds over the course of a project. So we had Simon Wilson on the podcast who is software engineer who's using a lot of Gen AI. And one interesting thing he said that Gen AI makes him a lot more ambitious.

So is it fair to say that with Gen AI, if you have it at your workplace, which most companies will have at this point. As a software engine, you can just be more ambitious and do, you know, either sign up or start doing these products or just contribute to project which, you know, before these tooling, you might have been like, nah, too much time. I'm not going to be able to get my head around. It's a code base that I've never touched. I've never programmed this.

Let me just leave it alone. I personally haven't noticed this huge increase in ambition at work. just because like typically the hardest parts of like complicated software that has to be correct, the hardest part is that extra 20% on top of what AI can give you. So I would be reluctant, I think, to kind of swan into, to let's say the load balancer, like GitHub's load balancer and say, hey, you know, me and Copilot built this, built this feature that I think is really cool. Like, you know, like I wouldn't expect a super positive response.

where I have found like the chat GPT or chatchpT like gen AI lets me be more ambitious is personal projects for sure when I'm working on like stuff by myself or learning stuff by myself I almost feel like I can learn anything I almost feel like I can I can get into any area like so much quicker than before just because the ability to ask questions of of copilot of chat GPT, the ability to like just ask endless questions of someone that never gets tired of the questions that always gives you like considered answers. That's sort of the dream, right, when you're, learning something new. So I love it for that kind of work and I love it for like prototyping and spiking things out. And, you know, I sat down the other weekend and I was like,

what would it look like if you had five models playing Texas Hold'em against each other, who would win? And I, you know, under a day, I had that working and I, It turns out it's GPT4O, like, it's just, it's just not something I would have done. I wouldn't, you know, building a Texas Holden Poker, like, simulator to, like, back this, back this app. It's just, you know, I don't think I would have bothered doing it because it would have taken, you know, multiple days. It would have taken a lot of effort. But with LLMs, you know, I just kind of had it up in half an hour. It was great.

And this is a little different question, but related to this, do you see parts of the software engine that are getting actually harder as a result of more Gen AI being used, more people using Gen AI or just because of these tools themselves? Yeah, I mean, I'm in a great spot to talk about this as somebody who's worked on Copilot and is working on GitHub models. One part of software engineering that's getting harder is building these Gen AI tools, which is increasingly becoming a large part of software engineering. LLMs are really hard to make into working software.

The text, text modality is hard to work with. Hallucinations is hard to work with the fact that it's all moving so fast. It's a really fun area, but man, isn't more difficult than the work I was doing before, like on so many different levels. Like, even just the fact that many people you work with are kind of struggling to keep up as well. You just don't have this kind of baseline level of understanding that I did say at Zendesk when I was working on apps. We've had apps for years and everyone had been working on app stores for years. when knew what an app store was.

Now I'm at GitHub working on LLMs and people are kind of learning as we go. Like what are e-vowals? What are what are logits? What does it mean to put like a grammar enforcer in front of the sampler when you're like, you know, enforcing this JSON schema. How does that work? Like so many things, the learning code is just so steep. So that's definitely getting harder.

If you actually have to like build these tools, man. Yeah, that's harder than sanded kind of. engineering cradwork. Well, and I guess more and more of this over time, I would think it would leak over into software engine, the sense that these days, if you're a back-in engineer, you're kind of expected to understand how like the front end works at a conceptual level. You know, if need be, okay, like, you're probably not your job, but like every now and then you should be able to touch a little CSS or HTML. Same way. I would expect, you know, like the term full-stack engineer is exploding for good reason.

And I would assume that in a few years, software engineers understanding things like, you know, what is a rack pipeline or what does it mean to train a model versus to fine tune it? Again, you probably don't need to do it, but just knowing some of these and increasing more concepts will just be part of the data job, especially as more companies, you know, like incorporate LLMs into their different projects. Yeah, for sure. Assuming we still have rag pipelines, assuming we still have fine tuning. Like, I think some of those things are definitely still up in the air. And even two days ago, Open AI released a new way of fine-tuning models that isn't low-rank adaptation. So who knows where it's going to go. Exactly.

Well, it's an exciting space. Yeah, sure is. Another area that is very challenging and interesting is working across time zones. You're doing this. You're based in Australia. So I'm assuming you work with teams who are in vast different time zones. what is your experience working with teams who are in different time zones? What works well and what doesn't?

Yeah. Well, one thing I do want to say just on the topic of me being based in Australia is a lot of people got angry about my post about shipping and said that they're going to plant their feet and be really disagreeable and tell their manager to screw off and do whatever. I've got to say that the calculus on that looks a little different when you're in Australia. an engineer and you don't live in San Francisco and you don't have, you know, 20,000 jobs that you can walk to from your front door. So I think one thing about being in a different time zone is, is I think you sort of don't have this, there are some areas in which your experience is not like your US peers, which definitely affects how you work and maybe makes you a bit more prone to kind of do what the company needs as opposed to planning your feet. But in terms of

my direct experience working across time zones, as then, desk, I sort of worked with like teams in different time zones, but my team was kind of co-located, which was nice. And then I had a bit of a rude awakening at GitHub. I was hired as the first Australian engineer on my team. Yeah, just me, with with the goal to kind of grow this APAC group. But I think two weeks after I was hired, there was a hiring freeze, as there were many hiring phrases around that time. And so for for nine months, it was like five US engineers and a US manager and me in APEC doing doing my doing my APEC thing. And that was a that was a very interesting experience. It was a it was a very lonely experience in some ways. But but there were there were real

upsides that I think are kind of hard to anticipate if you haven't been in that position. One of them is that my mornings were really busy because everybody was closing up their day and I had to catch up on everything that had happened. But my afternoons were almost all clear. So I was in this position where like every afternoon I could do deep work. And I think that really set me up for success. I do think like I'm a pretty high agency engineer. And GitHub was lucky that it got somebody like me. who is happy maybe with a little bit less support.

I do think like being the solo engineer for nine months in a different time zone, like not for everybody. Yeah. So now having seen, you know, like both co-located teams working with different times, is also you working by yourself. When it comes to remote work, like talking to engineering manager, or people who are organizing this. Like, what is a strategy that you've actually seen work the best and most sufficient?

Am I incorrect to assume that is going to be the co-located teams where there's multiple people? Yeah, definitely don't just hire one person in a time zone if you can. Right now we have like a sort of a sort of squad of APEC time zone engineers and a squad of US engineers. And I think that works well. one thing I think that's really important for managers kind of trying to do this and I recommend all managers do this. I think every US company should open many hiring racks in Australia, please. That would make me very happy. But one thing that I think is a really good benefit of this approach and something we've done at GitHub the whole time I've been there

is to sort of double down on the ability to have people working follow the sun 24, 247. So many, many, many times when we've had critical projects or like critical bugs, we've had the US squad do a bunch of work and then hand off to the APAC squad, and then the APAC squad hands off to the US squad. And the result is that it sounds like a silver bullet, but you can basically 2X the speed when you're delivering some things, things that whether the handoff cost is pretty minimal, like bug investigations, that kind of thing. And we've really been able to deliver, I think, some projects,

that we just could not have delivered if we'd been restricted to working in the US time zone only users as well. When a user reports a bug in the afternoon and they go to sleep and they wake up and it's fixed, that feels really good. And that's harder to do with NA engineers. They kind of have to work really late for that to happen. But we sort of got that for free. Like anytime an important bug came in, just got given to the APAC engineers. And if we were able to solve it by the end of the day, it was just this really good user experience of like, wow, they literally literally fixed it overnight. I wake up and it works. That's amazing. I do think one way in which GitHub is really well set up for this kind of work is that it's very async first. And that's super

important for like managing these these two kind of time zones. Because you can't all be in meetings, you need to be producing kind of written artifacts and demos, these sort of things that can like speak for you and do work while you're asleep. So definitely like you need to kind of hire people who are comfortable both writing and reading documents and demos and that kind of thing. I think that reading part is important. I think it's comparatively easy to hire people who are happy to write demos and write RFCs. I think it's slightly harder to find people who are happy to read RFCs. But that's almost more important on these distributed teams.

People need to be communicating async for it to work. And you're now in a, I think of an increasingly enviable. position in the sense that you are working full remote at a at a team and a company that does support full remote and we're seeing data that the percentage of these uh job uh options is unfortunately just going down a bit who knows if if it'll stabilize or go up in the future but right now it is going down and and the the willingness of engineers wanting to do that is is not going down so i'd like to ask in your experience what are the practices and the traits that help you excel in a full remote environment and what what do you think you needed to uh to actually do this because

as you said you know you work for nine months by yourself and you mentioned that it's not for for everyone yeah i mean to sort of sound like i see some people talk on twitter these days i think part of it is that i'm built different um and i mean that in in like a maybe you know uh not necessarily entirely positive way but like some people i think need human interaction need to be like talking with people they work with, whether it's on Zoom or not, I think I can go a long time communicating with people, I think. That's just kind of how I'm put together, which makes it a really good fit for me, sort of first and foremost.

That's sort of how I've like, because I know a lot of good engineers who I think could excel in this role kind of flame out because they can't stand being by themselves. And that's just not something that's bothered me. So I've kind of had that obstacle, like, removed, which is really nice. Of course, then you still have to excel. And I think, you know, what I've focused on, what I've done, well, you know, we've talked about it.

You can go and read the blog post. But the one sentence is, you know, if you're in this enviable position and you want to keep it, you kind of have to align with what the company needs from you. And you have to take what your job is really seriously. I see a lot of engineers who, like, have their own goals for how they want to work. they want to have all their code perfectly clean or they want to always have unit tests or they want to work on like front end stuff only and they sort of push those goals up against the company's goals and that's fine you can you can do that and you can make those decisions but if you're on a

spot that you really want to keep like working fully remote for a company like GitHub that I love you know I have a lot of reason to try and kind of absorb the company's goals and and to try and make them my own. And I think that's broadly been successful and has kind of put me in this position where like I'm able to keep doing this and hopefully I can keep doing this for some time to come. I like this reflection, especially on the part of you can go a long time without communicating async. I guess a lot of this will just be learning. But is it safe to say that did you know what it would be like to work proper full remote before you did it or was it like just you know something that you needed to go through to see because as as i understand at zendesk it was more in office and together with the squad and this was the first time where it was a proper for remote job for you well recall recall the timing right like i i i moved to github in 2021 and uh Australia Melbourne had had one of the one of the longest one of the most extreme lockdowns uh in the world. So for a decent amount of time before I moved to Zendesk, I'd been working like co-located

in a time zone with people, but I'd been working from my house. I hadn't been working from the office. I hadn't seen these people. So I think that was pretty unpleasant in a lot of ways, but in hindsight, it did kind of give me the confidence that like I could do this remote role at a company like GitHub and that I could kind of, I sort of knew that I could work without going into the office. And before I wasn't so sure how I would cope with that. It turns out I cope really well. But I don't know if you can anticipate that ahead of time. You kind of just have to try it and see.

On the flip side, a lot of us who have worked throughout 2020, 2021, have likely had that experience. And I think people are going to reflect on like how did it feel? Like, could you do this for longer? Did you actually thrive? Because some people thrive. Some people hated it. Yeah, for sure.

I do think it was very, very, different being like in the same time zone as people from being alone in my time zone that that that was a big shift that was something that i i think i got i got lucky that that that i ended up being a good a good fit for that because had i known that was going to be the case i might have been a bit more skittish about about taking the role i probably still would have done it oh yeah well some things you just got out wing it and sounds like that's what you've did let's close up with some rapid questions so i'll just ask a question and just like bird out whatever uh it comes first to mind. What's an interesting framework or library you've recently used? And what did you like about

it? I mean, this is a silly answer, but Ruby on Rails, man, I love Ruby on Rails. It's not new. It's not sexy, but I am a huge Ruby on Rails believer. And it's just, it's the most programmer-friendly way to build large web apps. And its flaws are honestly endearing to me. Is GitHub still using Ruby on Rails, I heard that it was started on for Rupon Rails or it migrates for Rubeon Rails. So internally it's being used or this is just using it on the side. Oh, no. GitHub is still like in very large part, still a Ruby on Rails shop. We run a lot of different things for a lot of different services.

You know, sometimes Rubyon Rales doesn't make sense. But if you go to GitHub and you go through GitHub.com and you interact on GitHub, almost every single thing you do is going through Rubyon Rouse still. that's a good fun fact reminder yeah what is a book that you would recommend and why um well one tech book is actually underneath my mic right now it's the little book of deep learning by um by by by francois fleurray which which was sort of something i picked up when i was getting into lLMs and i wanted to have a bit of a better understanding of like the sort of maths that was that was backing them um and it was just a really nice

like short primer on how it works, didn't go into too much detail. It was excellent. But to recommend a book that I like, I read every year, I got a shout out, The Name of the Rose by Umberto Echow. I read it every year. It's an incredible book about an Italian monastery in the 1300s where somebody commits a murder and then they have to do an investigation. I think it's a very funny book, but it's a book about people doing their best in complicated systems, which kind of speaks to me and maybe some of the things we've talked about. So yeah, name of the rose by Unberto Echo. good read it. Well, thank you very much for sharing all these details and being on the podcast. No problem. It was a lot of fun. Thanks to Sean for the deep dive into the topic of shipping projects

as a tech lead with the focus on how to do this at mid-sized and above companies. If you'd like to read more about Sean's thoughts on software engineering, check out his website or connect with him on LinkedIn, both linked in the show notes below. If you've enjoyed this podcast, please subscribe on your favorite podcast platform and on YouTube. And if you're interested in more tips on how to lead the project as a software engineer, check out the deep eyes from the pragmatic engineer linked in the show notes below. Thanks and see you at the next one.