The Pragmatic Engineer - Stacked diffs and tooling at Meta with Tomas Reimers

Episode Date: April 2, 2025

Supported by Our Partners• Swarmia — The engineering intelligence platform for modern software organizations.• Sentry — Error and performance monitoring for developers.—Why did Meta build its own internal developer tooling instead of using industry-standard solutions like GitHub? Tomas Reimers, former Meta engineer and co-founder of Graphite, joins the show to talk about Meta's custom developer tools – many of which were years ahead of the industry.From Phababricator to Sandcastle and Butterflybot, Tomas shares examples of Meta’s internal tools that transformed developer productivity at the tech giant. Why did working with stacked diffs and using monorepos become best practices at Meta? How are these practices influencing the broader industry? Why are code reviews and testing looking to become even more critical as AI transforms how we write software? We answer these, and also discuss:• Meta's custom internal developer tools• Why more tech companies are transitioning from polyrepos to monorepos• A case for different engineering constraints within the same organization• How stacked diffs solve the code review bottleneck• Graphite’s origin story and pivot to their current product • Why code reviews will become a lot more important, the more we use AI coding tools• Tomas’s favorite engineering metric • And much more!—Timestamps(00:00) Intro(02:00) An introduction to Meta’s in-house tooling (05:07) How Meta’s integrated tools work and who built the tools(10:20) An overview of the rules engine, Herald (12:20) The stages of code ownership at Facebook and code ownership at Google and GitHub(14:39) Tomas’s approach to code ownership (16:15) A case for different constraints within different parts of an organization (18:42) The problem that stacked diffs solve for (25:01) How larger companies drive innovation, and who stacking diffs not for (30:25) Monorepos vs. polyrepos and why Facebook is transitioning to a monorepo(35:31) The advantages of monorepos and why GitHub does not support them (39:55) AI’s impact on software development (42:15) The problems that AI creates, and possible solutions(45:25) How testing might change and the testing AI coding tools are already capable of (48:15) How developer accountability might be a way to solve bugs and bad AI code(53:20) Why stacking hasn’t caught on and Graphite’s work (57:10) Graphite’s origin story (1:01:20) Engineering metrics that matter (1:06:07) Learnings from building a company for developers (1:08:41) Rapid fire round(1:12:41) Closing—The Pragmatic Engineer deepdives relevant for this episode:• Stacked Diffs (and why you should know about them)• Inside Meta’s engineering culture• Shipping to production• How Uber is measuring engineering productivity—See the transcript and other references from the episode at ⁠⁠https://newsletter.pragmaticengineer.com/podcast⁠⁠—Production and marketing by ⁠⁠⁠⁠⁠⁠⁠⁠https://penname.co/⁠⁠⁠⁠⁠⁠⁠⁠. For inquiries about sponsoring the podcast, email [e-mail]. Get full access to The Pragmatic Engineer at newsletter.pragmaticengineer.com/subscribe

What does actual development look like when you join? Fabricator is our internal code review tooling. Fabricator is well integrated with Sandcastle, which is our internal CI tooling, which is integrated with On Demand, which are our internal dev boxes, which is integrated with Landcastle, which is how code makes its way out to users. And sort of the thing goes from there. And then it doesn't just stop at code review. It's integration across the rest of the developer platform.

For example, Meta built its own task system. And so tasks and pull requests could be very deeply integrated in a way that we just don't see out here. One tool that I kind of experienced because of fabricator was called Harold. Harold, if I recall correctly, was the rules engine, which later got replaced by Butterfly Bot. And the idea of both of them was the same. It was be able to match events that might happen in the normal course of code review. I used to work on internal developer tools there. When we were deprecating APIs, you might go ahead and do something like create a Harold rule that

would say something like, if a user calls this function or uses this API, post a comment, letting them know, hey, this is deprecated. And also the new syntax is this. And then as you got later and later into the rollout, you could start to do things instead like, hey, notify me if anyone's adding callsites of this API so that I can go ahead and go leave a comment saying, hey, like, no, we're really trying to deprecate this. Meta is famous for his customer internal tooling. Devs that the company use the likes of Fabricator, Sandcastle or Butterfly bot.

But what are these tools? And why did Meta not use a more standard stack to build software? We get answers from Thomas Rainer, who was previously at Meta working on developer tooling before co-founding graphite, a developer productivity platform. Today we talk about what stackdifs are and how they became the defective way devs that meta work with pull requests. Why met a move to a mono repo and why there seems to be an industry wide trend in companies moving from polypropos to mono repos.

Why AI coding tools will make code reviews a lot more important and more. If you're interested in how effective engineering teams work these days and approaches trending in this group, this episode is for you. If you enjoy this show, please subscribe to the podcast and consider leaving a rating. This great to help the show. And with this, let's jump in. Thomas, it's great to have you on a podcast. Great to be here. So you, before you started a graphite, you worked at Meta, and Meta is known for a lot of their in-house tooling. Can we talk about, like, when you joined, what was interesting tools that you didn't see at other companies before? So funny. It's a great question, too. And yes,

certainly. It's when I joined, I was a new grad. So I had just graduated college. I joined Meta. I used a lot of the tooling. My frame of reference was personal projects. I used to do a lot of hackathons back in college and internships. And so I think when I learned, when I joined Meta, a lot of it came in sort of eyes wide, sort of like, I was about to say eyes wide open, but I mean the opposite of that. I came in sort of just like blissfully unaware that this was an industry standard. When I left was when I had the bigger shock. Because I think I had used tools like GitHub before. I'd used analytics tools before. I had left meta after having really gotten sort of immersed in all of this wonderful tooling. And then when I had to start to do all of these same activities myself, I found

myself reaching for tools that didn't exist and there weren't clear analogs for. Yeah, it's usually the other way around, right? Like usually people who work in industry at a startup or even at a larger company, go to meta. And unless you've been in like Google where you know it's all custom, is just a shock because I think most people imagine like, okay, you know, they must use GitHub or at least something similar. But what does actual development look like? You know, when you join, right? Like you went into boot camp, I assume. What did it look like to like, you know, like they didn't even push a poll request, but that's not what they call it, do they? No, no, they call it a diff. Good, good, good trivia for it, by the way.

I think that though, if I had to use one word to describe meta's internal tooling, it would be integrated. Everything, everything, everything integrates with each other. And I tell this to people and they never really understand the extent of it. Meta builds its own internal calendar, right? So meeting room scheduling can be integrated in with the rest of the tooling. And that really sort of permeates the whole thing, right? Fabricator is our internal code review tooling.

Fabricator is well integrated with Sandcastle, which is our internal CI tooling, which is integrated with on-demand, which are internal dev boxes, which is integrated with Landcastle, which is how code makes its way out to users, and sort of the thing goes from there. And then it doesn't just stop at code review. It's integration across the rest of sort of like the developer platform. For example, meta built its own task system.

And so tasks and pull requests could be very deeply integrated in a way that we just don't see out here. What was Sandcastle? So Sandcastle was the internal CI system. So out here, a lot of people, People use GitHub actions. They use BuildKite. And Sandcastle was just our flavor of that.

It allowed us to build versions of our apps and websites before we released and went ahead and tested them for us. And you mentioned this like integration. Like, you know, when you say it, it sounds like it's just a word. But like in practice, what did it mean? You know, you did mention that, for example, tasks, which is, I guess we can give it as the Jira or, you know, linear or whatever that is. that was integrated with the other stuff. And this was like, you know, like 10 years ago, right? Or so.

Actually. Yeah, I think integration here, I think one of the best examples I have of integration was actually a translation system. So at scale, internationalization becomes a real problem, right? You need to be able to include internationalization. As product developers start to update the website, create new pages, etc. There's always this question or this tension that happens of, well, how do we know that the string that we're pushing has a translator equivalent.

At Facebook, that could actually just be built into the code review tooling, right? And so the code review tooling can tell you something like, hey, this diff is using a string for which the translations don't exist. And because of that, you can't deploy it yet. Or what was more interesting to me was when you looked at a diff that had been merged, that had been landed, that was what we called it, you would see on that same page like, hey, this diff has been rolled out to employees. This diff has been rolled out to 1% of users.

This diff has been rolled out to 10% of users. Also, if this diff included any feature flags or sort of like AB tests, here are the results of those AB tests. And so you had this wonderful ecosystem where everything was tightly tied together. And so you as a developer could just go to one place and be like, I know what I'm looking for. I'm looking for this piece of like code that I landed. Let's go see like how it's changed statistics or how it's impacted users.

And you could see that all in one place. And then there was a bunch of other stuff that started to make its way into the public tool chain now of if you wanted to revert a diff, you could just go ahead and do that directly from Fabricator. This episode is brought to by Swarmia, the engineering intelligence platform for modern software organizations. Swarmia gives everyone in your organization the visibility and tools they need to get better and getting better.

Engineering leaders use Swarmia to balance the investment between different types of work, stay on top of cross-team initiatives, and automate the creation of cost capitalization reports. Enduring managers and team leads get access to a powerful combination of research-back engineering metrics and developer experience surveys to identify and eliminate process bottlenecks. Software engineers speed up their daily workflows with Swarmia's two-way slack notifications, working agreements, and team-focused insights. You can learn more about how some of the world's best software organizations, including Miro, Docker, and Webflow, use Swarmia to bit better software, faster,

at Swarmia.com slash pragmatic. That is S-W-A-R-M-I-A-D-com slash pragmatic. Fabricator was a really central part of this. I know because at Uber, we used Fabricator. It was an open-source tooling, and I think Uber, and you mentioned Dropbox also took it on, maybe a few other companies, but not as many. So for a while, we kind of used it.

I saw some of this integration, but it sounds a little wild because I don't really know any other company, maybe outside of Google, where you can have this thing, that, you know, as you say, you deploy a code and you can see the experiments, the localization, the rollout, all from one UI. Usually it's different systems. Even at Uber, it was different systems. We had, you know, like an experiment issue system.

It was called Morpheus at the time or Flipper, the feature flags. It was, you know, and we had, I had this list of like systems that I, you know, like a cheat sheet of where I would go to check the rollout, the internationalization, et cetera, et cetera. So all of this was really kind of like one click away? I think that's the way it exists for a lot of people today. You know, I think when we look of right now best in class dev tooling is starting to become more integrated. I think around like linear, it's now starting to post the linear issue in the GitHub poll request.

And you can see and it updates the status as the pull request moves through review. You're also seeing the same with Roussel, right? When you create a PR, Roussel can go ahead and be like, hey, went ahead and like made a preview environment for you, you can go ahead and click that there. I think what was so magical about Facebook is that all of that was native. And it didn't appear to be integrations, but just part of the system. And who built this tool, right? Like the developer tooling, you know, fabricator, the Sandcastle, all of these things.

Was it just like devs like, okay, I'm going to build this? Obviously it was internal teams, right? But was it like dedicated teams or people just kind of went and like went and tweaked stuff? I think it really depends on, so it really depends on the tool. Some internal dev tools were just built by developers themselves. They saw a problem. They're like, nothing's addressing this. I need to fix it.

Some came more from the company, isn't it, it's a wonderful culture to be a part of where you can solve your own problems. But some of it came more from the like, oh, we as a company have noticed that there's an issue here. We need to address the issue. Let's spin up a team to do it. And then the place where all of these tools ultimately found their home, was under the dev infrastructure team.

So developer infrastructure, other companies called developer platform and developer velocity, really maintain the bulk of Facebook's developer tooling. Yeah, and I guess one tool that I kind of experienced because a fabricator was called Harold. I'm not sure if you know. So it was the, of course. You could set up special rules on the code base, anything.

Like if basically on a commit, anything happens. like weird rules could go to good run. Were there like, can you, like, do you remember some, some, like, there were some really creative examples of like using Harold rules for developers to start to create their own rules or code ownership or that kind of stuff. Totally. So Harold was, Harold, if I recall correctly, was the Rules Engine, which later got replaced by Butterfly Bot.

Yeah. And the idea of both of them was the same. It was be able to match events that might happen in the normal course of code review. Something like someone puts up. up a PR that includes some line of code or call some function or CI fails regarding something. And then what it would do is it could take a variety of actions. So some of the examples that I remember was when I used to work on internal developer tools there, when we were deprecating APIs, you might go ahead and do something like create a

Harold's rule that would say something like if a user calls this function or uses this API, like post a comment letting them know, hey, this is deprecated. also the new syntax is this. And then as you got later and later into the rollout, you could start to do things instead like, hey, notify me if anyone's adding callsites of this API so that I can go ahead and leave a comment saying, hey, no, we're really trying to deprecate this.

Like, please stop. And then you could take it from there. Other examples were simpler. I remember there were some cases where there were, let's call it privileged or important parts of the code base. And what you'd say is like, if someone goes ahead and creates code that touches that part of the codebase, just make sure this team is added as a reviewer or at very least as a subscriber.

Subscribers were an idea and Fabricator of being able to CC people. It wasn't that you expected a review more just that you wanted to keep their eyes on it. And there were a handful of rules like that. And do I remember correctly that code ownership was super, super important for Fabricator and for Facebook? And it was just really trivial to say I own parts of the code and then set up different rules around that. No, so that's a great question. And so code ownership actually became – code ownership went through a few stages at Facebook. There was a while where code owners were required at Facebook.

And so you could have, like, different parts of the codebase could have different owners. Then the company took it away. And they said, actually, we don't believe that we should have code owners. We're a collaborative environment. We should go ahead and get rid of that. It was then reintroduced. And I think it was then again taken away with the idea of, like, we're not sure. The company alternated between, oh, they're really privileged parts of our monorepo.

And we can talk more around sort of how Facebook structured their code that need certain people to review it. And while actually we should trust our developers, developers need to be able to understand organizational practice. One thing that was always existing within the platform, though, was this idea of if you're in some part of the code base, tag the correct reviewers. And so I think one of the things that actually I most missed when I got out, here was the ability to structure repository in a way where you could say, oh, if you touch this part of it, we should tag these people in. I think in GitHub, code owners is a very coarse approximation of that where you have to define one file. It's at the root of the repository. It's difficult to use. A person who I've heard has done this even better company is Google. So in Google code reviewers were assigned hierarchically. And so in every folder you could basically say like, hey, if you touch code in this folder, like you need to go talk to these people, either as a heads up or as a requirement. And it allowed people to get, there were two really nice aspects of that. One was that it was really easy to define ownership.

And then the second was that it was easy as you were writing pull requests there to be able to go ahead and say, like, who does own this code? Who can I go ask for? And so if you're going to say that you're using an API that you weren't certain of, you need to figure out, like, who can I go talk to about this API? What you could do is you could just navigate the file tree, find the team or people in charge of it, and then just go DM them. And so that became a really powerful system there. Now, you've gone back and forward between, you know, at Facebook, there was code owners, no code owners. And you're now working at a startup. You actually work with a lot of startups and larger companies.

What is your kind of unfiltered view on code ownership? Is it a good thing? Is it a bad thing? Is it necessary only when you're bigger or when you're smaller? I'm sure you've got some opinions now. I work at a startup now at Graphite. We have the privilege of working with some of the largest DevTools teams in the world, which is really cool. At this point, I think I've heard all sides of the story.

I've heard the, we are in a financially regulated market, and because of that, we have a legal requirement to have code owners. And so I've heard that side of the world. I've also heard the side of the world of, oh, we couldn't possibly do code owners here. Like, it would just slow us down too much. And so I think it really depends on the nature of the business. I once saw this matrix, which I really liked, which is if you trust people a lot and are willing to tolerate, like, small mistakes, you should lean on culture, not process. As you start to get into a place of, oh, actually, maybe I trust the people less,

or I can't tolerate mistakes at all, you move to automation and enforcement. And so it really depends sort of where your business falls on that. I also will say that I think that as Facebook's grown, the way they treat different parts of their codebase has changed, whereas some parts of the code base absolutely cannot go down, right? Things that deal with privacy cannot go down. That's not a place where you can move fast. Some other things, perhaps, maybe the way that you display comments, if it's not a big deal, that actually is a little bit easier or a new product even.

There you have a little bit more leeway to move quickly, and so maybe code owners isn't the right match. Yeah, and one thing that I've kind of learned over time is maybe inside a company, as much as you'd like to think that there's one engineering culture, maybe that's not the answer. Maybe, you know, like a new startup inside a company, like an established company like meta, a new startup. Maybe it should operate differently because maybe they'll be shut down in six months if they don't find product market fit and they should just throw away the things. It's a little bit at the same time, I think companies don't really like to admit it. But, you know, like the and maybe even inside a small company, right? Like if you start just a small prototype project to see if it goes anywhere, you're probably not going to have the same rigid thing as when you're deploying to, you know, like customers who actually count on it. It needs to stay up.

I think that's a really wise point. I think that ultimately engineering anywhere exists within a system of constraints, and I think those constraints within a company can be very different. And so recognizing what are the constraints on the product I'm building right now, and architecting your systems around it is an engineering problem like any other, right? Sometimes you have constraints of, I think the place where this actually comes to me most clearly is when I deal with developer tools teams who release mobile apps versus web apps. So web apps tend to release on a hourly basis, maybe multiple times a day, but probably not

slower than that. Whereas mobile apps release on a weekly cadence. Because of that, just the shape of problems they deal with are so different even within a company. So mobile app teams generally don't have to hold pagers because even if something goes down, what are you going to do? Submit a fix to Apple, great, that's two days. Versus a web app team like paging is an important part of the culture, right?

As you're shipping things really quickly, things will. break. Things will go down and you need to, you're on the hook to fix them. And so watching the difference between those cultures, I think, is probably the like two extremes you see within a company. And then again, as you start to work in different parts, if you work on payment processing versus abuse and fraud detection versus like the consumer front end versus authentication, you actually have different constraints, which are subtly different. And I think that as you're saying, engineering organizations can develop different cultures around how they need to treat them. I like this thing of constraints and how, you know, at least different practices. Now, one thing

that really surprised me at Uber, which came really from Meta, from Fabricator, is we used this thing called Stackdiffs. And we did a deep dive into Pragmatic Engineer. It's the, it'll be linked in the notes below. But it's something that it just like really clicked for me and also everyone around me, everyone I talk with it at Meta. You've now actually built a like a business on top of this. But can we talk about what is stack diffs and you know how did you come across it and why why is it so you know like useful once you get a hang of it absolutely um so you're right we we've spent a lot of time talking to people about stacking about stack diffs and so stack PR says they're called on GitHub and i think that we've we've had a lot of practice about how to talk about it I think the

way i like to introduce stacking is by talking around the problem that it solves and the problem that it solves is in any given engineering organization, you have a bunch of engineers who at any given moment might tell you they're blocked. They're blocked on code review. And I think the, when you really sort of like start to ask, well, why are they blocked? Why is code review blocking them? What it comes down to is in most engineering organizations, the way development works is you have a main line, so you have a main branch or a master branch. You go ahead, you fork off of that. You go ahead and you create your feature branch. And then you kind of get stuck because before you can continue to develop on top of that feature branch, it needs to merge back into Maine.

Exactly. You need to merge it back because you only create feature branches off of Maine. The downside of that, though, is that your personal development process now becomes blocked on review because you've created some PR. You've created the perfect way to do, I don't know, authentication, login. And you now want to build forgot password. And so your options here are you can either add forgot password back into that branch, but suddenly that branch is going to become really big and your reviewer is not going to be able to

review it or if they review it, they're going to give it a rubber stamp looks good to me. Yeah, yeah. That's kind of, there's no point in a code review when you do that. Exactly. And so Google has a funny academic paper where they refer to it as reviewer frustration. As poll requests get longer, reviewers buy out of the process and end up just approving things rather than actually reading them. And so the solution, the solution which meta stumbled upon is what they called stacking.

Google, I've heard, has stumbled on something similar. There might be some cross-pollination with meta there. But the idea is you create your feature branch. You put it up for review. And rather than waiting for it to be reviewed for you to start building, you just branch off of that and keep going. And you create another feature branch. And then you put that up for review. And then while you wait for that to be review, you create another feature branch and you create that to be reviewed.

Now, people who have been using G8Nobruch, and then you create that to be reviewed. Now, people who have been using Git long enough, most of them have tried something like this. And I think inevitably you end up with a few questions. One is, well, what if someone rightly calls out that that earlier PR needs to be updated? The general answer is that's actually fine. Generally, when you need to update a PR, you're not changing the abstraction boundary between PRs. An example of how you might break it out is if you're building a new feature, I have server and then I have front end. And so even if someone's like, hey, can we refactor the server?

Usually the endpoint abstraction is actually staying in place. And so the next PR can continue to be stacked on top of it. The other question is, well, how do I know what I'm supposed to break up? And I think for that, generally, and this is actually really interesting to me, generally developers build their systems in order, right? You might start building the server and then building the front end. I think stacking is just a way to communicate that same order to your reviewers. A lot of people do this automatically using commits, right?

They just say like, oh, I'm going to build commits in the order I would build them, and you should review them by commits. Sacking goes a step further and says, well, if the first three commits in our branch are approved, why not merge those while you wait for the latter ones to be approved? And that has a few other benefits. You can run Sierra on them separately. You can reduce merge conflicts, and you tend to be able to reduce your time to merge, because if there are sticky things,

you get stuck there but can still merge a lot of other code. The other advantages then happen sort of afterwards of in the case that there's an error or something like that, you can go ahead quickly identify which PR was offending. And if you tell me a 10-line PR is offending as opposed to like a 2000-line PR, it's much easier for me to identify the issue. So basically you get to work in just like smaller increments.

It's almost like forking off of your current branch, you know, like smaller and have a bunch of small forks, except it's just a tooling, right? That like, so why do people not do this? You know, like just forget about like the tooling that might have. But you could just literally, you know, create a branch and then create yet another branch. But I guess it's just really complex to look at what's happening, complex to merge it back. I think Git makes it scary. And I think GitHub makes it scary.

So in Git, the most basic way to do this would be to create branch stuff of branches. But then if you need to update something, you need to update something, you need to go ahead and rebase it. Now that's where I just use a really scary word for a lot of people, rebase. Rebase. Oh, we hate rebasing. I mean, okay. I can't talk for anyone, but I'm going to admit, it always freep having the rebase conflict and figuring out what I need to do. Yeah. It scares me, and I've been doing this for years. I actually have a Stack Overflow post bookmark, which is the three types of rebase, because depending on how many arguments you give it, it behaves differently. And so super frightening. I think that's the place. I'm very fortunate that I learned

this at meta because I think what meta has and what I've learned Google has and what I've learned places like Stripe and Uber have built are internal tools to handle this for you, where you can basically say, like, I'm creating a stack now. I now need to update a branch in my stack, do that for me. And under the hood, it takes care of this intricate set of rebases that while you can do by hand are easy to mess up. Now, there's a lot of tooling that does. That, that, that does that client side portion. The other half of this is the pull request side, is if your host does not, if you're, wherever you have your pull request does not have support for stacking, I think stacks can look very foreign because what you end up looking with is a lot of PRs that appear

to be merging into, like, that appear to be recursively merging into different branches that they were all based off of. And so it can be hard to understand what's going on. And without any clear explanation or affordance in the UI, your reviewer might become confused too. So I'm kind of like wondering if you have a theory, why did, you know, like a few companies like Meta and Google invent this? And why is why did, why is it just not part of Git or GitHub? I'm sure you speculated about this. You must have. Certainly. So why I think some companies built this.

I think first and foremost, the current tools weren't solving their own needs. Current tools being Git and GitHub like and. Or whatever they had at the time, right? Mercury. It was Mercurial. Yep. I think they were shipping a high volume. I think they were really early on monorepos.

And I think that they were getting to this point where, okay, I'm blocked on review, progressively larger parts of this org are blocked on review. And as you sort of like start to grow up as a company, you realize like, oh, like if developers are spending 20% of their time waiting, that's actually a lot of inefficiency that's being created across the org, right? Most people that run an org of over 100 engineers would be like, I would love to have 20 more engineers. Second thing is that they saw the potential impact. So I think for a lot of people, they sort of like saw that lift of like, okay, Facebook, for example, really prioritize being able to improve their ship velocity, for whatever reason, it mattered

to them a lot. Google, I think, did as well of, well, we're a web company. If we can like continuously that we deploy, we can continue to improve things. And thirdly, I think they had near infinite resources. And this is the thing that I think most of us don't have, which is they saw the potential impact, they saw a problem, they had the resources, and so they could build a solution. I think it's much harder when you're a startup or even just a smaller scale company than one of the three most valuable companies in the world to be able to say, like, I'm going to take a few engineers and I'm going to build this. Yeah. And so they have a lot of time to support. printing money despite having, you know, thousands of engineers, even 10 to thousands.

So, so they, I mean, obviously they're going to like, like be rational. They're not just going to create a team out of thin air. But this is very interesting because what you mentioned is I remember I talked with Nicol, with Dr. with Nicole Forscan about developer productivity. And, you know, one of the things that came up when we discussed is like, well, you do want to like measure certain things just to get a sense. And one of them is typically code review. Like how or how long does it take from writing the code to get and get in there?

So I have a feeling that companies like meta, Google, and some of these bigger ones who did have the resources, they start to measure it earlier. Therefore, they decided let's build better tooling. And one interesting part that I've only learned recently. I was talking with one of the three Linux fellows, Greg, he's, you know, there's, and he told me something interesting that Git was created by the Linux community, well, specifically, you know, Star Wars, but it was to solve their issue. of Linux development where they had about three to four changes per, I think, per per hour or so. And it was an email baseless. So they created Git to solve their own problem. And now I think they have something like 40, 50 changes per day or per hour, something like that.

I think it's per day, actually. That's their rate. And they built it. It's all for them. They open source it. But it was for them. There's this Linux. There's this big.

It's also a monolith. the changes are pretty small for the most part. There's not as much, let's say, you know, like user-facing code, et cetera. But they sell for themselves. So I wonder if they just didn't really have the problem that Stactif does because they don't have, if you look at the average Linux change, it's a lot smaller than, let's say, you know, like a new feature being added to some of the co-bases at like Meadow or Uber.

I think you hit the nail on the head with it's really important to use tools that are built for your use case. I think that open source development generally, and Linux specifically, stacking makes no sense for. So people always ask me sort of like, who is stacking not for? My answer is open source. And the reasoning is that in open source, you're dealing with a lot of untrusted authors who come to you. They're like, hey, I have this feature. It's partially built. I don't have tests for it yet. I haven't fully tested it. Can I merge it? And it's funny in that in a company, if you're like, I have this feature, it's partially built. Maybe you even have some tests for it. But the feature is not totally there. You're like, I know you're going to be here tomorrow, right? And so,

cool, thank you for doing your job. I appreciate that you're contributing back to the codebase and that you're going to prevent me from having to resolve rebase conflicts later, merge complex. I think that in Linux, you kind of see the opposite, right? And if you've ever spent time on those email lists, you'll see people throw back and forth sort of like patches. And what will happen is that they'll say, like, oh, like, this isn't complete. You're not going to work on this edge case, right? And rather than saying, like, you know what, like, we can put this behind a feature flag and merge it. They're instead like, go take this back, fix it, fully finish it, wrap it in tests and documentation and everything else, and only then can you merge it. And that makes a ton of

sense in open source, right? Having worked on open source repos myself, the biggest fear is that the contributors is going to disappear. It happens more often than we like to admit they get busy, something happens. And so in that case, you don't want to land partial stacks. You want to make sure that the thing is fully complete before you even consider like reviewing it truthfully. Yeah. And this is actually like when I had the discussion, this trust came up so much with open source. So it's interesting how like, you know, as you said, the context, like inside a company where you have people, you can actually trust them that they'll follow up or if they don't follow up. It's not a big deal. You know, like it might be deleted. Interesting how how this context like makes such a big difference.

Now another thing that is unique, I guess, maybe. Maybe. Maybe. maybe not just to large companies, but it kind of started there, mono repos. Yeah. So what was, you know, Meta and Google will probably one of the two companies to start doing monorepoles. You've seen it at Meta. What was the history with the monorapole story there and the pain points you saw? Yeah, totally.

I think you have to remember, so first things first, you have to remember that source control at meta has a funny history. When that company started, and this is well before my time, rumor has. has it the way they did source control is emailing back and forth zip files. And so at some point, they started. Yeah. I mean, that wasn't so uncommon, right? Git itself is a fairly new innovation. It came around in 2007, 2008. And so I think that when Facebook started to do source control, they had this idea that, well, we want to be able to version everything together. And I think there are a lot of advantages to that. I think one is it makes collaboration a lot.

lot easier. It makes discovery a lot easier. I think it makes it easy to put dependencies between different services. So I think all of us have had that case where it's like, oh, well, I need to change this, but that also means I need to change the call sites. Or that also means I need to change this other thing. Monor repos allow you to do that, right? Because what's saying is there's one like unified state of the world that we can look at, can make sense and everything's good. We can test against even. Poly repos, the opposite of that, when an organization has many, many repositories, I think have different advantages. You can version them separately. You can have different practices within them. You can have entirely different owners within them.

I actually think that GitHub on many companies pushes a culture of polyrepos because it's what's open, it's what's common in open source. Right. So again, going back to this narrative of sort of like, oh, open source is different than closed source. I think in open source, you actually want things like I don't want to have to version at the same time that all of my dependencies are versioning on. That doesn't make sense. I'm not really collaborating with my dependencies. Yes, I depend on other open source projects, but perhaps I'm not contributing to them.

I think within a company, those constraints change, right? Within a company, you're looking at constraints of like, I want to collaborate with other people. I want to be able to version things. And so you get pushed towards sort of like one unified source of truth for like what is the state of our code base. Now, by the time I was there, Facebook wasn't even one mono repo.

It was multiple monorepos. I remember hearing someone referred to it, as a polylith, which made me laugh a lot, where there was the main web monorepo. There was Instagram, which was separate. There were the mobile apps, which were separate than that. There was internal tooling, yeah, which was completely separate. And they were on a mission to try and unify these into all of them, because into one massive mono repo, because the negatives of not having one monorepo were starting to affect the company.

Some examples of that might be that if you wanted to do an N10 test on mobile apps, I remember there was some tests that would pull down a copy of the, other repository, spin up a server from that repository, and then spin up the mobile app such that they could test against that. And that led to all sorts of conflicts, right? It makes your CI flakier. It makes it hard diversion against. When we had dependencies between repositories, that was a nightmare. And so by bringing everything under one roof, it allowed you to really collaborate across the company more actively. It allowed you to collaborate across the company more actively and allowed you to share dependencies.

So Facebook was being slowed down despite having, you know, a few mega repos, if you will, or these large ones. So they saw this advantage. In the end, they moved to one unified monover between iOS Android and even web? When I was leaving, they were on the journey to. Wow. I remember hearing recently that they're still on the journey to. That's really, it's such a different call, especially for native mobile.

But yeah. But the advantage of it, though, and again, going back to it, is I think that it allows you to enforce a common set of Eng practices and requirements, right? Like the idea of like, oh, every repository should have at least one reviewer, right? In a company, yeah, that probably is a company-wide requirement. But in open source, like, I can't be guaranteed that every open source project wants the same set of requirements. And it's interesting because it's not just met up, like other companies when they grow up. certain size, they start to start to have the same journey.

You know, I've seen it at Uber as far as I understand, you know, Shopify has, is, is doing similar things. I assume that as companies go larger, they'll have these typically starts with like, oh, what about a vulnerability of a library being updated? Would it not be nice if it was in one place? Now, you've, you've seen more of these through, obviously through with graphite when you're interrogating with some of these companies. Do you see some trends of like when monorapels are starting to be?

become important and if they are starting to become more important, why do you think we're not seeing GitHub support too much of this still? It's a great question. So first and foremost, I think that we're watching an industry-wide move towards monorepos. Most companies we're talking to are already in a monorepo or are migrating to a monor repo. At least for the people that we talk within Silicon Valley, a lot of people have decided, you know, this is actually the future. I see the advantage, the tooling's there.

Someone should be moving here. And so we're actually seeing a broader sort of like trend towards monorepos. Why do I not think GitHub supports it? I think it doesn't make sense for open source. I think we're just starting to see the like advent of monorepos in open source. I think React is closer to a monorepo than not. I think that Versel follows the same pattern, probably a lot of the same energy there. I think we're seeing a lot of repos say like, well, maybe multiple packages should be in one repo.

But there are a lot of advantages to being able to put them all together. But I still think it's not the norm. When you look at like the JavaScript ecosystem or the Rust ecosystem or the Python ecosystem, we still live in a world of many separate packages from many different authors with many different cultures. And the way that you want to manage those is through poly repos. Yeah. And then when you say, you know, you see a lot of companies having, can you just get a sense of what types of companies in terms of the size, the funding stage? You know, like I appreciate you might not be able to say specific names here.

do not advertise it, but it was pretty surprising for me to say that a lot of the companies you work with. A lot of them are moving. No, I'd say that graphite tends to work with with Silicon Valley-based companies that are somewhere between, call it like, 100 and high thousands of people, maybe 10,000 people. And so for us, I think it's a lot of the forward thinkers who are really investing in developer velocity, as you were talking about earlier, that are starting to invest in the like, how do I speed up my team, are very quickly realizing that the coordination overhead of polyrepos makes it harder to speed up your team.

It both slows your team down in code review in weird insidious and second order effect ways. And it makes it hard to enforce a common culture. So when we talk to DevTools teams and they're like, oh, we want to migrate CI. When you're in a mono repo, that is a somewhat straightforward operation. When you're in a poly repo, that is a multi-year journey that you're on. Oh, yeah.

And yeah, I think I can emphasize what Yeah, and I think what you said, like, it needs to be some investment. Like, it doesn't just happen like this. I mean, there's better tools now, but you need to have like dedicated people looking at vendors who are doing it, et cetera. So probably makes sense that the teams who are investing in better tooling are probably the ones who are looking at these alternatives. And there's probably some selection effect in who we talk to. Yeah. Right.

When teams want to invest in tooling, they start talking to us. And at that point, they've already started to consider the Mono repo journey. Well, I mean, you don't want to tell me because I've seen the benefits of mono repo. I mean, I see the costs as well, but it was so good at a grouper, so many benefits. And like, I would, you know, if I had a magic wand, I would say, let's do mono repo. Obviously, there's a cost. So I see this. I'm just surprised that it's kind of an encouraging way that there's so much of this happening.

I think another place where things got confused, let's call it, is that people associate microservices with many refos. Yes. And I think, again, that's because that's a way that I think GitHub very much encourages you to think of like, oh, you have abstractions between code. You should have different repos. And I think that I have a friend also from Facebook who likes to joke that the discovery of monorepos is really just as a discovery of folder structure. Where it's like, well, yeah, so you have multiple different pieces of code, but you can just put them in different folders. That's okay. I think probably the one of the best examples, Uber is known for advertising how they have 5,000 plus.

microservices and they do have mono repos. I think per language, like four or five, but yes, you can, it's not dependent. It's totally independent from one another. Exactly right. Now, one other trend that's happening now everywhere, especially with the forward thinking companies, is AI tools for code generation or for coding. How do you see this changing how code is written, both through the companies that you work at or maybe even at graphite? It's incredible. It's been a whole conversation at graphite. and it's definitely been a whole conversation with a lot of the companies we work with.

I think the most immediate impact of how will AI impact software engineering is there will be more software written, right? There is going to be more code written as the tools make it easier and easier for developers to ship code. I think what's not as obvious as the second order effect, right?

If you have more code written, well, what needs to happen? Well, more code needs to be reviewed. And if more codes reviewed, more code needs to be tested, and more code needs to be merged and more code needs to be deployed. And I think that's suddenly the place where we're seeing people run into issues where they say, okay, I think that we're

ready to adopt one of these AI tools that helps us generate code, that helps our developers code faster. Let's do that. And then what immediately happens is we see the volume of pull requests go up. We see the size of those pull requests go up. We see the amount of bugs or CI bills also go up. And I think getting ready for that transformation is a thing that the industry is just starting to grapple with.

This episode was brought to you by Century. Buggy lines of code and long API calls are impossible to debug and random app crashes are things no software engineer is a fan of. This is why over 4 million developers use Sentry to fix errors in crashes and solve hidden or tricky performance issues. Centricos debugging time in half, no more soul-crushing lock sifting or vague user reports like, it broke, fix it.

Get the context you need to know what happened, when it happened and the impact, down to the device, browser and even a replayable. what the user did before the error. Century will alert the right div on your team with the exact broken line of code so they can push a fix fast, or let AutoFix handle the repetitive fixes so your team can focus on the real problems. Central Help Monday.com reduce their errors by 60% and spend up time to resolution for next door by 45 minutes per dev per issue.

Get your whole team on Century and seconds by heading to century.io slash pragmatic. That is S-E-N-T-R-Y-O-O-S-E-E-O. slash pragmatic or use the code pragmatic on sign up for three months on the team plan and 50,000 errors per month for free. Now, you know, this is speculative, but it's good to speculate sometimes. How do you think successful companies will cope with it? And this is, we can just speculate, right? Like, we know there's going to be more code being churned out. Devs will be able to generate stuff faster, build features faster. And yes, it will have all these. A lot of them will just tap, tab except they're going to not notice certain things being there.

We'll even have like attacks of like, you know, some companies trying to have malicious, a bit like in a stack overflow, copy paste, and you have the memory leak. We'll have more of those. What, you know, let's just do the thought experiment. Like how, how will good teams deal with this? I think AI necessarily is both the problem and the solution. So I agree with, I think that the fastest moving teams are going to adopt AI. I think we move past the world where you can say like,

this is a fad, this will pass. AI is going to mean a lasting change on the software development industry, period. I think that you're right. It's going to create some problems, right? You have developers just clicking tab nonstop, probably importing code that they neither right. It's called vibe coding. Yeah.

I love the Twitter memes about it. But yeah, vibe coding is a thing, right? You have developers at your company who are probably just talking to their computer telling it like, yeah, create this website. No, I'm not going to read that code. And they're pointing up for code review. And I think that, to me, that just emphasizes the point of code review, because if the author didn't read the code, then someone else certainly should.

And the question is, how are you going to make sure that happens? And so I have two answers for you. One is, I think a lot of the practices that already existed at these major companies, at the Google and Facebook, and to a lesser extent, but still in extent, Uber, Twitter, Snapchat, which are used to seeing a high volume of changes. written by a diverse employee pool, a lot of those practices are going to percolate down. Because those used to be the problems of like, well, you only run into this issue if you have 100 developers. We see it on Twitter. People are saying teams of 10 can start acting like teams

of 100 developers. And so that also comes with the problems of hundreds developers. The second solution is I think that AI is going to transform the way that we view code review. I think the largest change which has happened, something which we are trying to, trying to keep up with internally is that I think we're seeing the same advent that happened when something like grammarly or the grammar checker in word happened where AI can actually check a lot of the minutia or mechanics of code review. It can tell you that like yes, this code works or yes, this code does what it intended to do. The question though of code review then isn't, is this code literally bug free? It's is this aligned with what we wanted it to do, right? Is this, I originally,

we specced out the system. It was supposed to work this way. Or maybe we have built the system and it does work this way. What are the other effects of that? How does this interplay with everything else we do at this company? What's going to happen there? And I think that we're going to see AI tools built. We're building one ourselves or AI reviewer that let developers automatically check this code for sort of like the simple stuff allowing the reviewers to focus on the more complicated stuff. Do you think testing will become more important in this world where we do have a lot more AI tools,

more AI code, a lot more, you know, pushing to production. I think testing is going to change. And so I, is it going to, is it going to, is it going to become more important? Probably. In some form, definitely. Because as I said, I think that if developers are not putting thought into every keystroke, but instead we're just seeing like this code written quickly, checking that code for correctness, both the like literal, like, does the function work as intended and the end-to-end correctness of like are the buttons in the right place and is it usable or critical. Now, I think what's really interesting or the trend that we're seeing is that it appears that AI can actually write tests for you in some limited cases.

And when we look at some of the more advanced AI agents, they actually do this as part of their coding, right? If you ask an agent to like generate a change and also test it, it'll do a pretty good job. And I think that a lot of the previous types of tests that before were restricted to QA people or to humans of like look at this website and try to like click around it with the mouse where before we had to use, we had to use an antenna testing tool. It was hard. We'd kind of be like, oh, go click at like X, Y location this. You can now give to like computer. We even hired, you know, like testers who we call the sanity testing. You describe what to do. Go there. Take a credit card. And like that was because I was on payments team.

Like we actually had some of the fake credit cards that were take take one of them. And we did it because we couldn't really automate it for different reasons. The payments is a bit tricky. But yeah, it was. It was exactly the same. Terrible job to do. But someone had to do it. And someone was being paid.

Exactly. And it's, I think that again, the machines can start to do a lot of this work. I think the question that we have to grapple. with then is so what is the purpose of this review step right why and i think that for a lot of people this is a question we used to talk about like three years ago and i think with time with advantage of time we've become we've had a lot more clarity of like the purpose of code review isn't just the mechanical it's not just hey can you proof read my essay for me because i want to make sure they're no grammar mistakes met i think there's also a type of shared learning that happened of hey i want other people to

code review this because I want to distribute my knowledge of how the system works. And I think that there is an alignment checking that happens of, hey, this is how we build software here and there's tribal knowledge that may be more senior staff level engineers at your company have, that they disseminate through the process of code review. And I think that as we take away these mechanical parts to reduce their need, you're going to see those latter two become a lot more emphasized. I am a little worried, though, about there is a big, we do see that AI-12. tools are very capable of generating code.

And I think, you know, like experience developers who've been in the industry for 10 plus years and I've seen, for example, they work with the teams of junior engineers and you see what happens. You know, it's a bit of a mess and you need to really clean it up. Like they, I think, you know, they see that this has limitations, right? Like, it's a very powerful tool when you know what good output is, when you can check it, when you can step in and say this is BS stop or or you just like stop prompting after a while because it's not going anywhere and you start take over and you just write whatever you need. to do. But I do feel that people overindex on how good it generates code. And I have a feeling that a lot of companies are, especially, you know, like company leaders, even CTOs, they'll kind of ignore this part of learning because it's invisible, right? Like you, you don't really see the difference between like two teams. One, there's no code review or just AI code review. Everyone's doing their own thing and, you know,

and they're both shipping for a while. And at some point, something breaks. And this team where people are actually doing code reviews, they're thoughtful, they understand the code. they can start, you know, nothing stoles down. They build the new features they innovate, whereas this one kind of gets stuck. There's just bugs. No one knows why. But because it's not visible, like I'm,

I have this feeling that there's, there might be. I don't want to like, you know, be all negative about it. But it's just so many steps removed from, from the work that I wonder if like few companies will appreciate it. A bit like how few companies used to appreciate developer tools. You know, like it's only fashionable these days, right?

Like, like, if you. think about a 10 or 15 years ago, like everyone was looking like, why is Google having a dedicated team to build internal dev tools? That's weird. I think it became popular with, I'll give a lot of credit to Heroku. I think I remember just like growing up in that era, there was, there was a lot of love for like, oh, this is what good can look like, you know? And I think it really set a standard of like, oh, we don't have to accept worse. We can actually, we can have better tooling sort of like in the popular opinion. And then a lot of

Heroku developers went on to go to companies, brought a lot of that energy and feeling with them and then we started to see it there. But yeah, I remember when Google,

there was a lot of sort of like question marks around like, what is Google doing? Like, why would you invest so much in this in meta too? I think that where this will be different with AI is the fact that

at the end of the day, code has really profound business decisions. Yeah, like if you, if you decide that you want to give a refund or you want to charge this or the user experience wants to look like this or this is the way the product works, that's at the end of the day business decision. And I think that what leaders will agree with is that they don't want the machines making business decisions. And so at that point, at some level, that means that a human needs to be reviewing it. And now the question is sort of like, what is that level and what is that granularity that we review on? But I think that we can we can find a common understanding and like, okay, humans will need to at least check intent, the same way humans prompt. And that's not to say that like prompting is bad or it doesn't work. It's more of that just, I think fundamentally, like human natural language communication

is flawed, right? Like you and I have a conversation and you can understand something different than what I mean. And so at a company, the way we get around this is we then check on the other side, right? We align at the start and we check at the end. And I don't think that's going to go away. Yeah, I like this thinking. And also, I wonder if we might have more companies discover. Again, I'm just a little bit inspired by Linux as I recently talked with Greg from there. The way Linux works, you know, when I talked about how the whole project works, I was really surprised on how much it's about people. And the core concept there is when you take a patch, they call a PR a patch there,

you now own it and you are personally responsible for it. And this actually, you know, if something goes, and this is how there's a pyramid of like subsystem owners, system owners. And then in the end there's Linux Torval. So every single release for Linux, It goes through, still to the day, it goes through him. He owns it. And, you know, he's very happy to take that responsibility a bit.

And, you know, Greg was telling me that because of this, like, if you think about it, and it's very open, the whole world sees your changes. So because of this, there, you know, people take pride in this. They will both review, but there's also trust levels, right? So if you're a system owner and there's a subsystem owner, if you work now for 10 years, you know this person is solid, you're not going to spend all the energy. And I wonder if, you know, the, you know, the, the correct response to AI is like, yeah, just go wild, use these tools.

But responsibility, maybe it needs to go back to developers. Like, you know, like, so we'll see. I suspect that is what we'll see. And as I said, that's not to say that AI is not going to solve your problems. I think AI will actually free the humans to focus more on those bigger picture things. I think AI is going to be able to take away the like, oh, does this function say it, do what it intends to do? Because they can understand that.

It can tell us that. but I think it's going to be a tool in then saying, and so does this holistic change do what you wanted to do? So speaking of change, you know, one thing that I was really surprised that when I worked at Uber, I talked with other companies, and stacking didn't really take off anywhere else because Fabricator was just not there. Why do you think that didn't take off?

And how did you get the idea to start graphite around this idea of, you know, building this tooling to make it accessible for a lot more people? Yeah, of course. Well, so I'll start by saying I actually don't think that's stacking caught on inside meta at the start. Oh. So originally when it was built, it was built into Mercurial. There were, it was advertised at the company.

There were a few senior engineers who saw the vision, felt blocked, and picked it up. And I think that not in an exaggerating way, they saw themselves get 3x faster. And what happened was those senior engineers started to become evangelist. I had one on my team who was like the stacking evangelist. And they used to go throughout the organization and literally give PowerPoint presentations of like, this is how stacking speeds you go. Like you need to believe me. I've seen the other side. And at the time, I think you had a lot of developers who were very skeptical who were like, I've been doing it for so long. Like I don't need you to tell me how to change my work. Well, why would I do that? I hear just all the sign from current people who hear about stacking. Yeah. And then like what ended up happening was that as these evangelists like made inroads and started to.

convince people like, no, no, no, it's actually speeding you up. People would adopt it, see the value and become that evangelist themselves. We like to joke that once you stack, you don't go back. I think why it works so well is it did two things really. It was like the perfect convergence of two forces. One is that it gave individuals what they want. So I think no developer likes being blocked. If you ask a developer, like, what is the feeling towards code review or towards having to wait on code review? It's almost always negative. No one's like, I love that feeling. People are like, it sucks.

It's the worst part. And I think that even if you have a more senior developer that's been doing, that's had code review for 20 years, if you ask them like, why do you enjoy coding on personal projects as opposed to company projects? The overhead of collaboration becomes quite clear there of like, yeah, because when I work with others, things just slow down. We need to review things.

We need to align on them. They get a lot slower. And I think what stacking did is it took away some of that overhead, right? Where rather than having to wait on other people to review your code that you already thought was pretty good and you had already aligned on before, you could just keep going and keep building without needing to sort of wait for them. The second side is on, I guess it made it was great for three parties. The second side was the reviewer. The reviewer started to get smaller PRs. Yeah.

Right. And whether you know it or not, when I send you a 10-line BR, you are so much happier than if I send you a 2,000. Yeah, but so many companies eventually, when they write down code review guidelines, they're always going to have small peers. You know, Uber at some point we had like recommendations within the 50 or 70 lines, whatever that is, but you realize, I mean, there's jokes going around and I think it was viral on social media of the, you know, the 5,000 line review looks good to me and then the 10 line, which gets like 10 comments.

Exactly right. And I think people, as we were saying before, people buy out of it. And the third is, I think it really gave the company something they needed, which is, that they started to see their engineering organization speed up. They started to see time to merge go down. That's usually a metric that people measure, and we can talk about engineering metrics separately. And in the case that something went wrong,

it made it much easier to correct things, right? In the case that QA or even users, like, found something that wasn't working the way it should be, it was much easier to then go to history and be like, which one of these changes did it, than when the changes were like monolithic feature, monolithic feature, monolithic feature. Yeah, now you know, you build this tooling,

and you are still a pretty small startup. How did you convince companies to use a pretty small startup, pretty unknown methodology, you know, rumors are meta uses it a lot. Because I assume it's not a really easy sell, is it? No, it's not an easy sell at all. Well, so back to the company's origins, we didn't, so I don't know if you know this. We didn't actually start as a code review company. So we actually started by building tools for mobile releases.

That's the background of the founders. That's the area we wanted to like build it. I was fortunate enough to have two of my teammates from Mata join the team. And I think one of the things that the three of us felt was like, oh, like the tooling out here, I think they were both new grads when they had joined Mata, had been there for a handful of years, call it like four. And then when they left, they had the same experience I did, which was like, oh, like, the Tullians out here is different.

And at the start, we kind of were like, well, you know, we came from Mata, we must be the weird one. And after about like a few months, we're like, ah, no, like there's actually something missing here. And I think the context that I now have that I wish I had then was, one, most companies, if you have more than 50 people, you have at least one person who builds tooling on top of GitHub. And I think that if you've worked at one of these companies, you kind of just take that as a truth of like, yeah, I have the DevTools team, or on the other Jenkins. But these days, Jenkins is GitHub action.

So same difference. Exactly. You have someone building tooling there to make your developers faster. The second thing is that these companies, So like meta and Google, so GitHub released their Pell request page back in 2013, they haven't really made a ton of major edits to it since. Those companies had their changes then and have continued to iterate on their tooling for like the past decade. And so when we say their tooling is like a decade more advanced, that's a quite literal statement, actually. That's not a figurative one.

And so we realized that. We were like, hold on. Like there's missing tooling out here. we tried a lot of the alternatives. As I said, stacking was sort of that first paid point, and so we tried a lot of the sort of like open source, CLI first alternatives. Realized it

it was like an approximation of what we wanted, but it wasn't quite it. And then we built it. And then what happened actually was a lot of our old coworkers, many of whom had gone on to startups, ended up asking us like, hey, like, code review out here is different. Like, how do you all deal with it?

And we're like, we built that. Let us know if you ever want us to tell us, you how to build it and they're like, I don't, but like, can I just use your thing? And after we heard that for a few months, we ended up pivoting back in, I guess this would have been November of 2021. Okay. So quick pivot in the beginning of the, just, so did you just feel a poll? Like, okay, like this is a good idea. Seems like there's a man for it. I don't think it was the other company idea was working quite well. I think what we felt was we had a all of our friends asking us for this thing that we had built.

And it wasn't the thing that we were selling. And you can only listen to that for so long before you decide to say, maybe we got this one wrong. Maybe this is the thing. It's just fascinating. I guess it's hard to appreciate if you're not working at a startup. Because, you know, my imagination of joining a startup, I have friends that my brother was working at a startup.

But I didn't join a super early stage startup ever is you would imagine you're joining. This is what we're doing. This is what we're going to do. but, you know, sounds like you need to be flexible enough because you might do something different that actually works and there's a bigger demand for it. Yeah, we were fortunate enough that at the time that we pivoted, we were just six people. And so it was six people and three of us had felt the pain from Facebook. So it was us telling people like, no, no, no, we've heard it.

Like our coworkers have heard it. We'd gotten other people at the company to buy into this workflow at that point. And, yeah, it was certainly stressful, but it was also a thing we'd, just had enough conviction. We're like, this is the way the world will build software. It's more a question of who's going to bring it into the world, not if. So you previously mentioned engineering metrics. What are engineering metrics you see these fast-moving companies measure and actually care about and want to improve? So I'll start with a statement that all engineering metrics are a proxy, right? So I think that measuring engineering velocity is an art, not a

science. And I think that the value of engineering metrics is to create some holistic metrics so that you can see major outliers and be like, what's going on here. The kinds of metrics, which I see most people measure, are number of poll requests as just a proxy for sort of like how is velocity doing, time to merge of those pull requests. And then my favorite metric I've actually ever seen came out of Uber. And so maybe you might have more context on this than I, But it was basically time that a poll request is just spent waiting in review with no clear action. And so the idea was like if an author puts up a poor request and it's just waiting on review, you take that, then the reviewer goes ahead, request changes.

And like it's that you then ignore that time. You then put it back up for review. You start summing that time again. And you end up with this composite of like, this is the amount of time this poll request just sat waiting on review without clear next steps. So this metric was introduced after I left Uber. I think I might have covered it in one of the deep ties on how Uber did it. But this was a really big problem at Uber because we were distributed sites.

And like my team specifically had this problem where we would have to wait almost 24 hours for something to happen on the poor request. Because, you know, we had the we had strong code ownership. And I think Uber still might have it to this day. I'm not when I was there, we still had it. So the mobile platform team would own the platform code in San Francisco. Francisco and Amsterdam, you know, at last thing at night, you would make a change and you would have to head off and it would go to them. They would reject it because of whatever reason you come back in the morning. It's now nighttime in San Francisco. You fix it quickly. And then

you have to wait another like eight hours and then stay late. So like I'm glad that they started to measure it because and this is one of the things where it's not really a big deal if you're a startup, if you're in the same time zone, if you're in the same location. But, But and then, you know, we came up with creative workarounds. Like we started to get like we onboarded someone to be an owner of the, essentially, you can do a lot of things. But it was really killing us. Like, and I knew it because I was the manager of the team.

And this was the number one complaint. So we actually, you know, we took it into our hands and we solved it. But so many teams were doing this independently. And then it kind of goes back to like, you know, like as a dev, I didn't really like to hear too much about strategy, whatnot. But there is a thing of like, you know, who owns what, where how do you place your teams? It's kind of easier when you have teams working on similar stuff.

close to each other in terms of time zone. So, yeah. And, you know, one other thing that you were measured, which I'm not sure how successful it was, but it was an interesting thing is they tried to measure focus time on how much uninterrupted focus blogs deaths had. I think it's really noble. It's just really hard to measure. But it's true that when, like, I was always happier, like, developing what I actually

had time. It wasn't fragmented with, like, you know, the random meetings here, here or there, et cetera. Yeah, totally. I mean, makers schedule. manager schedule is super real. I think that as we were going back to you before, though, these problems which used to be more isolated to big companies, I think as we start to see a higher volume of full requests and more of the time is spent sort of like in code review, not authoring pull requests, they're going to become everyone's problems. And I think the question

is sort of like how ready are companies for that change. Yeah. I like this, you know, thinking, because this will happen. So it's interesting how small companies, might need to take some of the lessons from the larger companies on how they solve it and prepare for it because it's going to come like I'm kind of tempted to agree with you on this one it'll be fascinating to see it'll be also fascinating to see if small companies speed up a lot more than let's say big companies where there's larger teams and and there's now more dependencies it'll be interesting one for us to I think small companies are certainly better position to embrace the change where I think big companies will be slow I think small companies we've already seen them adopt a lot of AI tools faster I think some of the most advanced big companies are just trying to adopt them at the same pace of these small companies, but they're definitely not trying to outperform them. And I think we've seen this actually in quite a few areas, right? Like every time that you have a new technology that makes something more accessible easier, the same way I think AI is making coding more accessible easier, I think what ends up happening is that like practices that were previously only meant

for like the tippity top of the market start to distill their way down to everyone. So you're a co-founder of a DevTools company. And I think this is a little bit of a dream of a lot of engineers. I actually see this pattern pretty much is you join a company. You work on a product. If it's a big enough company, there's a platform team. A lot of engineers gravitate to there because you can solve pretty cool problems. In your case, you actually did the, you were on a DevTools team.

And then you find the courage or find the right people to actually start a startup. And what better thing to do as a dev and what, you, you know, you already know how to do this is built for other developers. What is something that you've, you've kind of learned the hard way of starting a DevTools company? Because I assume it's, it's a bit trickier than what you might. And what was different than when you started out? Because it sounds like the dream, right?

You kind of went out, you actually managed to raise some funding. And now you're going to build these cool tools for developers. And your developer is great. You know, your customer is wonderful. I mean, a ton of things. I think that building a company can be very rewarding in a lot of ways. I think there are a lot of challenges.

I think some of the things that you learn to deal with, some of the things I think we've learned to deal with here are sort of the diversity of engineering workflows even across Silicon Valley and non-Silican Valley style companies, right, of like what does code review mean to them? What are they trying to get out of it? What's the purpose of it? And finding the middle road to blend all of those,

to blend all of those, I don't know, needs, desires, etc. It's difficult. It's challenging. It's something we build. I think one of the things that has most surprised me coming out of Facebook is what we talked around earlier is the tooling. I think working within a company, you probably have that team that is building really awesome tooling, you know, and like they really benefit from that.

When you come out here into the wild, as I call it, a lot less of that exists. And you find yourself being like, okay, how can you. can I set that up? How can I get it back? How can I get back to that? A company which I'll go a shout out to is stat sig. So Facebook had really wonderful sort of like had really wonderful experimentation tooling, feature flagging tooling. And when I left immediately, not immediately, but like six months in, I was like, okay, cool. Like we probably have our first like feature flag. Like how do we go ahead and turn this on? And I think you just mentioned you're at Uber. You had wonderful tooling there too. There's all of this stuff that's set up for you that you just take

for granted. And once you come out and have to build it, you have to start and start to be like, okay, how do we do this from scratch again, you know, or who can do this for me? Awesome. So with that, let's wrap up with some rapid questions. I'll ask a question and you just shoot whatever comes to your mind. Cool. So first, what is a tool that you saw inside Facebook that is now outside Facebook as well? And that can be used and you think it's pretty cool. I think Statsig is still the answer I'm going to give here. It's just, it's very cool having the feature flag system as used to. And it's actually an amalgamation of a handful of different Facebook tools.

Our feature flagging system gatekeeper and then our experimentation feature deltoid. And so it's cool to see both of those and QE2, I guess. It's kind of cool to see that sort of like reborn outside of Facebook. That's awesome. And this is the beauty of like, I think, you know, there was this like several years where DevTools started got funded because I feel a lot of the things that were hidden inside these companies are now outside as well. Totally.

I think also one of the things that's having worked at DevTools within Facebook and out of Facebook, I think one of the things that's so funny to me is that in Facebook, building DevTools is a lot easier. You don't need to worry about things like marketing. You don't need to worry about things like hiring. And I think it takes a lot of courage, and it's a very difficult path to build a tool outside of one of these large companies and say, no, I'm going to figure out how to make this work for the market. Because, again, the engineering cultures are different. The constraints are different.

The other tools you're integrating with are different. And so you have to start to rebuild all that. And so what is your favorite programming language? Always an interesting question. TypeScript by far. Love type. I think TypeScript is one of the most beautiful languages. And you're talking to a person that learned JavaScript when he was like 10.

And then the type system on top of TypeScript, I think, is really clean and does a great job with the constraints they have. I remember when TypeScript was created. And I remember there was a bit of skepticism around A, it was Microsoft. B, do we even need this? because, you know, the dynamic typing system is kind of like this wonderful thing that allows you to do so many crazy things. And, yeah, like, I think everyone has turned. Well, not everyone, but like most devs have slowly or surely seen the benefits of what we get. But what's typing and with a nice language.

My controversial opinion is I'm glad Microsoft doesn't want to do it because I think that Microsoft has some really wonderful compiler knowledge locked inside of it. And I similarly had that skepticism. I remember at the time thinking, like, we have flow. What's the purpose of this? And I think that the formal grammars and proofs that they've done around it are really, really excellent. You can go really deep on group theory or category theory just by looking at TypeScript. Yeah, I frequently look at other untyped languages. I'm looking at Ruby and I'm looking at Python.

And I don't think they've gotten close to what TypeScript has done to JavaScript yet. And then finally, what's a... a book that you read and would recommend fiction or nonfiction, one or two books? So I am an avid mystery reader. So the mystery book I always recommend people is the last days of night. It's a book around a Tesla, Edison, and Westinghouse and the race to invent the light bulb. It's written as historical fiction. I think it's phenomenal.

As one who currently works in technology in New York, there isn't a better book I can recommend to anyone in that position. Nonfiction, I read a lot of books these days. I've been actually reading a lot more product books. The timeless way of building is a really interesting book about architecture. The beginning is a little abstract to get through. But once you get to the sort of latter half, it talks around the importance of having the people who ultimately use tools or buildings be the ones who build those tools or buildings. Because that's the only way that you create a self-sustaining cycle.

I guess it kind of resonates with developers, developer tools. Yeah. Which seems kind of obvious because who else would program the tools that we use then developers? It's one of these weird things, but yeah. There's an MCSher painting to be made about that. Well, this is really interesting. I'm glad we were able to go a bit deeper into stacking than we previously touched on that article as well. And thank you for your time.

Cool. Of course. Thank you. This was always very fun. Thanks a lot to Thomas for all the details of meta's tooling, stack diffs, and on tooling trends. You can find Tomas on social media as listed in the show notes below. For more deep dives on tooling at Meta and more details on sagdifts, check out the pragmatic engineer deep dives linked in the show notes below.

If you enjoy the podcast, please do subscribe on your favorite podcast platform and on YouTube. This helps more people discover the podcast. A special thank you if you leave a rating. Thanks and see you in the next one.