Software at Scale - Software at Scale 52 - Building Build Systems with Benjy Weinberger

Episode Date: November 17, 2022

Benjy Weinberger is the co-founder of Toolchain, a build tool platform. He is one of the creators of the original Pants, an in-house Twitter build system focused on Scala, and was the VP of Infrastructure at Foursquare. Toolchain now focuses on Pants 2, a revamped build system.Apple Podcasts | Spotify | Google PodcastsIn this episode, we go back to the basics, and discuss the technical details of scalable build systems, like Pants, Bazel and Buck. A common challenge with these build systems is that it is extremely hard to migrate to them, and have them interoperate with open source tools that are built differently. Benjy’s team redesigned Pants with an initial hyper-focus on Python to fix these shortcomings, in an attempt to create a third generation of build tools - one that easily interoperates with differently built packages, but still fast and scalable.Machine-generated Transcript[0:00] Hey, welcome to another episode of the Software at Scale podcast. Joining me here today is Benji Weinberger, previously a software engineer at Google and Twitter, VP of Infrastructure at Foursquare, and now the founder and CEO of Toolchain.Thank you for joining us.Thanks for having me. It's great to be here. Yes. Right from the beginning, I saw that you worked at Google in 2002, which is forever ago, like 20 years ago at this point.What was that experience like? What kind of change did you see as you worked there for a few years?[0:37] As you can imagine, it was absolutely fascinating. And I should mention that while I was at Google from 2002, but that was not my first job.I have been a software engineer for over 25 years. And so there were five years before that where I worked at a couple of companies.One was, and I was living in Israel at the time. So my first job out of college was at Check Point, which was a big successful network security company. And then I worked for a small startup.And then I moved to California and started working at Google. And so I had the experience that I think many people had in those days, and many people still do, of the work you're doing is fascinating, but the tools you're given to do it with as a software engineer are not great.This, I'd had five years of experience of sort of struggling with builds being slow, builds being flaky with everything requiring a lot of effort. There was almost a hazing,ritual quality to it. Like, this is what makes you a great software engineer is struggling through the mud and through the quicksand with this like awful substandard tooling. And,We are not users, we are not people for whom products are meant, right?We make products for other people. Then I got to Google.[2:03] And Google, when I joined, it was actually struggling with a very massive, very slow make file that took forever to parse, let alone run.But the difference was that I had not seen anywhere else was that Google paid a lot of attention to this problem and Google devoted a lot of resources to solving it.And Google was the first place I'd worked and I still I think in many ways the gold standard of developers are first class participants in the business and deserve the best products and the best tools and we will if there's nothing out there for them to use, we will build it in house and we will put a lot of energy into that.And so it was for me, specifically as an engineer.[2:53] A big part of watching that growth from in the sort of early to late 2000s was. The growth of engineering process and best practices and the tools to enforce it and the thing i personally am passionate about is building ci but i'm also talking about.Code review tools and all the tooling around source code management and revision control and just everything to do with engineering process.It really was an object lesson and so very, very fascinating and really inspired a big chunk of the rest of my career.I've heard all sorts of things like Python scripts that had to generate make files and finally they move the Python to your first version of Blaze. So it's like, it's a fascinating history.[3:48] Maybe can you tell us one example of something that was like paradigm changing that you saw, like something that created like a magnitude, like order of magnitude difference,in your experience there and maybe your first aha moment on this is how good like developer tools can be?[4:09] Sure. I think I had been used to using make basically up till that point. And Google again was, as you mentioned, using make and really squeezing everything it was possible to squeeze out of that lemon and then some.[4:25] But when the very early versions of what became blaze which was that big internal build system which inspired basil which is the open source variant of that today. Hey one thing that really struck me was the integration with the revision controls system which was and i think still is performance.I imagine many listeners are very familiar with Git. Perforce is very different. I can only partly remember all of the intricacies of it, because it's been so long since I've used it.But one interesting aspect of it was you could do partial checkouts. It really was designed for giant code bases.There was this concept of partial checkouts where you could check out just the bits of the code that you needed. But of course, then the question is, how do you know what those bits are?But of course the build system knows because the build system knows about dependencies. And so there was this integration, this back and forth between the, um.[5:32] Perforce client and the build system that was very creative and very effective.And allowed you to only have locally on your machine, the code that you actually needed to work on the piece of the codebase you're working on,basically the files you cared about and all of their transitive dependencies. And that to me was a very creative solution to a problem that involved some lateral thinking about how,seemingly completely unrelated parts of the tool chain could interact. And that's kind of been that made me realize, oh, there's a lot of creative thought at work here and I love it.[6:17] Yeah, no, I think that makes sense. Like I interned there way back in 2016. And I was just fascinated by, I remember by mistake, I ran like a grep across the code base and it just took forever. And that's when I realized, you know, none of this stuff is local.First of all, like half the source code is not even checked out to my machine.And my poor grep command is trying to check that out. But also how seamlessly it would work most of the times behind the scenes.Did you have any experience or did you start working on developer tools then? Or is that just what inspired you towards thinking about developer tools?I did not work on the developer tools at Google. worked on ads and search and sort of Google products, but I was a big user of the developer tools.Exception which was that I made some contributions to the.[7:21] Protocol buffer compiler which i think many people may be familiar with and that is. You know if i very deep part of the toolchain that is very integrated into everything there and so that gave me.Some experience with what it's like to hack on a tool that's everyone in every engineer is using and it's the sort of very deep part of their workflow.But it wasn't until after google when i went to twitter.[7:56] I noticed that the in my time of google my is there the rest of the industry had not. What's up and suddenly i was sort of stressed ten years into the past and was back to using very slow very clunky flaky.Tools that were not designed for the tasks we were trying to use them for. And so that made me realize, wait a minute, I spent eight years using these great tools.They don't exist outside of these giant companies. I mean, I sort of assumed that maybe, you know, Microsoft and Amazon and some other giants probably have similar internal tools, but there's something out there for everyone else.And so that's when I started hacking on that problem more directly was at Twitter together with John, who is now my co-founder at Toolchain, who was actually ahead of me and ahead ofthe game at Twitter and already begun working on some solutions and I joined him in that.Could you maybe describe some of the problems you ran into? Like were the bills just taking forever or was there something else?[9:09] So there were...[9:13] A big part of the problem was that Twitter at the time, the codebase I was interested in and that John was interested in was using Scala. Scala is a fascinating, very rich language.[9:30] Its compiler is very slow. And we were in a situation where, you know, you'd make some small change to a file and then builds would take just,10 minutes, 20 minutes, 40 minutes. The iteration time on your desktop was incredibly slow.And then CI times, where there was CI in place, were also incredibly slow because of this huge amount of repetitive or near repetitive work. And this is because the build tools,etc. were pretty naive about understanding what work actually needs to be done given a set of changes.There's been a ton of work specifically on SBT since then.[10:22] It has incremental compilation and things like that, but nonetheless, that still doesn't really scale well to large corporate codebases that are what people often refer to as monorepos.If you don't want to fragment your codebase with all of the immense problems that that brings, you end up needing tooling that can handle that situation.Some of the biggest challenges are, how do I do less than recompile the entire codebase every time. How can tooling help me be smart about what is the correct minimal amount of work to do.[11:05] To make compiling and testing as fast as it can be?[11:12] And I should mention that I dabbled in this problem at Twitter with John. It was when I went to Foursquare that I really got into it because Foursquare similarly had this big Scala codebase with a very similar problem of incredibly slow builds.[11:29] The interim solution there was to just upgrade everybody's laptops with more RAM and try and brute force the problem. It was very obvious to everyone there, tons of,force-creation pattern still has lots of very, very smart engineers.And it was very obvious to them that this was not a permanent solution and we were casting around for...[11:54] You know what can be smart about scala builds and i remember this thing that i had hacked on twitter and. I reached out to twitter and ask them to open source it so we could use it and collaborate on it wasn't obviously some secret sauce and that is how the very first version of the pants open source build system came to be.I was very much designed around scarlet did eventually.Support other languages. And we hacked on it a lot at Foursquare to get it to...[12:32] To get the codebase into a state where we could build it sensibly. So the one big challenge is build speed, build performance.The other big one is managing dependencies, keeping your codebase sane as it scales.Everything to do with How can I audit internal dependencies?How do I make sure that it is very, very easy to accidentally create all sorts of dependency tangles and cycles and create a code base whose dependency structure is unintelligible, really,hard to work with and actually impacts performance negatively, right?If you have a big tangle of dependencies, you're more likely to invalidate a large chunk of your code base with a small change.And so tooling that allows you to reason about the dependencies in your code base and.[13:24] Make it more tractable was the other big problem that we were trying to solve. Mm-hmm. No, I think that makes sense.I'm guessing you already have a good understanding of other build systems like Bazel and Buck.Maybe could you walk us through what are the difference for PANs, Veevan? What is the major design differences? And even maybe before that, like, how was Pants designed?And is it something similar to like creating a dependency graph? You need to explicitly include your dependencies.Is there something else that's going on?[14:07] Maybe just a primer. Yeah. Absolutely. So I should mention, I was careful to mention, you mentioned Pants V1.The version of Pants that we use today and base our entire technology stack around is what we very unimaginatively call Pants V2, which we launched two years ago almost to the day.That is radically different from Pants V1, from Buck, from Bazel. It is quite a departure in ways that we can talk about later.One thing that I would say Panacea V1 and Buck and Bazel have in common is that they were designed around the use cases of a single organization. is a.[14:56] Open source variant or inspired by blaze its design was very much inspired by. Here's how google does engineering and a buck similarly for facebook and pansy one frankly very similar for.[15:11] Twitter and we sort of because Foursquare also contributed a lot to it, we sort of nudged it in that direction quite a bit. But it's still very much if you did engineering in this one company's specific image, then this might be a good tool for you.But you had to be very much in that lane.But what these systems all look like is, and the way they are different from much earlier systems is.[15:46] They're designed to work in large scalable code bases that have many moving parts and share a lot of code and that builds a lot of different deployables, different, say, binaries or DockerDocker images or AWS lambdas or cloud functions or whatever it is you're deploying, Python distributions, Java files, whatever it is you're building, typically you have many of them in this code base.Could be lots of microservices, could be just lots of different things that you're deploying.And they live in the same repo because you want that unity. You want to be able to share code easily. you don't want to introduce dependency hell problems in your own code. It's bad enough that we have dependency hell problems third-party code.[16:34] And so these systems are all if you squint at them from thirty thousand feet today all very similar in that they make that the problem of. Managing and building and testing and packaging in a code base like that much more tractable and the way they do this is by applying information about the dependencies in your code base.So the important ingredient there is that these systems understand the find the relatively fine grained dependencies in your code base.And they can use that information to reason about work that needs to happen. So a naive build system, you'd say, run all the tests in the repo or in this part of the repo.So a naive system would literally just do that, and first they would compile all the code.[17:23] But a scalable build system like these would say, well, you've asked me to run these tests, but some of them have already been cached and these others, okay, haven't.So I need to look at these ones I actually need to run. So let me see what needs to be done before I can run them.Oh, so these source files need to be compiled, but some of those already in cache and then these other ones I need to compile. But I can apply concurrency because there are multiple cores on this machine.So I can know through dependency analysis which compile jobs can run concurrently and which cannot. And then when it actually comes time to run the tests, again, I can apply that sort of concurrency logic.[18:03] And so these systems, what they have in common is that they use dependency information to make your building testing packaging more tractable in a large code base.They allow you to not have to do the thing that unfortunately many organizations find themselves doing, which is fragmenting the code base into lots of different bits andsaying, well, every little team or sub team works in its own code base and they consume each other's code through, um, so it was third party dependencies in which case you are introducing a dependency versioning hell problem.Yeah. And I think that's also what I've seen that makes the migration to a tool like this hard. Cause if you have an existing code base that doesn't lay out dependencies explicitly.[18:56] That migration becomes challenging. If you already have an import cycle, for example.[19:01] Bazel is not going to work with you. You need to clean that up or you need to create one large target where the benefits of using a tool like Bazel just goes away. And I think that's a key,bit, which is so fascinating because it's the same thing over several years. And I'm hoping that,it sounds like newer tools like Go, at least, they force you to not have circular dependencies and they force you to keep your code base clean so that it's easy to migrate to like a scalable build system.[19:33] Yes exactly so it's funny that is the exact observation that let us to pans to see to so they said pans to be one like base like buck was very much inspired by and developed for the needs of a single company and other companies were using it a little bit.But it also suffered from any of the problems you just mentioned with pans to for the first time by this time i left for square and i started to chain with the exact mission of every company every team of any size should have this kind of tooling should have this ability this revolutionary ability to make the code base is fast and tractable at any scale.And that made me realize.We have to design for that we have to design for not for. What a single company's code base looks like but we have to design.To support thousands of code bases of all sorts of different challenges and sizes and shapes and languages and frameworks so.We actually had to sit down and figure out what does it mean to make a tool.Like this assistant like this adoptable over and over again thousands of times you mentioned.[20:48] Correctly, that it is very hard to adopt one of those earlier tools because you have to first make your codebase conform to whatever it is that tool expects, and then you have to write huge amounts of manual metadata to describe all of the dependencies in your,the structure and dependencies of your codebase in these so-called build files.If anyone ever sees this written down, it's usually build with all capital letters, like it's yelling at you and that those files typically are huge and contain a huge amount of information your.[21:27] I'm describing your code base to the tool with pans be to eat very different approaches first of all we said this needs to handle code bases as they are so if you have circular dependencies it should handle them if you have. I'm going to handle them gracefully and automatically and if you have multiple conflicting external dependencies in different parts of your code base this is pretty common right like you need this version of whatever.Hadoop or NumPy or whatever it is in this part of the code base, and you have a different conflicting version in this other part of the code base, it should be able to handle that.If you have all sorts of dependency tangles and criss-crossing and all sorts of things that are unpleasant, and better not to have, but you have them, the tool should handle that.It should help you remove them if you want to, but it should not let those get in the way of adopting it.It needs to handle real-world code bases. The second thing is it should not require you to write all this crazy amount of metadata.And so with Panzer V2, we leaned in very hard on dependency inference, which means you don't write these crazy build files.You write like very tiny ones that just sort of say, you know, here is some code in this language for the build tool to pay attention to.[22:44] But you don't have to edit the added dependencies to them and edit them every time you change dependencies.Instead, the system infers dependencies by static analysis. So it looks at your, and it does this at runtime.So you, you know, almost all your dependencies, 99% of the time, the dependencies are obvious from import statements.[23:05] And there are occasional and you can obviously customize this because sometimes there are runtime dependencies that have to be inferred from like a string. So from a json file or whatever is so there are various ways to customize this and of course you can always override it manually.If you have to be generally speaking ninety.Seven percent of the boilerplate that used to going to build files in those old systems including pans v1 no. You know not claiming we did not make the same choice but we goes away with pans v2 for exactly the reason that you mentioned these tools,because they were designed to be adopted once by a captive audience that has no choice in the matter.And it was designed for how that code base that adopting code base already is. is these tools are very hard to adopt.They are massive, sometimes multi-year projects outside of that organization. And we wanted to build something that you could adopt in days to weeks and would be very easy,to customize to your code base and would not require these massive wholesale changes or huge amounts of metadata.And I think we've achieved that. Yeah, I've always wondered like, why couldn't constructing the build file be a part of the build. In many ways, I know it's expensive to do that every time. So just like.[24:28] Parts of the build that are expensive, you cache it and then you redo it when things change.And it sounds like you've done exactly that with BANs V2.[24:37] We have done exactly that. The results are cached on a profile basis. So the very first time you run something, then dependency inference can take some time. And we are looking at ways to to speed that up.I mean, like no software system has ever done, right? Like it's extremely rare to declare something finished. So we are obviously always looking at ways to speed things up.But yeah, we have done exactly what you mentioned. We don't, I should mention, we don't generate the dependencies into build for, we don't edit build files and then you check them in.We do that a little bit. So I mentioned you do still with PANSTL V2, you need these little tiny build files that just say, here is some code.They typically can literally be one line sometimes, almost like a marker file just to say, here is some code for you to pay attention to.We're even working on getting rid of those.We do have a little script that generates those one time just to help you onboard.But...[25:41] The dependencies really are just generated a runtime as on demand as needed and used a runtime so we don't have this problem of. Trying to automatically add or edit a otherwise human authored file that is then checked in like this generating and checking in files is.Problematic in many ways, especially when those files also have to take human written edits.So we just do away with all of that and the dependency inference is at runtime, on demand, as needed, sort of lazily done, and the information is cached. So both cached in memory in the surpassed V2 has this daemon that runs and caches a huge amount of state in memory.And the results of running dependency inference are also cached on disk. So they survive a daemon restart, etc.I think that makes sense to me. My next question is going to be around why would I want to use panthv2 for a smaller code base, right? Like, usually with the smaller codebase, I'm not running into a ton of problems around the build.[26:55] I guess, do you notice these inflection points that people run into? It's like, okay, my current build setup is not enough. What's the smallest codebase that you've seen that you think could benefit? Or is it like any codebase in the world? And I should start with,a better build system rather than just Python setup.py or whatever.I think the dividing line is, will this code base ever be used for more than one thing?[27:24] So if you have a, let's take the Python example, if literally all this code base will ever do is build this one distribution and a top level setup pie is all I need. And that is, you know, this,sometimes you see this with open source projects and the code base is going to remain relatively small, say it's only ever going to be a few thousand lines and the tests, even if I runthe tests from scratch every single time, it takes under five minutes, then you're probably fine.But I think two things I would look at are, am I going to be building multiple things in this code base in the future, or certainly if I'm doing it now.And that is much more common with corporate code bases. You have to ask yourself, okay, my team is growing, more and more people are cooperating on this code base.I want to be able to deploy multiple microservices. I want to be able to deploy multiple cloud functions.I want to be able to deploy multiple distributions or third-party artifacts.I want to be able to.[28:41] You know, multiple sort of data science jobs, whatever it is that you're building. If you want, if you ever think you might have more than one, now's the time to think about,okay, how do I structure the code base and what tooling allows me to do this effectively?And then the other thing to look at is build times. If you're using compiled languages, then obviously compilation, in all cases testing, if you start to see like, I can already see that that tests are taking five minutes, 10 minutes, 15 minutes, 20 minutes.Surely, I want some technology that allows me to speed that up through caching, through concurrency, through fine-grained invalidation, namely, don't even attempt to do work that isn't necessary for the result that was asked for.Then it's probably time to start thinking about tools like this, because the earlier you adopt it, the easier it is to adopt.So if you wait until you've got a tangle of multiple setup pies in the repo and it's unclear how you manage them and how you keep their dependencies synchronized,so there aren't version conflicts across these different projects, specifically with Python,this is an interesting problem.I would say with other languages, there is more because of the compilation step in jvm languages or go you.[30:10] Encounter the need for a build system much much earlier a bill system of some kind and then you will ask yourself what kind with python because you can get a bite for a while just running. What are the play gate and pie test and i directly and all everything is all together in a single virtual and.But the Python tooling, as mighty as it is, mostly is not designed for larger code bases with multiple, that deploy multiple things and have multiple different sets of.[30:52] Internal and external dependencies the tooling generally implicitly assume sort of one top level set up i want top level. Hi project dot com all you know how are you configuring things and so especially using python let's say for jango flask apps or for data scienceand your code base is growing and you've hired a bunch of data scientists and there's more and more code going in there. With Python, you need to start thinking about what tooling allows me to scale this code base. No, I think I mostly resonate with that. The first question that comes to my mind is,let's talk specifically about the deployment problem. If you're deployed to multiple AWS lambdas or cloud functions or whatever, the first thought that would come to my mind isis I can use separate Docker images that can let me easily produce this container image that I can ship independently.Would you say that's not enough? I totally get that for the build time problem.A Docker image is not going to solve anything. But how about the deployment step?[32:02] So again, with deployments, I think there are two ways where a tool like this can really speed things up.One is only build the things that actually need to be redeployed. And because the tool understands dependencies and can do change analysis, it can figure that out.So one of the things that HansB2 does is it integrates with Git.And so it natively understands how to figure out Git diffs. So you can say something like, show me all the whatever, lambdas, let's say, that are affected by changes between these two branches.[32:46] And it knows and it understands it can say, well, these files changed and you know, we, I understand the transitive dependencies of those files.So I can see what actually needs to be deployed. And, you know, many cases, many things will not need to be redeployed because they haven't changed.The other thing is there's a lot of performance improvements and process improvements around building those images. So, for example, we have for Python specifically, we have an executable format called PEX,which stands for Python executable, which is a single file that embeds all of your Python code that is needed for your deployable and all of its external requirements, transitive external requirements, all bundled up into this single sort of self-executing file.This allows you to do things like if you have to deploy 50 of these, you can basically have a single docker image.[33:52] The different then on top of that you add one layer for each of these fifty and the only difference in that layer is the presence of this pecs file. Where is without all this typically what you would do is.You have fifty docker images each one of which contains a in each one of which you have to build a virtual and which means running.[34:15] Pip as part of building the image, and that gets slow and repetitive, and you have to do it 50 times.We have a lot of ways to speed up. Even if you are deploying 50 different Docker images, we have ways of speeding that up quite dramatically.Because again, of things like dependency analysis, the PECS format, and the ability to build incrementally.Yeah, I think I remember that at Dropbox, we came up with our own, like, par format to basically bundle up a Python binary with, I think par stood for Python archive. I'm notentirely sure. But it did something remarkably similar to solve exactly this problem. It just takes so long, especially if you have a large Python code base. I think that makes sense to me. The other thing that one might ask is, with Python, you don't really have,too long of a build time, is what you would guess, because there's nothing to build. Maybe myPy takes some time to do some static analysis, and, of course, your tests can take forever,and you don't want to rerun them. But there isn't that much of a build time that you have to think about. Would you say that you agree with this, or there's some issues that end,up happening on real-world code basis.[35:37] Well that's a good question the word builds means different things to different people and we recently taken to using the time see i more. Because i think that is clear to people what that means but when i say build or see i mean it in the law in in the extended sense everything you do to go from.Human written source code to a verified.Test did. deployable artifact and so it's true that for python there's no compilation step although arguably. Running my pie is really important and now that i'm really in the habit of using.My pie i will probably never not use it on python code ever again but so that are.[36:28] Sort of build-ish steps for Python such as type checking, such as running code generators like Thrift or Protobuf.And obviously a big, big one is running, resolving third-party dependencies such as running PIP or poetry or whatever it is you're using. So those are all build steps.But with Python, really the big, big, big thing is testing and packaging and primarily testing.And so with Python, you have to be even more rigorous about unit testing than you do with other languages because you don't have a compiler that is catching whole classes of bugs.So and again, MyPy and type checking does really help with that. And so when I build to me includes, build in the large sense includes running tests,includes packaging and includes everything, all the quality control that you run typically in CI or on your desktop in order to go say, well, I've made some edits and here's the proof that these edits are good and I can merge them or deploy them.[37:35] I think that makes sense to me. And like, I certainly saw it with the limited number of testing, the limited amount of type checking you can do with Python, like MyPy is definitelyimproving on this. You just need to unit test a lot to get the same amount of confidence in your own code and then unit tests are not cheap. The biggest question that comes tomy mind is that is BANs V2 focused on Python? Because I have a TypeScript code base at my workplace and I would love to replace the TypeScript compiler with something that was slightly smarter and could tell me, you know what, you don't need to run every unit test every change.[38:16] Great question so when we launched a pass me to which was two years ago. The.We focused on python and that was the initial language we launched with because you had to start somewhere and in the city ten years in between the very scarlet centric work we were doing on pansy one. And the launch of hands be to something really major happened in the industry which was the python skyrocketed in popularity sky python went from.Mostly the little scripting language around the edges of your quote unquote real code, I can use python like fancy bash to people are building massive multi billion dollar businesses entirely on python code bases and there are a few things that drove this one was.I would say the biggest one probably was the python became the. Language of choice for data science and we have strong support for those use cases. There was another was the,Django and Flask became very popular for writing web apps more and more people were used there were more in Intricate DevOps use cases and Python is very popular for DevOps for various good reasons. So.[39:28] Python became super popular. So that was the first thing we supported in pants v2, but we've since added support for or Go, Java, Scala, Kotlin, Shell.Definitely what we don't have yet is JavaScript TypeScript. We are looking at that very closely right now, because that is the very obvious next thing we want to add.Actually, if any listeners have strong opinions about what that should look like, we would love to hear from them or from you on our Slack channels or on our GitHub discussions where we are having some lively discussions about exactly this because the JavaScript.[40:09] And TypeScript ecosystem is already very rich with tools and we want to provide only value add, right? We don't want to say, you have to, oh, you know, here's another paradigm you have to adopt.And here's, you know, you have to replace, you've just done replacing whatever this with this, you know, NPM with yarn. And now you have to do this thing. And now we're, we don't want to beanother flavor of the month. We only want to do the work that uses those tools, leverages the existing ecosystem but adds value. This is what we do with Python and this is one of the reasons why our Python support is very, very strong, much stronger than any other comparable tool out there is.[40:49] A lot of leaning in on the existing Python tool ecosystem but orchestrating them in a way that brings rigor and speed to your builds.And I haven't used the word we a lot. And I just kind of want to clarify who we is here.So there is tool chain, the company, and we're working on, um, uh, SAS and commercial, um, solutions around pants, which we can talk about in a bit.But there is a very robust open source community around pants that is not. tightly held by Toolchain, the company in a way that some other companies open source projects are.So we have a lot of contributors and maintainers on Pants V2 who are not working at Toolchain, but are using Pants in their own companies and their own organizations.And so we have a very wide range of use cases and opinions that are brought to bear. And this is very important because, as I mentioned earlier,we are not trying to design a system for one use case, for one company or a team's use case.We are trying, you know, we are working on a system we want.[42:05] Adoption for over and over and over again at a wide variety of companies. And so it's very important for us to have the contributions and the input from a wide variety of teams and companiesand people. And it's very fortunate that we now do. I mean, on that note, the thing that comes to my mind is another benefit of your scalable build system like Vance or Bazel or Buck is that youYou don't have to learn various different commands when you are spelunking through the code base, whether it's like a Go code base or like a Java code base or TypeScript code base.You just have to run pants build X, Y, Z, and it can construct the appropriate artifacts for you. At least that was my experience with Bazel.Is that something that you are interested in or is that something that pants V2 does kind of act as this meta layer for various other build systems or is it much more specific and knowledgeable about languages itself?[43:09] It's, I think your intuition is correct. The idea is we want you to be able to do something like pants test or pants test, you know, give it a path to a directory and it understands what that means.Oh, this directory contains Python code. Therefore, I should run PyTest in this way. And oh, Oh, it also contains some JavaScript code, so I should run the JavaScript test in this way.And it basically provides a conceptual layer above all the individual tools that gives you this uniformity across frameworks, across languages.One way to think about this is.[43:52] The tools are all very imperative. say you have to run them with a whole set of flags and inputs and you have to know how to use each one separately. So it's like having just the blades of a Swiss Army knife withno actual Swiss Army knife. A tool like Pants will say, okay, we will encapsulate all of that complexity into a much more simple command line interface. So you can do, like I said,test or pants lint or pants format and it understands, oh, you asked me to format your code. I see that you have the black and I sort configured as formatters. So I will run them. And I happen to know that formatting, because formatting can change the source files,I have to run them sequentially. But when you ask for lint, it's not changing the source files. So I know that I can run them multiple lint as concurrently, that sort of logic. And And different tools have different ways of being configured or of telling you what they want to do, but we...[44:58] Can't be to sort of encapsulate all that away from you and so you get this uniform simple command line interface that abstract away a lot of the specifics of these tools and let you run simple commands and the reason this is important is that. This extra layer of indirection is partly what allows pants to apply things like cashing.And invalidation and concurrency because what you're saying is.[45:25] Hey, the way to think about it is not, I am telling pants to run tests. It is I am telling pants that I want the results of tests, which is a subtle difference.But pants then has the ability to say, well, I don't actually need to run pi test on all these tests because I have results from some of them already cached. So I will return them from cache.So that layer of indirection not only simplifies the UI, but provides the point where you can apply things like caching and concurrency.Yeah, I think every programmer wants to work with declarative tools. I think SQL is one of those things where you don't have to know how the database works. If SQL were somewhat easier, that dream would be fulfilled. But I think we're all getting there.I guess the next question that I have is, what benefit do I get by using the tool chain, like SaaS product versus Pants V2?When I think about build systems, I think about local development, I think about CI.[46:29] Why would I want to use the SaaS product? That's a great question.So Pants does a huge amount of heavy lifting, but in the end it is restricted to the resources is on the machine on which it's running. So when I talk about cash, I'm talking about the local cash on that machine. When I talk about concurrency, I'm talking about using,the cores on your machine. So maybe your CI machine has four cores and your laptop has eight cores. So that's the amount of concurrency you get, which is not nothing at all, which is great.[47:04] Thanks for watching![47:04] You know as i mentioned i worked at google for many years and then other companies where distributed systems were saying like i come from a distributed systems background and it really. Here is a problem.All of a piece of work taking a long time because of. Single machine resource constraints the obvious answer here is distributed distributed the work user distributed system and so that's what tool chain offers essentially.[47:30] You configure Pants to point to the toolchain system, which is currently SAS.And we will have some news soon about some on-prem solutions.And now the cache that I mentioned is not just did this test run with these exact inputs before on my machine by me me while I was iterating, but has anyone in my organization or any CI run this test before,with these exact inputs?So imagine a very common situation where you come in in the morning and you pull all the changes that have happened since you last pulled.Those changes presumably passed CI, right? And the CI populated the cache.So now when I run tests, I can get cache hits from the CI machine.[48:29] Now pretty much, yeah. And then with concurrency, again, so let's say, you know, post cache, there are still 200 tests that need to be run.I could run them eight at a time on my machine or the CI machine could run them, you know, say, four at a time on four cores, or I could run 50 or 100 at a time on a cluster of machines.That's where, again, as your code base gets bigger and bigger, that's where some massive, massive speedups come in.The other aspects of the... I should mention that the remote execution that I just mentioned is something we're about to launch. It is not available today. The remote caching is.The other aspects are things like observability. So when you run builds on your laptop or CI, they're ephemeral.Like the output gets lost in the scroll back.And it's just a wall of text that gets lost with them.[49:39] Toolchain all of that information is captured and stored in structured form so you have. Hey the ability to see past bills and see build behavior over time and drill death search builds and drill down into individual builds and see well.How often does this test fail and you know when did this get slow all this kind of information and so you get.This more enterprise level.Observability into a very core piece of developer productivity, which is the iteration time.The time it takes to run tests and build deployables and parcel the quality control checks so that you can merge and deploy code directly relates to time to release.It directly relates to some of the core metrics of developer productivity. How long is it going to take to get this thing out the door?And so having the ability to both speed that up dramatically through distributing the work and having observability into what work is going on, that is what toolchain provides,on top of the already, if I may say, pretty robust open source offering.[51:01] So yeah, that's kind of it.[51:07] Pants on its own gives you a lot of advantages, but it runs standalone. Plugging it into a larger distributed system really unleashes the full power of Pants as a client to that system.[51:21] No, I think what I'm seeing is this interesting convergence. There's several companies trying to do this for Bazel, like BuildBuddy and Edgeflow. So, and it really sounds like the build system of the future, like 10 years from now.[51:36] No one will really be developing on their local machines anymore. Like there's GitHub code spaces on one side. It's like you're doing all your development remotely.[51:46] I've always found it somewhat odd that development that happens locally and whatever scripts you need to run to provision your CI machine to run the same set of testsare so different sometimes that you can never tell why something's passing locally and failing in in CI or vice versa. And there really should just be this one execution layer that can say, you know what, I'm going to build at a certain commit or run at a certain commit.And that's shared between the local user and the CI user. And your CI script is something as simple as pants build slash slash dot dot dot. And it builds the whole code base for,you. So yeah, I certainly feel like the industry is moving in that direction. I'm curious whether You think that's the same.Do you have an even stronger vision of how folks will be developing 10 years from now? What do you think it's going to look like?Oh, no, I think you're absolutely right. I think if anything, you're underselling it. I think this is how all development should be and will be in the future for multiple reasons.One is performance.[52:51] Two is the problem of different platforms. And so today, big thorny problem is I want to, you know, I want to,I'm developing on my Mac book, but the production, so I'm running, when I run tests locally and when I run anything locally, it's running on my Mac book, but that's not our deployable, right?Typically your deploy platform is some flavor of Linux. So...[53:17] With the distributed system approach you can run the work in. Containers that exactly match your production environments you don't even have to care about can this run.On will my test pass on mac os do i need ci the runs on mac os just to make sure the developers can. past test on Mac OS and that is somehow correlated with success on the production environment.You can cut away a whole suite of those problems, which today, frankly, I had mentioned earlier, you can get cache hits on your desktop from remote, from CI populating the cache.That is hampered by differences in platform.Is hampered by other differences in local setup that we are working to mitigate. But imagine a world in which build logic is not actually running on your MacBook, or if it is,it's running in a container that exactly matches the container that you're targeting.It cuts away a whole suite of problems around platform differences and allows you to focus because on just a platform you're actually going to deploy too.[54:35] And the...[54:42] And just the speed and the performance of being able to work and deploy and the visibility that it gives you into the productivity and the operational work of your development team,I really think this absolutely is the future.There is something very strange about how in the last 15 years or so, so many business functions have had the distributed systems treatment applied to them.Function is now that there are these massive valuable companies providing systems that support sales and systems that support marketing and systems that support HR and systems supportoperations and systems support product management and systems that support every business function,and there need to be more of these that support engineering as a business function.[55:48] And so i absolutely think the idea that i need a really powerful laptop so that my running tests can take thirty minutes instead of forty minutes when in reality it should take three minutes is. That's not the future right the future is to as it has been for so many other systems to the web the laptop is that i can take anywhere is.Particularly in these work from home times, is a work from anywhere times, is just a portal into the system that is doing the actual work.[56:27] Yeah. And there's all these improvements across the stack, right? When I see companies like Versel, they're like, what if you use Next.js, we provide the best developer platform forthat and we want to provide caching. Then there's like the lower level systems with build systems, of course, like bands and Bazel and all that. And at each layer, we're kindof trying to abstract the problem out. So to me, it still feels like there is a lot of innovation to be done. And I'm also going to be really curious to know, you know, there'sgoing to be like a few winners of this space, or if it's going to be pretty broken up. And like everyone's using different tools. It's going to be fascinating, like either way.Yeah, that's really hard to know. I think one thing you mentioned that I think is really important is you said your CI should be as simple as just pants build colon, colon, or whatever.That's our syntax would be sort of pants test lint or whatever.I think that's really important. So.[57:30] Today one of the big problems with see i. Which is still growing right now home market is still growing is more more teams realize the value and importance of automated.Very aggressive automated quality control. But configuring CI is really, really complicated. Every CI provider have their own configuration language,and you have to reason about caching, and you have to manually construct cache keys to the extent,that caching is even possible or useful.There's just a lot of figuring out how to configure and set up CI, And even then it's just doing the naive thing.[58:18] So there are a couple of interesting companies, Dagger and Earthly, or interesting technologies around simplifying that, but again, you still have to manually,so they are providing a, I would say, better config and more uniform config language that allows you to, for example, run build steps in containers.And that's not nothing at all.[58:43] Um, but you still manually creating a lot of, uh, configuration to run these very coarse grained large scale, long running build steps, you know, I thinkthe future is something like my entire CI config post cloning the repo is basically pants build colon, colon, because the system does the configuration for you.[59:09] It figures out what that means in a very fast, very fine grained way and does not require you to manually decide on workflows and steps and jobs and how they all fit together.And if I want to speed this thing up, then I have to manually partition the work somehow and write extra config to implement that partitioning.That is the future, I think, is rather than there's the CI layer, say, which would be the CI providers proprietary config or theodagger and then underneath that there is the buildtool, which would be Bazel or Pants V2 or whatever it is you're using, could still be we make for many companies today or Maven or Gradle or whatever, I really think the future is the integration of those two layers.In the same way that I referenced much, much earlier in our conversation, how one thing that stood out to me at Google was that they had the insight to integrate the version control layer and the build tool to provide really effective functionality there.I think the build tool being the thing that knows about your dependencies.[1:00:29] Can take over many of the jobs of the c i configuration layer in a really smart really fast. Where is the future where essentially more and more of how do i set up and configure and run c i is delegated to the thing that knows about your dependencies and knows about cashing and knows about concurrency and is able,to make smarter decisions than you can in a YAML config file.[1:01:02] Yeah, I'm excited for the time that me as a platform engineer has to spend less than 5% of my time thinking about CI and CD and I can focus on other things like improving our data models rather than mucking with the YAML and Terraform configs. Well, yeah.Yeah. Yeah. Today you have to, we're still a little bit in that state because we are engineers and because we, the tools that we use are themselves made out of software. There's,a strong impulse to tinker and there's a strong impulse sake. Well, I want to solve this problem myself or I want to hack on it or I should be able to hack on it. And that's, you should be able to hack on it for sure. But we do deserve more tooling that requires less hacking,and more things and paradigms that have tested and have survived a lot of tire kicking.[1:02:00] Will we always need to hack on them a little bit? Yes, absolutely, because of the nature of what we do. I think there's a lot of interesting things still to happen in this space.Yeah, I think we should end on that happy note as we go back to our day jobs mucking with YAML. Well, thanks so much for being a guest. I think this was a great conversation and I hope to have you again for the show sometime.Would love that. Thanks for having me. It was fascinating. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Welcome to Software at Scale, a podcast where we discuss the technical stories behind large software applications. I'm your host, Utsav Shah, and thank you for listening. Hey, welcome to another episode of the Software at Scale podcast. Joining me here today is Benji Weinberger, previously a software engineer at Google and Twitter, VP of infrastructure at Foursquare, and now the founder and CEO of Toolchain. Thank you for joining us. Thanks for having me. It's great to be here. Yeah. So right from the beginning, I saw that you worked at Google in 2002, which is like forever ago, like 20 years ago at this point.

What was that experience like? What kind of change did you see as you worked there for a few years? As you can imagine, it was absolutely fascinating. And I should mention that while I was at Google from 2002, but that was not my first job. I have been a software engineer for over 25 years. And so there were five years before that, where I worked at a couple of companies, one was, and I was living in Israel at the time. So my first job out of college was at Checkpoint, which was a big, successful network security company. And then I worked for a small startup. And then I moved to California and started working at Google. And so I had the experience that I think many people had in those days, and many people still do, of the work you're doing is fascinating, but the tools you're given to do it with as a software engineer are not great. There's this, I'd had five years of experience of

sort of struggling with builds being slow builds being flaky with everything requiring a lot of effort. There was a almost a hazing ritual quality to it of like, this is what makes you a great software engineer is struggling through the mud and through the quicksand with this like awful substandard tooling and no you know we are not users we are not people for whom products are meant right we make products for other people then i got to google and google when i joined it was actually struggling with a very massive, very slow make file that took forever to parse, let alone run. But the difference was, that I had not seen anywhere

else was that Google paid a lot of attention to this problem. And Google devoted a lot of resources to solving it. And Google was the first place I'd worked. And I still, I think in many ways, the gold standard of developers are first class participants in the business and deserve the best products and the best tools. And we will, if there's nothing out there for them to use, we will build it in house and we will put a lot of energy into that and so it was for me specifically as an engineer a big part of watching that growth from in in the sort of early to uh late 2000s was the growth of engineering process and best practices and the tools to enforce it and the thing I personally am passionate about is build and CI,

but I'm also talking about code review tools and all the tooling around source code management and revision control and just everything to do with engineering process. It really was an object lesson and so very, very fascinating and really inspired a big chunk of the rest of my career. I've heard all sorts of things like Python scripts that had to generate make files.

And finally, they moved the Python to your first version of Blaze. It's a fascinating history. Maybe, can you tell us one example of something that was like paradigm changing that that you saw like something that created like a magnitude like order of magnitude difference in your experience there and maybe your first aha moment on this is how good like developer tools can be um sure i think i had been used to using make basically up till that point and google again was as you mentioned uh using make and really squeezing everything it was possible to squeeze out of that lemon and then some but when the very early versions of what became blaze uh which was their big internal build system

which inspired uh basil which is the sort of open source variant of that today they one thing that really struck me was the integration with the revision control system, which was, and I think still is, Perforce. So Perforce, I imagine many listeners are very familiar with Git. Perforce is very different. And I can only partly remember all of the intricacies of it because it's been so long since I've used it.

But one interesting aspect of it was you could do partial checkouts. It really was designed for giant code bases. And so there was this concept of partial checkouts where you could check out just the bits of the code that you needed. But, of course, then the question is how do you know what those bits are? But, of course, the build system knows because the build system knows about dependencies.

And so there was this integration, this back and forth between the Perforce client and the build system that was very creative and very effective and allowed you to only have locally on your machine the code that you actually needed to work on the piece of the code base that you were working on, basically the files you cared about and all of their transitive dependencies. And that to me was a very creative solution to a problem that involved some lateral thinking about how seemingly completely unrelated parts of the tool chain could interact.

And that's kind of been, that made me realize, oh, there's a lot of creative thought at work here, and I love it. Yeah, no, I think that makes sense. Like, I interned there way back in 2016, and I was just fascinated by, I remember by mistake, I ran like a grep across the code base, and it just took forever. And that's when I realized, you know, none of this stuff is local. First of all, like half the source code is not even checked out to my machine. And my poor grep

command is trying to check that out. But also how seamlessly it would work most of the times behind the scenes. Did you have any experience or did you start working on developer tools then? Or is that just kind of what inspired you towards thinking about developer tools? I did not work on the developer tools at Google. I worked on ads and search and sort of Google products, but I was a big user of the developer tools at Google. I worked on ads and search and Google products, but I was a big user of the developer tools. There was one exception, which was that I made some contributions to the protocol buffer compiler, which I think many people may be familiar with. And that is, you know, a very deep part of the tool chain there that is very integrated into everything there. And so that gave me some experience with what it's like to hack on a tool that, you know, everyone in sort of every engineer is using, and it's a sort of every engineer is using and it's a sort of very deep part of their workflow

but it wasn't until after google when i went to twitter and noticed that the in in my time at google my eight years there the rest of the industry had not caught up and suddenly i was sort of thrust 10 years into the past and was back to using very slow, very clunky, flaky tools that were not designed for the tasks we were trying to use them for. And so that made me realize, wait a minute, I spent eight years using these great tools. They don't exist outside of these giant companies. I mean, I sort of assumed that maybe, you know, Microsoft and Amazon and some other giants probably have similar internal tools, but there's something out there for everyone else. And so that's

when I started hacking on that problem more directly was at Twitter, together with John, who is now my co-founder at Toolchain, who was actually ahead of me and head of the game at Twitter and already begun working on some solutions. And I joined him in that. Could you maybe describe some of the problems you ran into? Like, were the bills just taking forever? Or was there something else?

So there were a big part of the problem was that uh twitter at the time the the code base i was interested in and that john was interested in was using Scala. And Scala is a fascinating, very rich language. Its compiler is very slow. And we were in a situation where, you know, you'd make some small change to a file, and then builds would take just 10 minutes, 20 minutes, 40 minutes. The iteration time on your desktop was incredibly slow. And then CI times, where there was CI in place, were also incredibly slow because of this huge amount of repetitive or

near repetitive work. And this is because the build tools, SBT, etc. were pretty naive about understanding what work actually needs to be done given a set of changes and there's been a ton of work specifically on sbt since then it has uh incremental compilation and things like that but nonetheless that still doesn't really scale well to large sort of corporate code bases that are sort of what people often refer to as monorepos. So if you don't want to fragment your code base with all of the immense problems that that brings, you end up needing tooling that can handle that situation.

So some of the biggest challenges are, how do I do less than recompile the entire code base every time? How can tooling help me be smart about what is the correct minimal amount of work to do to make compiling and testing as fast as it can be. And I should mention that I dabbled in this problem at Twitter with John. It was when I went to Foursquare that I really got into it because Foursquare similarly had this big Scala code base with a very similar problem of incredibly slow builds.

And the interim solution there was to just upgrade everybody's laptops with more RAM and try and brute force the problem. But it was very obvious to everyone there, like tons of Foursquare still has lots of very, very smart engineers. And it was very obvious to them that this was not a permanent solution and we were casting around for you know what can be smart about scala builds and i remembered this thing that uh i john and i had hacked on at twitter and i reached out to twitter and asked them to open source it so we could use it and collaborate on it.

It wasn't obviously some secret source. And that is how the very first version of the Pants open source build system came to be. It was very much designed around Scala, although it did eventually support other languages. And we hacked on it a lot at Foursquare to get the code base into a state where we could build it sensibly. So the one big challenge is build speed, build performance. The other big one is managing dependencies, keeping your code base sane as it scales, everything to do with how can I audit internal dependencies? How do I make sure that it is very, very easy to accidentally create all sorts of dependency tangles and cycles and create a code base whose dependency structure is unintelligible, really hard to work with, and actually impacts performance negatively, right? If you have a big tangle of dependencies, you're more likely to invalidate a large chunk of your

code base with a small change. And so tooling that allows you to reason about the dependencies in your code base and make it more tractable was the other big problem that we were trying to solve. No, I think that makes sense. I'm guessing you already have like a good understanding of other like build systems like Bazel and Buck. Maybe could you walk us through like, you know, what are the difference, maybe this is for Pants V1, like what are the difference for PANS V1? What is the major design differences? And even maybe before that, like, how was PANS designed? And is it something similar to like creating a dependency graph? You need to explicitly include your dependencies.

Is there something else that's going on? Sure. Yeah. Absolutely. So I should mention i i was careful to mention you mentioned pants v1 the version of pants that we use today and base our entire technology stack around is what we very unimaginatively call pants2, which we launched two years ago almost to the day. And that is radically different from Pantv1, from Buck, from Bazel. It is quite a departure in ways that we can talk about later. But one thing that I would say Pantv1 and Buck and Bazel have in common is that they

were designed around the use cases of a single organization. Bazel is an open source variant or inspired by Blaze. Its design was very much inspired by, here's how Google does engineering. And Buck, similarly for Facebook and pants v1 frankly very similar for twitter and we sort of because foursquare also contributed a lot to it we sort of nudged it in the that direction quite a bit but it's still very much if you did engineering in this one company's specific image then this might be a good tool for you but you had to be um very much um you know in in that lane but what these systems all look like is and and the way they are different from much earlier systems is they're designed to work in large scalable code bases

that have many moving parts and share a lot of code and that build a lot of different deployables, different, say, binaries or Docker images or AWS Lambdas or cloud functions or whatever it is you're deploying, Python distributions, JAR files, whatever it is you're building, Python distributions, JAR files, whatever it is you're building, typically you have many of them in this code base. Could be lots of microservices, could be just lots of different things that you're deploying,

and they live in the same repo because you want that unity. You want to be able to share code easily. You don't want to introduce dependency hell problems in your own code. It's bad enough that we have dependency hell problems with third-party code. And so these systems are all, if you squint at them from 30,000 feet, they are all very similar in that they make the problem of managing and building and testing and packaging in a code base like that much more tractable. And the way they do this is by applying information about the dependencies in your code base. So the important ingredient there is that these systems understand the relatively fine-grained dependencies in your code base, and they can use that information to reason about work that needs to happen.

So a naive build system, you'd say, you know, run all the tests in the repo or in this part of the repo. So a naive system would literally just do that, right? And first they would compile all the code. A scalable build system like these would say, well, you've asked me to run these tests, but some of them have already been cached and these others, okay, haven't. So I need to look at these are the ones I actually need to run. So let other ones I need to compile, but I can apply concurrency because there are multiple cores on this machine, so I can know through dependency analysis which compile jobs can run concurrently and which cannot, and then when it actually comes time to run the tests, again, I can apply that sort of concurrency logic. And so these systems, what they have in common is that they use dependency information to make your building testing packaging more tractable in a large code base into lots of different bits and saying, well, every little team or sub team works in its own code base, and they consume each other's code through

services, third party dependencies, in which case you are introducing a dependency versioning hell problem. Yeah. And I think that's also what I've seen that makes the migration to a tool like this hard. Because if you have an existing code base that doesn't lay out dependencies explicitly, then that migration becomes challenging. Like if you already have like an import cycle, for example, Bazel is not going to work with you. You need to clean that up or you need to create one large target where the benefits of using a tool like Bazel just goes away and I think that's that's a key bit which is so fascinating because it's the same thing over several years and I'm hoping that it sounds like newer tools like Go at least

they force you to not have circular dependencies and they force you to keep your code base clean so that it's easy to migrate like a scalable build system yes exactly so it's funny that is the exact observation that led us to pants v2 so as i said pants v1 like basil like buck was very much inspired by and developed for the needs of a single company and other companies were using it a little bit, but it also suffered from many of the problems you just mentioned. With Pansv2, for the first time, by this time, I'd left Foursquare and I had started Toolchain with the exact mission of every company, every team of any size should have this kind of tooling,

should have this ability, this revolutionary ability to make their codebases fast and tractable at any scale. And that made me realize we have to design for that. We have to design for not for what a single company's codebase looks like, but we have to design to support thousands of code bases of all sorts of different challenges and sizes and shapes and languages and frameworks. So we actually had to sit down and figure out what does it mean to make a tool like this, a system like this, adoptable over and over again, thousands of times. You mentioned correctly, that it is very hard to adopt one of those earlier tools, because you have to first make your code base conform to whatever it is that tool expects. And then you have to write huge amounts of manual metadata

to describe all of the dependencies in your the structure and dependencies of your code base in these so-called build files. If anyone ever sees this written down, it's usually BUILD with all capital letters, like it's yelling at you, like build. And those files typically are huge and contain a huge amount of information describing your code base to the tool. With PANSv2, we took very different approaches.

First of all, we said this needs to handle code bases as they are. So if you have circular dependencies, it should handle them. If you have... And it should handle them gracefully and automatically. And if you have multiple conflicting external dependencies in different parts of your code base, this is pretty common, right? Like you need this version of whatever Hadoop or NumPy or whatever it is in this part of the code base, and you have a different

conflicting version, this other part of the code base, it should be able to handle that. If you have all sorts of dependency tangles and crisscrossing and all sorts of things that are unpleasant, and better not to have, but you have them, the tool should handle that. It should help you remove them if you want to, but it should not let those get in the way of adopting it. So it needs to handle real-world code bases. And the second thing is it should not require you to write all this crazy amount of metadata. And so with Panzer V2, we leaned in very hard on dependency inference, which means you don't write these crazy build files. You write like very tiny ones that just sort of say, you know, here is some code in this language for the build

tool to pay attention to, but you don't have to edit, add dependencies to them and edit them every time you change dependencies. Instead, the system infers dependencies by static analysis. And it does this at runtime. So almost all your dependencies, 99% of the time, the dependencies are obvious from import statements. and there are occasional and you can obviously customize this because sometimes there are runtime dependencies that have to be inferred from like a string or from a json file or whatever it is so there are various ways to customize this and of course you can always override it manually if you have to but generally speaking 97 of the boilerplate that used to go into build files in those old systems,

including Pans V1. I'm not claiming we did not make the same choice, but it goes away with Pans V2 for exactly the reason that you mentioned. These tools, because they were designed to be adopted once by a captive audience that has no choice in the matter, and it was designed for how that code base, that adopting code base already is, is these tools are very hard to adopt. They are massive, sometimes multi-year projects outside of that organization.

And we wanted to build something that you could adopt in days to weeks and would be very easy to customize to your code base and would not require these massive wholesale changes or huge amounts of metadata. And I think we've achieved that. Yeah, I've always wondered, why couldn't constructing the build file be a part of the build? In many ways, I know it's expensive to do that every time.

So just like parts of the build that are expensive you cache it and then you redo it when things change and it sounds like you've done exactly that with pants v2 we have done exactly that the results are cached on a profile basis so the very first time you run something then dependency inference can take some time. And we are looking at ways to speed that up. I mean, like, no software system is ever done, right? It's extremely rare to declare something finished. So we are obviously always looking at ways to speed things up. But yeah, we have done exactly what you mentioned. We don't, I should mention, we don't generate the dependencies into build. We don't edit build files and then you check them in.

We do do that a little bit. So I mentioned you do still with PansyV2, you need these like little tiny build files that just sort of say, here is some code. They typically can literally be one line sometimes, just to say almost like a marker file, just to, here's some code for you to pay attention to. And we're even working on getting rid of those. So we do have a little script that generates those one time just to help you onboard.

But the dependencies really are just generated at runtime as on demand as needed and used at runtime so we don't have this problem of trying to automatically add or edit a otherwise human authored file that is then checked in like sort of this generating and checking in files is problematic in many ways especially when those files also have to take human written edits. So we just do away with all of that and the dependencies are read at, the dependency inference is at runtime, on demand, as needed, sort of lazily done, and the information is cached. So both cached in memory, so Pants V2 has this daemon that runs and caches a huge amount of state in memory. And the results of running dependency inference are also cached on disk, so they survive a daemon restart, etc.

I think that makes sense to me. My next question is going to be around, why would want to use pants v2 for a smaller code base right like usually with the smaller code base i'm not running into a ton of problems around the build like i guess do you notice like these inflection points when that people run into it's like okay my current build setup is not enough what's the smallest code base that you've seen that you think could benefit? Or is it like any code base in the world? And I should start with a better build system rather than just Python setup.py or whatever. I think the dividing line is, will this code base ever be used for more than one thing?

So if you have a, let's take the Python example. If literally all this code base will ever do is build this one distribution, and a top level setup pie is all I need. And that is, you know, this, sometimes you see this with open source projects. And the code base is going to remain relatively small, you know, say it's, oh, it's only ever going to be a few 1000 lines. And the tests, even if I run the tests from scratch every single time, it takes, you know, under five minutes, then you're probably fine. But I think two things I would look at are, am I going to be building multiple deploy multiple cloud functions, I want to be able to deploy multiple distributions or, you know, third party artifacts, I want to be able to, you know, multiple sort of data science jobs, whatever it is that you're building, if you want, if you ever think you might have more than one, now's the time to think about, okay, how do I structure the code base and what tooling allows me to do this effectively. And then the other thing to look at

is build times. If you're using compiled languages, then obviously compilation, in all cases, testing, if you start to see like, I can already see that tests are taking five minutes, 10 minutes, 15 minutes, 20 minutes, surely, you know, I want some technology that allows me to speed that up through caching through concurrency, through fine grained invalidation, namely, don't even attempt to do work that isn't necessary for the result that was asked for, then it's probably time to start thinking about tools like this. Because the earlier you adopt it, the easier it is to adopt. So if you wait until you've got a tangle of multiple setup pies in the repo, and it's unclear

how you manage them, and how you keep their dependencies synchronized so there aren't version conflicts across these different projects. Specifically with Python, this is an interesting problem. I would say with other languages, there is more, because of the compilation step in, say, JVM languages or Go, you encounter the need for a build system much, much earlier, a build system of some kind. And then you will ask yourself, well, what kind?

With Python, because you can get by for a while just running whatever, Flake 8 and PyTest and MyPy directly, and everything is all together in a single virtual env but the python tooling as mighty as it is mostly is not designed for larger code bases with multiple that deploy multiple things and have multiple different sets of internal and external dependencies. The tooling generally implicitly assumes one top-level setup Py, one top-level pyproject.toml, or however you're configuring things.

And so especially if you're using Python, let's say, for Django or Flask apps, or for data science, and your code base is growing, and you've hired a bunch of data scientists, and there's more and more code going in there, you really, with Python, you need to start thinking about, well, what tooling allows me to scale this code base? No, I think I mostly resonate with that. The first question that comes to my mind is, let's talk specifically about the deployment problem. Like if you deploy to like multiple AWS lambdas or cloud functions or whatever. The first thought that would come to my mind is I can use like separate Docker images that can let me easily produce like this container image that I can ship independently. Would you say that's not enough?

I totally get that for the build time problem, a Docker image is not going to solve anything. But how about the deployment step? So again, with deployments, I think there are two ways where a tool like this can really speed things up. One is only build the things that actually need to be redeployed. And because the tool understands dependencies, and can do change analysis, it can figure with Git. And so it natively understands how to figure out Git diffs. So you can say something like, show me all the whatever lambdas, let's say,

that are affected by changes between these two branches. And it knows and it understands. It can say, well, these files changed. And, you know, we I understand the transitive dependencies of those files. So I can see what actually needs to be deployed. And, you know, many cases, many things will not need to be redeployed, because they haven't changed. The other thing is, there's a lot of performance improvements and process improvements around building those images. for Python executable, which is a single file that embeds all of your Python code that is needed for your deployable and all of its external requirements, transitive external requirements, all bundled up into this single sort of self-executing file. This allows you to do things like if you have to deploy 50 of these you can basically have a single docker image

that diff and then on top of that you add one layer for each of these 50 and the only difference in that layer is the presence of this pex file whereas without all this typically what you would do is you have 50 docker images each one of which contains a in each one of which you have to build a virtual env which means running pip right as part of building the image and that gets slow and repetitive and you have to do it 50 times so we have a lot of ways to speed up even if you are deploying 50 different docker images we have ways lot of ways to speed up. Even if you are deploying 50 different Docker images, we have ways of speeding that up quite dramatically because, again, of things like dependency analysis,

the PEX format, and the ability to build incrementally. Yeah, I think I remember that at Dropbox, we came up with our own like par format to basically bundle up a Python binary with, I think par stood for Python archive. I'm not entirely sure, but it did something remarkably similar to solve exactly this problem. It just takes so long, especially if you have a large Python code base. I think that makes sense to me. The other thing that one might ask is with Python, you don't really have too long of a build time is what you would guess because there's nothing to build. Maybe MyPy takes some time to do some static

analysis. And of course, your tests can take forever and you don't want to rerun them. But there isn't that much of a build time that you have to think about would you say like you agree with this or there's there's some issues that like end up happening on real world code basis well that's a really good question the word builds means different things to different people and we've recently taken to using the term CI more, because I think that is clear to people what that means. But when I say build or CI, I mean it in the extended sense. So it's everything you do to go from human written source code to a verified, tested, deployable artifact. And so it's true that for Python, there is no compilation step, although arguably running MyPy is really important.

And now that I'm really in the habit of using MyPy, I will probably never not use it on Python code ever again. But so there are sort of build-ish steps for Python, such as type checking, such as running code generators like Thrift or Protobuf. And obviously, a big, big one is resolving third-party dependencies, such as running pip or poetry or whatever it is you're using. So those are all build steps. But with Python, really the big, big, big thing is testing and packaging and primarily testing. And so with Python,

you have to be even more rigorous about unit testing than you do with other languages because you don't have a compiler that is catching whole classes of bugs. So, and again, mypy and type checking does really help with that. And so when I build to me includes, build in the large sense includes running tests,

includes packaging and includes everything, all the quality control that you run typically in CI or on your desktop in order to go say, well, I've made some edits and here's the proof that these edits are good and i can merge them or deploy them i think that makes sense to me and like i certainly saw it with with the limited number of testing and the limit amount of type checking you can do with python like my buy is definitely improving on this you just need to unit test a lot to get the same amount of confidence in your own code.

And then unit tests are not cheap. The biggest question that comes to my mind is that is PantsV2 focused on Python? Because I have a TypeScript code base at my workplace and I would love to replace the TypeScript compiler with something that was slightly smarter and could tell me, you know what, you don't need to run every unit test on every change. Great question.

So when we launched Pants V2, which was two years ago, we focused on Python, and that was the initial language we launched with because you had to start somewhere. And in the sort of 10 years in between the very Scala-centric work we were doing on Pants V1 and the launch of Pants V2, something really major happened in the industry, which was that Python skyrocketed in popularity. Python went from mostly the little scripting language around the edges of your quote-unquote real code, people use Python like fancy bash, to people are building massive multi-billion dollar businesses

entirely on Python code bases. And there were a few things that drove this. One was, I would say the biggest one probably was that Python became the language of choice for data science, and we have strong support for those use cases. There was another was that Django and Flask became very popular for writing web apps. More and more people were using, there were more intricate DevOps use cases, and Python is very popular for DevOps for various good reasons. So Python became very popular for DevOps for various good reasons. So Python became super popular.

So that was the first thing we supported in Pants V2. But we've since added support for Go, Java, Scala, Kotlin, Shell. Definitely what we don't have yet is JavaScript, TypeScript. We are looking at that very closely right now because that is the very obvious next thing we want to add. Actually, if any listeners have strong opinions about what that should look like, we would love to hear from them or from you on our Slack channels, or on our GitHub discussions where we are having some lively discussions about exactly this, because the JavaScript and TypeScript ecosystem is already very rich with tools,

and we want to provide only value add. We don't want to say, here's another paradigm you have to adopt, and you've just done replacing whatever this with this, NPM with Yarn, and now you have to do this thing. And now we don't want to be another flavor of the month. We only want to do the work that uses those tools,

leverages the existing ecosystem, but adds value. I mean, this is what we do with Python. And this is one of the reasons why our Python support is very, very strong, like much stronger than any other comparable tool out there is just a lot of leaning in on the existing python tool ecosystem but orchestrating them in a way that brings rigor and speed to your builds and i haven't used the word we a lot and I just kind of want to clarify who we is here.

So there is Toolchain the company and we're working on SaaS and commercial solutions around Pants which we can talk about in a bit. But there is a very robust open source community around Pants that is not tightly held by Toolchain the company in a way that some other companies open source projects are. So we have a lot of contributors and maintainers on

Pants v2 who are not working at Toolchain but are using Pants in their own companies and their own organizations and so we have a very wide range of use cases and opinions that are brought to bear and this is very important because as i mentioned earlier we are not trying to design a system for one use case for one company or team's use case we are trying you know we are working on a system that we want adoption for over and over and over again at a wide variety of companies. And so it's very important for us to have the contributions

and the input from a wide variety of teams and companies and people. And it's very fortunate that we now do. I mean, on that note, the thing that comes to my mind is another benefit of your scalable build system like Pants or Bazel or Buck is that you don't have to learn various different commands when you are spelunking through the code base, whether it's like a Go code base

or like a Java code base or TypeScript code base. You just have to run Pants build XYZ and it can construct the appropriate artifacts for you. At least that was my experience with Bazel. Is that something that you are interested in, or is that something that Pants V2 does, kind of act as this meta layer for various other build systems, or is it much more specific and knowledgeable about languages itself?

I think your intuition is correct the idea is we want you to be able to do something like pants test or pants test you know give it a path to a directory and it understands what that means oh this directory contains python code therefore i should run pi test in this way and And oh, it also contains some JavaScript code, so I should run the JavaScript test in this way. And it basically provides a conceptual layer above all the individual tools that gives you this uniformity across frameworks, across languages.

One way to think about this is the tools are all very imperative. You have to run them with a whole set of flags and inputs, and you have to know how to use each one separately. So it's like having just the blades of a Swiss army knife with no actual Swiss Army knife. A tool like Pants will say, okay, we will encapsulate all of that complexity into a much more simple command line interface. So you can do, like I said, Pants test or Pants lint or Pants format. And it understands, oh, you asked me to format your code. I see that you have the you have black and i sort configured as formatters so i will run them and i happen to

know that formatting because formatting can change the source files i have to run them sequentially but when you ask for lint it's not changing the source files so i know that i can run them multiple linters concurrently that sort of logic of logic. And different tools have different ways of being configured or of telling you what they want to do. But PANs v2 will sort of encapsulate all that away from you. And so you get this uniform, simple command line interface that abstracts away a lot of the specifics of all these tools and lets you run simple commands. And the reason this is important is that this extra layer of indirection is partly what allows pants to apply things like

caching and invalidation and concurrency. Because what you're saying is, hey, the way to think about it is not I'm telling pants to run tests, it is I am telling pants that I want the results of tests, which is a subtle difference. But pants then has the ability to say, well, I don't actually need to run pi test on all these tests, because I have results for some of them already cached, so I will return them from cache. So that layer of indirection not only simplifies the UI, but provides the point where you can apply things like caching and concurrency. Yeah, I think every programmer wants to work with declarative tools. I think SQL is one of those things where you don't have to know how the database works. If SQL were somewhat easier, that dream

would be fulfilled. But I think we're all getting there. I guess the next question that I have is, what benefit do I get by using the tool chain like SaaS product versus Pants V2? When I think about build systems, I think about local development, I think about CI, why would I want to use like the sas product it's a great question so pants does a huge amount of heavy lifting but in the end it is restricted to the resources on the machine on which it's running so when i talk about cash i'm talking about the local cash on that machine when I talk about cache, I'm talking about the local cache on that machine. When I talk about concurrency, I'm talking about using the cores on your machine. So maybe your CI machine has four cores and your laptop has eight cores. So that's the amount of concurrency you get, which is not nothing at all, which is great. But, you know, as I mentioned, I worked

at Google for many years, and then at other companies where distributed systems were a thing like I come from a distributed systems background. And it really here is a problem of a piece of work taking a long time because of single machine resource constraints. The obvious answer here is distributed, distribute the work, use a distributed system. And so that's what toolchain offers essentially you configure pants to point to the toolchain system and which is uh currently sas and we will have some news soon about some on-prem solutions. And now the cache that I mentioned is not just did this test run with these exact inputs before on my machine by me while I was iterating,

but has anyone in my organization or any CI run this test before with these exact inputs? So imagine a very common situation where you come in in the morning and you pull all the changes that have happened since you last pulled. Those pulls, those changes presumably passed CI, right? And the CI populated the cache. So now when I run tests, I can get cache hits from the CI machine. Now, pretty much, yeah. And then with concurrency, again, so let's say, you know, post cache, there are still 200 tests that need to be run. I could run them eight at a time on my machine, or the CI machine could run them, you know, whatever, say four at a time on four cores, or I could run 50 or 100 at a time

on a cluster of machines. So that's where, again, as your code base gets bigger and bigger, that's where the real, some massive, massive speed ups come in. The other aspects of the, and I should mention that the remote execution that i just mentioned is uh something we are about to launch it is not available today the remote caching is the uh other aspects are things like observability so with uh when you run builds on your laptop or CI, they're ephemeral, right? Like the output gets lost in the scroll back. And it's just a wall of text that gets lost. With toolchain, all of that information is captured and stored in structured form. So you now have the ability to see past builds and see build behavior over time and drill desk search

builds and drill down into individual builds and see well how often does this test fail and you know when did this get slow all this kind of information and so you get this more enterprise level observability into a very core piece of developer productivity which is the iteration time like the time it takes to run tests and build deployables and parcel the quality control checks so that you can merge and deploy code directly relates to time to release. It directly relates to some of the core metrics of developer productivity. How long is it going to take to get this thing out the door? And so having the ability to both speed that up dramatically through distributing the work and having observability into, you know, what work is going on.

That is what tool chain provides on top of the already, if I may say, pretty robust open source offering. So, yeah, that's kind of it. The, you know, Pants on its own gives you a lot of advantages, but plugging, but it runs standalone, plugging it into a larger distributed system really unleashes the full power of pants as a client to that system. No, I think what I'm seeing is this interesting, like convergence, like there's like several companies trying to do this for Bazel, like BuildBuddy and Engflow.

And it really sounds like the build system of the future, like 10 years from now, no one will really be developing on their local machines anymore. There's GitHub Codespaces on one side, it's like you're doing all your development remotely.

I've always found it somewhat odd that development that happens locally and whatever scripts you need to run to provision your CI machine to run the same set of tests are so different sometimes that you can never tell why something's passing locally and failing in CI or vice versa. And there really should just be this one execution layer that can say, you know what, I'm going to build at a certain commit or run at a certain commit. And that's shared between the local user and the CI user. And your CI script is something as simple as pants build slash slash dot dot dot, and it builds the whole code base for you. So yeah, I certainly feel like the industry is moving in that direction. I'm curious whether you think that's the same.

Do you have an even stronger vision of how folks will be developing 10 years from now? What do you think it's going to look like? Oh, no, I think you're absolutely right. I think if anything, you're underselling it. I think this is how all development should be and will be in the future for multiple reasons one is performance two is the problem of different platforms and so today big thorny problem is i want to you know i want to i'm developing on my macbook but the production so i'm running when i run tests locally and when i run anything locally it's running on my MacBook, but that's not our deployable.

Typically, your deploy platform is some flavor of Linux. So with the distributed system approach, you can run the work in containers that exactly match your production environment. So you don't even have to care about can this run on, you know, will my test pass on Mac OS? Do I need CI that runs on Mac OS just to make sure that developers can pass tests on Mac OS and that is somehow correlated with success on the production environment? You can cut away a whole suede of those problems, which today, frankly, I had mentioned earlier,

you can get cache hits on your desktop from remote, from CI populating the cache. That is hampered by differences in platform. That is hampered by other differences in local setup that we are working to mitigate. But imagine a world in which logic is not, you know, build logic is not actually running on your MacBook. Or if it is, it's running in a container that is exactly matches the container that you're targeting. So it cuts away a whole suite of problems around platform differences and allows you to focus on just the platform you're actually. So it cuts away a whole suede of problems around platform differences and allows you to focus on just the platform you're actually going to deploy to. And the

of you know, and yeah, and just the the speed and the performance of being able to work and deploy and the visibility that it gives you into the productivity and the operational work of your development team, I really think this absolutely is the future there is something very strange about how in the last 15 years or so so many business functions have had the distributed systems treatment applied to them right like every function is now that there are these massive valuable companies providing systems that support sales and systems that support marketing and systems that support HR and systems that support operations and systems that support product management and systems that support every business function. And there need to be more of these that support engineering as a business function and so i absolutely think the idea that i need a really powerful laptop so that my running tests can take 30 minutes instead of 40 minutes when in reality it should take three minutes

is that's not the future right the future is to as it has been for so many other systems to the way the laptop is that i can take anywhere um is um particularly in these work from home times is a work from anywhere times is just a portal into the system that is doing the actual work yeah and there's all these improvements across the stack, right? When I see companies like Vercel, they're like, what if you use Next.js? We provide the best developer platform for that, and we want to provide caching.

Then there's like the lower level systems with build systems, of course, like Pants and Bazel and all that. And at each layer, we're kind of trying to abstract the problem out. So to me, it still feels like there is a lot of innovation to be done. And I'm also going to be really curious to know, you know, if there's going to be like a few winners of this space or if it's going to be pretty broken up and like everyone's using different tools.

It's going to be fascinating like either way yeah that's really hard to know uh i think one thing you mentioned that i think is really important is you said your ci should be as simple as just pants build colon colon or whatever like that's our syntax would be sort of pants test l, lint, or whatever. I think that's really important. So today, one of the big problems with CI, and CI is still growing, right? That whole market is still growing as more and more teams realize the value and importance of automated,

very aggressive automated quality control. But configuring CI is really, really complicated. Every CI provider have their own configuration language. And you have to reason about caching, and you have to manually construct cache keys to the extent that caching is even possible or useful. You have to, there's just a lot of figuring out how to configure and set up ci and even then it's just doing the naive thing so there are a couple of interesting companies dagger and earthly or interesting technologies around simplifying that but But again, you still have to manually, so they are providing a, I would say,

better config and more uniform config language that allows you to, for example, run build steps in containers. And that's not nothing at all. But you're still manually creating a lot of configuration to run these very coarse-grained, large-scale, long-running build steps. I think the future is something like my entire CI config post-cloning the repo is basically pants build, colon, colon. Because the system does the configuration for you it figures out what that

means in a very fast very fine-grained way and does not require you to manually decide on workflows and steps and jobs and how they all fit together and if i want to speed this thing up then i have to manually partition the work somehow and write extra config to implement that partitioning. That is the future, I think, is rather than there's the CI layer, say, which would be the CI provider, proprietary config or Theo dagger. And then underneath that there is the build tool, which would be basil or pants v2 or whatever it is you're using, could still be make for many companies today or Maven or

Gradle or whatever. I really think the future is the integration of those two layers in the same way that I referenced much, much earlier in our conversation, how one thing that stood out to me at Google was that they had the insight to integrate the version control layer and the build tool to provide really effective functionality there. I think the build tool being the thing that knows about your dependencies can take over many of the jobs of the CI configuration layer in a really smart, really fast way. And so I think that is the future where essentially more and more of how do I set up and configure and run CI is delegated to the thing that knows

about your dependencies and knows about caching and knows about concurrency and is able to make smarter decisions than you can in a YAML config file. Yeah, I'm excited for the time that me as like a platform engineer has to spend less than 5% of my time thinking about CI and CD and I can focus on other things like improving our data models rather than mucking with YAML and Terraform configs. Well, yeah. Today you have to, we're still a little bit in that state

because we are engineers and because the tools that we use are themselves made out of software, there's a strong impulse to tinker and there's a strong impulse to say, well, I want to solve this problem myself or I want to hack on it or I should be able to hack on it. but we do deserve more tooling that requires less hacking and more things and paradigms that have tested and have you know survived a lot of tire kicking and do will we always need to hack on them a little bit yes absolutely because of the nature of what we do. I think there's a lot of interesting things still to happen in this space. Yeah, I think we should end on that happy note as we go back to our day jobs, mucking with YAML. Well, thanks so much for being a guest.

I think this was a great conversation and I hope to have you again for the show sometime. Would love that. Thanks for having me. It was fascinating.