CppCast - Bazel

Episode Date: October 16, 2020

Rob and Jason are joined by Lukács Berki and Julio Merino from Google's Bazel team. They discuss CppCon trip reports, the cpp subreddit and a video on C++ 20 Concepts. Then Lukács and Julio talk all about the advantages of Bazel and some of the key features of Google's open source build tool. News My favorite way to TDD CppCon Trip Reports r/cpp status update Concepts in C++20 Links Bazel Bazel on GitHub The final boss: Bazel's own JNI code

Episode 269 of CppCast with guest Lukasz Berki and Julio Moreno recorded October 14th, 2020. In this episode we talk about CppCon trip reports and concepts. Then we talk to Lukasz and Julio from Google. Lukasz and Julio tell us all about Bazel. Welcome to episode 269 of CppCast, the first podcast for C++ developers by C++ developers. I'm your host, Rob Irving, joined by my co-host, Jason Turner. Jason, how are you doing today? I'm all right, Rob. How are you doing? Doing okay. Any news you wanted to share? I don't think so. Anything going on? No? Sometimes when you say that, you're prompting me to say something, and I'm like, what did I forget? What did I forget?

No, I don't think so. Yeah. Just, uh, just another week. Uh, nothing going on. Yeah. Okay. Yeah. All right. Well,

at the top of it, I'd like to read a piece of feedback. Uh, this week, uh, we got an email about a recent show we did with, uh, Ola Gerbave.

He said, uh, this is from jerk room saying, Hey guys, love listening to your episode about unit testing. I've been doing TDD at my company since 2013. I'm glad to hear other proponents of unit testing. I agree with just about everything Oleg said,

except we differ about the best use of mocking. If you want the best take I've seen on the subject, you should have Justin Searles on the show. You can check him out at blog.testdouble.com. And then he also pointed us to this particular blog post saying it's an example of him demonstrating good mocking practices. So I will put that in the show notes. And yeah,

we have not done an episode specifically on mocking. We've certainly talked about it with Oleg a bit, but maybe that is something we'll follow up on. I think when we interviewed, we know two people that have made mockingocking Frameworks that we have interviewed, Peter Bindels and Bjorn Faller. And I know we've talked about mocking with those episodes, but I also agree that I feel like it's not really a topic that I really get.

Yeah. Something we can definitely do sometime in the future. Well, we'd love to hear your thoughts about the show. You can always reach out to us on Facebook, Twitter, or email us at feedback at speakass.com. And don't forget to leave us a review on iTunes or subscribe to us on YouTube. Joining us today are Lukasz Berkey and Julio Marino. Lukasz, after a number of diverse jobs with computers, including Linux system administration and GPS navigation devices, Lukasz has been working at Google and the Bazel team since 2009.

In this decade, he has worked on almost every part of it. Integration with remote execution, Android builds, its internal dependency management framework, internal telemetry, support for various programming languages, and a lot more. He played a significant role in creating Bazel from Blaze, its Google internal predecessor. Lukasz, welcome to the show. Welcome. It's nice to be here. Yeah.

And Julio has been interested in build systems since the early 2000s. Such interest started with frustrations over autoconf and automake and followed with amazements by what NetBSD and package source could achieve with a monorepo and a build system specially tailored to it. These adventures were continued with his own now-dead build system, BuildTool, his own testing framework, and his latter joining the Bazel team in 2016. Julio has been at Google for 11-plus years and is about to embark on some new challenges. Julio, welcome to the show.

Welcome. Thanks for having me. Well, both of you have spent, I mean, effectively entire careers thinking about and working on build systems. I'm not sure what to say about that. I mean, I'm fascinated by it. I don't know. I've never built my own build system personally, although it does seem like it's a rite of passage for some people. They feel like, well, I have to prove that I understand how this all works at some point. I haven't yet. Actually, I haven't built my own. When I started, it was already there. Okay.

I'm afraid it may be a rite of passage, but then I haven't passed that test yet. I certainly haven't either. I've said, you know what, it's good enough that the other tools do what I need them to do. At some point, I tried to write my own, and then I realized, enough that the other tools do what I need them to do. That's the whole point. I tried to write my own, and then I realized, hey, the other tools are ready to do a good job at it, more than I would ever do, so let's stop my craziness.

So don't go that route. Try to not do that. How much time did you invest in BuildTool before abandoning it? I was looking at the webpage I posted yesterday. It's still online. I think it's about five years. And I was looking at the manual I wrote for it, and I was like, wow. I had a lot of time when I was looking at the webpage I posted yesterday. It's still online. I think it's about five years. And I was looking at the manual I wrote for it, and I was like, wow.

I had a lot of time when I was a student. It wouldn't have happened today. Yeah, but it had to go. I mean, the code was a mess. And Autoconf, I had to make, actually, did a better job than I did at the time. I know this is an absolutely terribly biased opinion, and it's certainly not 100% true, I know this is an absolutely terribly biased opinion,

and it's certainly not 100% true, but if I see that a project is hosted on SourceForge, my immediate response is, oh, it's probably a dead project. I'm not sure if it's warranted, but I have the same intuition. Well, I know it's not 100% warranteded because a project that I know is in active development, CBP Check, is still hosted on SourceForge. Okay, I should have said Pijadist, not intuition.

Right. Okay, well, we got a couple news articles to discuss. Feel free to comment on these, both of you, and then we'll start talking more about Bazel, okay? Alright, so this first one we have, we obviously had CBCon recently. We talked a lot about the conference, but there have been a number of trip reports from various attendees. And this is a little collection of a few of them on the

CppCon website. And also just to note that, you know, a lot of the videos are still going up for CppCon. So definitely keep an eye on that. But yeah, several good trip reports here. Was there anything you might want to call out from any of these, Jason? Well, it's funny because I clicked on this link, obviously, because I'm reading the news for the day. And then I clicked on Connors, which is the first trip report listed. And I got so sucked into reading Connors that I totally forgot that there was, you know, several others that I should have done as well. But Connors is very well written

and covers an awful lot of topics and goes back historically. And it's like, well, if you don't like this version of the talk, check out this other version of the talk from three years ago or whatever. Yeah, he's really good about that. Like he points out, you know, if this video is not available yet, then you might be able to go watch it on some other platform. I think he did that with Sai's talk on building an intuition for composition. Yeah. So very, very good trip report.

He also referenced one of your recent polls, Jason. Yeah, I asked people if they were going to be working while attending cbp con or not and it seems yeah the a lot of people were were trying to do a normal day job while also virtually attending a conference which you know had some impact i guess now people experienced it yeah certainly not ideal but it's it's harder to get away when you're not physically at the conference yeah did either of you uh virtually attend cbp con this year i wasn't even away with deal, but it's harder to get away when you're not physically at the conference. Did either of you virtually attend CppCon this year? It wasn't even a way it was happening.

Oh, I did not. I attended a conference recently that was remote only and didn't really get much value out of it, unfortunately. So this CppCon and upcoming meeting C++ and what C++ on c so a lot of the c++ conferences this year are using remo right that's what it's called yeah right and so it kind of

gives you part of that that hallway track quote unquote feel where you can interact with people and chat um in between sessions which had helped a lot in making it feel more like a real conference. Yeah, which online conference did you attend recently, Julio? I think I dropped by, one of them was Microsoft Build. Okay. And then I think we had

some other internal stuff at Google for mobile development. But they were, in one case, they were like live sessions, which went okay. In our case, internally, we're like pre-recorded sessions, which is like at the point, I can watch the videos anytime. There was a feeling of confidence, unfortunately.

Okay. Next thing we have is a post on r slash cpp. This is r cpp status update. We briefly talked last week about this video that was posted from Jean-Huidh Manid, and it caused quite a lot of chaos amongst the RCPP community. So it's certainly worth knowing about. One of the moderators was temporarily removed but has since been reinstated, and it looks like they're going to try to apply some new rules

with posting content here. Right, Jason? Yeah, and put some things in place to make sure that they can respond to moderation issues more quickly. Yeah. I don't know. Is there anything else worth saying about all this right now uh just if you have an interest in the reddit c++ community it's a good conversation to read i think yeah

okay and then this last one uh jason you posted this video about concepts and c++ 20 from the copper spice youtube channel yeah we never really call attention to their channel, but they do good work and regularly produce videos. So that's Barbara and Ansel, who we've had on before, and regular conference speakers, and they just posted an episode on concepts. Yeah, and I thought it was a pretty good tutorial of concepts in C++20. They highlighted a couple of things that I haven't seen talked about too much elsewhere, like what they think are some of the better use cases for concepts.

Okay. Well, Julio and Lukasz, let's get started talking about Bazel. Could one of you give us kind of an overview about Bazel as a build system and maybe tell us why Google decided to build its own build system instead of using something existing? I will take this question because I have a bit longer tenure on this project itself than Julio. Although when I joined,

it has already been the build system at Google. And the story here is that at the beginning, Google used Make like almost everyone else at the time. But then as our code base grew, we started hitting the limitations of Make. So then we had a system that used Python scripts in files called Build that generated Make files. And then that was good enough for a while. And then that also started hitting its limitations and that's where when Bazel was started and you can see a few of the of things you can see traces of

why of the bust of Bazel in its steel for example that's why the build files are python-esque and that's why some of the syntax is very similar to that of Make. For example, the $, or that's what we call what Bazel builds targets. Go ahead, yeah. So in short, I think the main motivation was scalability. Okay, and what makes Bazel more scalable compared to something like Make? Compared to Make, I think that the main... There are two differences. One is that Bazel imposes more structure.

Because in Makefiles, you can do pretty much anything. You can call out the shell at any time, and the semantics are not very amenable to parallel execution, and they are not very amenable to understanding one Makefile in a build or parts of a makefile separately. In contrast, Bazel has a lot of structure and the last part of, I think the last part of when converting from make to Bazel was to import the structure on the makefiles.

One thing I would add is that in make-based projects it's very common to have recursive make invocations, right? So you have different makefiles per directory and then, yeah, if you that in Make-based projects, it's very common to have recursive Make invocations, right? So you have different Makefiles per directory, and then, yeah, if you're in a subdirectory, building from there is very fast because Make only has to load that single Makefile, and that's it. But then Make doesn't have a global view of the project, right? And then you're trying to track across multiple dependencies. So at Google, we had a gigantic Makefile that covered everything. And then, yeah, you start hitting points where your code base is too big,

the makefile becomes too big, and just parsing it takes, I think, minutes, two even, just to run a making location. So Bazel does the same whole project view, but it's smarter in knowing when to read parts of the build graph again. It doesn't have to reload everything in every single location, right? And that makes it more scalable. Yes. This is why that makes it more scalable. Yes. This is why more structure is more useful

because if you have built up barriers in the data flow within the tool, you can determine that if some input changes, you can determine exactly what needs to be rebuilt and what can stay as it is. So I think maybe for a point of clarity for me, would we say that Bazel is a build system

itself? Like, is it directly invoking GCC? Or is it a build generator? Like you were started, if I understand correctly, said original versions of those tools actually generated make files. So does Bazel directly invoke GCC? Or does it generate Ninja files or something else? That's a good question. Bazel currently directly invokes GCC. Okay. Or the other tools. The reason why I said currently is because every once in a while,

the idea comes up to make Bazel generate Ninja files, and it wouldn't be that hard to implement. It's just we haven't found a very compelling use case for it. Okay. And also, we have a pretty nice execution engine, so we think that we can do better than Ninja. Okay. So you just said Bazel is directly invoking GCC or other

build tools, but what other languages can Bazel compile? It's more than just C++, right? A lot. I think that, Julio, please correct me if I'm wrong, I would say that the main two languages are C++ and Java, and this is what drove the development. Python is a close third,

and over time, a number of other languages grew into Bazel. We have Golang, we have Haskell, we have Dart, we have Scala, we have Kotlin. Am I missing something important? Fortran? I mean, I'm missing the Apple... Internally, yes. Internally, we do Fortran. You have all the Apple Objective-Cs, whatever you want, really.

I mean, that's one of the nice things of Bazel, right? It's the couple from the languages. You can build anything you want with Bazel, really. Just like you can with Make. It's no couple from the languages. You can build anything you want with Bazel, really. Just like you can with Make. It's no different there. But one thing that's probably good to clarify is that at Google internally, we've always had some blessed languages

we use globally. So things like Haskell, for example, is not widespread inside Google. So our functionality to do Haskell is not great. But that's where our community, the open source community can really help us because other teams outside of Google would really want to use these languages and can very easily extend Bazel

to support those use cases where we haven't really put that investment in. So in all the languages that you listed, I heard like 10 compiled languages and I thought, I swore you said Python. Did you say Python? Why would Python be a target? I'm so confused.

That's a very good question. The reason why we started supporting Python is that because, sure, you don't compile Python, but you need to execute it. And what we originally did was a very simple thing, is that to assemble a tree of all the Python files a binary needs, and a little stub wrapper script to execute the Python interpreter in the exact way that the code needs to be executed. But then what happened after is that a lot of functionality grew around it. For example, internally we can package Python files into a single binary. We call them power files.

It's very similar to Pax. We can run tests. And since running tests has first-class support in Bazel, it's important to also be able to run tests in Python. And over time, we grew linters. And I think we also have a type checker, at least internally. And native dependencies. Native dependencies, too. Yes. Right, that's the critical thing, right? You might want

your Python scripts to depend on C++ libraries or whatever you want, right? Bazel lets you couple those things very easily. At least inside Google. I'm not sure if the external Python rules we have currently support that. It's a tricky aspect, and I'm not sure. It's not just that it's a tricky aspect,

it's also people have many different use cases. So the reason why life is more simple within Google is that our code base is pretty uniform for a code base of this size, and if you want to use native code from Python, you pretty much have

one best way to do it, and you can support it. But that's not how the open source committee works, so we can't impose one single thing on everyone. So I've heard you say a couple times, internally versus the public. Is there two different repos, one that's managed internally and one that's the public repo, or

is there something I'm missing here? Good question. So the way this works is that the code base of Bazel and its internal incarnation Blaze is almost the same. We have some code internally that we don't open source, mostly because

it doesn't make sense. This is mostly about interfacing with internal tools or programming languages that exist only within Google. And the way this works, that the... Wait a minute. You just said programming languages that exist only within Google,

and that's the first I've heard anyone say anything like that before. I hope I... No, I'm pretty sure I didn't give away a big secret. We do have that, although my personal opinion as a person who supports develop as a Google is that they didn't. So recently, I think that there aren't anything new

in the last few years. None of that would affect what we open source, right? None of that would affect what we open source because that would just be like extensions that are Starlark and not native inside the Bazel code tree. They still have some appendages in the Java code, unfortunately. We are working on removing them, but we are not done yet.

We did remove support for two of these in the summer. That was a great accomplishment. But for example, our internal parallel execution engine or our internal telemetry, it doesn't make any sense to open source those, so we don't. Okay. And then unfortunately, as part of the open sourcing project process, we kind of forked a few modules because it was easier to do it that way than having to abstract things out. And obviously suffering from trying to clean those up, right? But those are where the differences remain. Either in some of the rules that might be slightly different, like Lukasz was saying,

have special ways of building Python code internally. So the rules might differ in how they look externally and internally, and then how Bazel interacts with our internal production systems. But for the most part, everything else is shared. It's a unique code tree, code base. And yeah, when you see outside, it's what we use inside pretty much. Very cool. Okay.

Let's talk a little bit more about what Bazel does and does not support. Is it a cross-platform built system? I know you make references to both Mac and Linux. It supports many things, I would say. I mean, Bazel, as far as I know, can run on a Linux machine, Mac OS, Windows. It is an unofficial free BSD port. I think I've seen it can run on a Linux machine, macOS, Windows.

It is an unofficial FlippySD port. I think I've seen it also run on NetBSD. Actually, I know it runs because I tried to run it, but it had some rough edges, but that was like five years ago. So at this point, it probably runs better. We've seen a lot of fixes there. I mean, it's just a Java app, right? It has some native code, but if you can port that amount of native code,

which is very limited, it should be a Java app, right? It has some native code, but if you can port that amount of native code, which is very limited, it should be able to run pretty much anywhere. As for what it can target, I know Lukasz had a list of things he might want to add there, so I'll let him answer. My curious... Please go ahead. I will interrupt you, slash add things to you,

to what you say. I mean, targeting... We can target... I mean, internally or externally, people have used Bazel to build embedded code, server code, mobile code. So that would include targeting Intel platforms, ARM, I don't know, whatever, really. It's a matter of defining how you invoke the compiler, really. Bazel doesn't have a say on what you want to do with what you're building.

It's just whatever rules you use. Like if you're in the C rules, for the C or C++ rules, you have a tool chain definition that says how your compiler works, how to invoke the compiler, and then you can target that. Other rules like Apple rules, for example, to build mobile apps

are a bit more complicated because they have to care about different iOS levels, different watch OS, TV OS, whatever, and combine them in a way that generates you the right APK, sorry, IPA in this case, the right combinations of binaries. But again, it all depends on the rules. Bazel itself doesn't care.

No, I thought from your words, I thought that you know something that maybe I don't, but I essentially agree with what you said. It's mostly Linux and OS X and Windows plus the mobile operating systems. But it does sound like you at least implied that it does have support for cross-compilation, which historically can be a problem for the build system to find the dependencies, the cross-compiled dependencies. That's a big part of what we are doing. I mean, if you think about it, if you build for an iOS phone or iOS device or for Android device, you can't help but to do cross-compilation. Right.

And we put a lot of thought into making cross-compilation work. And the way we do it is by either not using the headers and libraries that are installed on the system or making it very clear when we do. I mean, the heritage of Bazel is that it comes from a monorepo where all code that used to build and all dependencies are checked in. And this comes very handy in this case because the fact that we try not to reach out to the system if we can help is very ingrained into what we are doing. Right, so in the monorepo scenario, you don't really have the question of where do my cross-compiled

dependencies live because you are compiling all of your dependencies at that moment. Correct. A very good example is what happened at Google 2012-ish, I believe. So we have this distributed execution system, and we used to rely on the C++ compiler installed on the workstations of developers and installed on each node of this execution system. But then people sometimes wanted to have a new version of the C++ compiler and then whenever a new version came out we had to do this elaborate dance because before we were able to flip the default, we had to make sure that it's installed on the registration of every developer, we had to make sure that it's

installed on every node of the remote execution system, and only after that could we flip the flag. So we decided that it's way too much procedure, and we decided to therefore just check in the new C++ compiler. And at that point, migrating to the new C++ compiler was just as simple as pointing Blaze to the new version. But just to clarify this, how we do things at Google, and we try to tell people this is the model they should follow, like checking your dependencies, checking your tool chains into the tree. But part of the things that had to go into making basel work for the process community was not forcing that model right people in basel want to use that install tool chain so we have to add features to basel to make sure you could do that and now basically for

example when you run it by default it will try to find what your system compiler is and generate kind of this configuration logic that you need to tell basel where the compiler lives it generates it automatically and you can use it from there. But if you want, and that's what we recommend you to do, if you have a big project, is to check in your dependencies so that Bazel can have a global view of how to build your project. And you are, just to clarify, including your compiler in that list of dependencies. Yeah, that would, GitHub would be sad if every project had their compiler checked in. Yes, I do realize that there's an imperilous mismatch between what we think is best practice and where the world is.

And what GitHub can live with, probably. But I'm going to cover here external dependencies, right? When you say check in the tree, you can literally do that, right? You can git add your compiler and add it, but I think you could possibly tell Bazel, hey, this lives in a separate repository that we have defined here that's shared across projects, and you

could pull that in transparently. Yeah, I think you could use git lfs, large file storage. I think you could use that. That's a different way too, but Bazel has its own external workspace handling process. But Lukas knows much more about this than I do. But instead, I will tell you another alternative to accomplish this, which is that you can tell Bazel to run every compilation step

in a particular Docker container, which has no set of tools installed. And then you spare yourself the step of telling every one of your developers to install that particular version of the C++ compiler or Java compiler or what have you. Or even worse, tracking down issues that come from version mismatches. It's pretty neat. So I apologize.

I may be misunderstanding part of this, but it's been my understanding for as long as I've been doing development that you never check into source control something that can be generated. You never check in generated files. So I feel like I'm hearing a little bit of a dissonance because it does sound like and this is off the track, I guess, at this point, but it does sound like you're kind of saying, no, we check in all of the things. And I don't know, does that include I don't know, am I include? I don't know.

Am I making sense? Is my question, my thought process sound weird? I think it makes sense. I mean, you're saying, when you said about good practice, I agree completely, you shouldn't check in generated files. Okay.

I agree. I don't know if Luke has this, but when we try to do, usually it's checking the sources, not the build binaries. Okay, so the only binaries you're checking in then are the system dependencies, the compiler, that kind of thing. Yeah, and even then we have the sources too, but for bootstrapping reasons, you will need to check in the binaries at some point. Only for the compiler and then everything else that we can build from source, we build from source. And then we rely on a remote cache to make sure not every single engineer

is building the compiler every day for everything they have to do because it would take too long, right? And that's highly cacheable because it's the same version for weeks, essentially. Do you use the same compiler

for an entire week at a time? Hopefully. I haven't checked. Maybe I'm wrong, but that's what they put in the connection. You just mentioned remote caching, and that's one of the I haven't checked. Maybe I'm wrong. That's what they put the connection in.

You just mentioned remote caching, and that's one of the big features of Bazel, right? The ability to cache binaries so that when you hit build, you're not actually rebuilding that much. Is that right? I would not go as far as to say it's a feature of Bazel, but it's something that Bazel makes possible. Because what Bazel can tell you is which files exactly your compilation steps need. And then if you know that, you know what to include in your cache key. And Bazel has an interface for remote execution systems that we use internally to run distributed builds,

and we do heavily use this. I think that the cache hit rates nowadays are around 99%, 99.5%. So if that stopped working, everything would fall apart pretty quickly. I think you mentioned here, you mentioned the cache key includes the file information of the inputs of an action. But something that's very important to mention is that the cache key also includes the command

that you have to run. So that's very different from make, right? In the make world, you have a rule that involves the compiler and you modify the make file to update the rule. Make will not know that and it will not reveal your output because it will have no idea that anything has changed. Whereas Bazel knows that the command that you had to type to be that output changed, and it will rebuild it. And that ensures that we are always up to date with changes to the build files. That's actually pretty common to the build rules and enforces this correctness aspect we like to highlight in Bazel.

It sounds similar to how ccache works. It hashes the command line effectively and then the file inputs and the pre-processed things too. Okay. Does it suffer from the same problem that other caching systems I'm familiar with, like ccache does, where if the directory structure changes slightly, like I'm building in Jason's directory, but Rob is building in Rob's directory, then it sees those as two different artifacts, and then it has a hard time sharing the cache?

I think we want to... Go ahead. I was about to say that I don't think we suffer from the same problem, although I'm not very familiar with C cache. Okay. So I'm not ever... I mean, I'm sure we don't, right?

We can reuse build artifacts in different client machines and different environments. So I think, I don't know if that's exactly, build artifacts in different client machines and different environments. So I think, I don't know if that's exactly, but I'm pretty sure we go to great extents to use relative paths to the workspace so that the command line remains stable across users and directories. Now my brain is coming online. Do I remember correctly that Ccache works by caching artifacts built by one developer for the other developer. That is the main goal of Ccache, but it has been getting better over the last few years to be able to more correctly handle relative paths so you can do a shared cache of some sort. Okay, so this is why, so in that case, we don't suffer from this failure mode because we have a,

we don't, so we don't do cross-user caching. Bazel only does caching on the same workstation, and the rest of the caching Bazel delegates to the remote execution system. Right, but an action that can run a compiler invocation, when I say action, I mean a compiler invocation, for example, right, that can run on a remote machine, it can also run in your local machine, and they will have the same command line because of these relative paths. It doesn't matter where the remote machine runs

in which directory it runs the action because it doesn't matter. Correct. Although the technique is correct, but if you don't cache actions between different developers, then you are not open to small differences between their setups.

For example, the aforementioned compiler version issue. Or attacks, right? You don't want to do that. I think we just unpopulate the remote cache with local invocations, right? We just rely on CI to do that for us. And then we trust that the CI machines are well managed and secure, and then you will not have interference between users. So if I build it on my system, and it's some artifact that doesn't already exist, I don't push that up to a shared location.

I'm only able to pull down something that was already from a known good location. Did I say that right? That's right, but the thing is, to clarify, that's only if you were to use remote caching. But the thing is, we pretty much always use remote execution as well. So when you said, I build this on my workstation, it really didn't happen in your workstation. It happened in a real machine that's trusted. The machine knows exactly what the action looks like

and generates the output for you and can download it later if you want it. So how does that work for me? I'm just sitting here at my laptop right now and I want to build. Can I build locally still? It will feel like it does build locally but it's not if you have enabled these features right they are not enabled by default because that's okay some setup of a remote execution engine of which there are multiple open source implementations right we don't have our own people have done their own so you can just either use some of the shelf if you want to deploy yourself or use some companies that offer that. But yeah, it will feel as if you were building locally. It's just the actions happening in the machine.

Okay. Interesting. Yeah. Can we talk a little bit about Bazel's support for libraries? I read a little bit about how it kind of knows what a library is versus what make as kind of the alternative. And also, what sort of support Bazel might have for package manager? I'm curious about.

The first question is easier. So, that's a bit of a historical fortunate accident or the person who invented this would probably disagree disagree with this assessment that we differentiate libraries and binaries pretty well so we have usually when you write language support for basel you write at least two kinds of rules language

underscore library and language underscore binary and you probably also want to add language underscore test okay Hmm, okay. So there's no confusion. Does this answer the first question? I think so. So you kind of can treat multiple libraries as dependencies, and Bazel is able to figure all that out. Correct.

I think that's a huge benefit of Bazel, right? It has this semantical information about what you're building. Its rule knows if it's a binary, it's a library, it's a test, it's a documentation artifact, it's a zip file, whatever, right? This can often call it in the build rules and your build files are higher level than what they would be in Make because Make just kind of tracks file dependencies and you can add these foreign targets in between to aggregate things, but Make doesn't know, right? It's harder to express this kind of concept. Yes, and the other thing is that Bazel will transitively traverse your dependencies and build them,

if you need them, and only build those. In a sense, this is similar to Make. So with package managers, you know, in the C++ community, we've had a lot of work over the past few years with Conan and VC package. Does Bazel work with either of those? Not to the best of my knowledge. So the reason why this is a bit of a sensitive subject is that because given Bazel's monorepo heritage, we haven't emphasized package manager support yet. We do support some things because we know that there is a need. I mean, Maven is a good example, but that's Java and Assyria plus.

But the issue I personally see is that a lot of these package managers work by having a cache in your home directory somewhere and keeping state per user. And what Bazel prefers to do is to keep state per project so that projects don't contaminate each other. Okay. And for some package managers, these two are easy to reconcile. For some package managers, it's not. I'm curious what you were asking. Your question was, how does Bizzit interact with these package managers? And I would ask, what does interact mean? What do you have in mind? Yeah, that's what I was just kind of thinking along the lines of that. So with Conan specifically, because that's what I'm most familiar with, I can just say Conan, install this project with this compiler if I want to.

So there's a couple of different ways of going about this. With CMake, I can run a cmake script that just automatically interacts with conan and then cmake just has available to it the libraries that were just installed for that project for conan but i can see if you did conan more in a manual perspective where you said oh conan please package manager install these four things then the only question remaining is is there a way inside of basil to say look in these four things, then the only question remaining is, is there a way inside of Bazel to say, look in these four directories for these four dependencies because I've already installed them? So I know Conan, I mean, excuse me, Bazel sounds like it, I mean, it prefers the monorepo thing, but it can work with external dependencies if you need it to, right? That's a

very good assessment of the situation. Okay. So you would just need someone who has done the legwork to say this is the best way to tell Bazel how to find my Conan install dependencies. Or else ask Conan to install the dependencies in some way that switchable to Bazel.

Right. Yeah. It does have it in its own hashed directory that's 14 levels deep in your home directory kind of thing. Yeah, it's not trivial, but it has been done. I mean, I think the obvious example are the CUDA libraries, but then video chips, and we have to use them from wherever they are,

and basically can do that. I don't know how it works, but it's definitely possible to use external dependencies. Okay. It's not as natural as you might want yet, but it works, but it's definitely possible to use external dependencies. It's not as natural as you might want yet, but it works. It's a need for the OpenSource community, so we have to support

that in some way. What is Bazel itself written in? I think you mentioned Java and C++? You're right. Mostly Java with bits of C++. And I believe the tool to invoke the core version of Bazel, called Bazelisk, is written in Golang. And we have shell and Python snippets in some obscure places that we pretty much don't want to have.

Before you ask, I mean, this sounds crazy, but keep in mind that Blaze, I'm saying this correctly, the internal version of Bazel before Bazel existed was started around 2006, I think, 2007? 2007, I believe 2007, but maybe 2006. Definitely not 2008. The state of C++ at the time wasn't as good as it is today, and the amount of languages we could use inside Google was pretty limited. But blessed languages, as I was mentioning before. So Java was really the only reasonable choice between those to write this kind of native code. Today, I think we wouldn't choose that. But it's too late to change it. Basically, it's a gigantic piece of code.

It has a ton of logic. It supports a lot of use cases. If you're writing something, it's difficult. No, I agree. I think that when the choice of Java was made, I don't think anyone expected Blaze to handle a code base as big as it handles now. And I'm saying Blaze and not Bazel because probably Google is the place

where Blaze is put to use in the largest code base. And Java... Sorry, go ahead. I'll finish up later. Sorry? No, I was going to say that Java has caused a lot of friction, right, with users, but we have gone through a lot of efforts to hide that from our users. When you download a Bazel binary, you don't even know it's

running in Java. And we it's kind of, we've bundled the GRE with the binary that we ship, so everything's kind of, we've bundled the GRE with the binary that we ship, so everything's kind of self-contained. Like, Linux distributions don't like that, because they don't like this kind of self-packaging. But for you as a user, if you just download the binary, it will work for you

without having to worry about Java at all. And then Bazel also has this internal client-server model. It runs on your machine always, but the client is the thing that's written in C++ for super fast startup. And the server, which is a Java part, keeps running in the background for a few hours

until it decides to exit to prevent, to mitigate this cost and to maintain state across builds. So the fact that it's Java may sound crazy, but in practice, it's not a big problem. I don't think it sounds crazy that it's Java. The only thing that would make me wonder is why five different programming languages or whatever you ultimately mentioned. I think it's mostly two because the Go is an optional tool.

And I think that Python and Shell are only used if you develop Bazel itself. And that's mostly for our convenience. I wish we didn't have for example Python because it's just yet one more programming language. Shell is useful. I don't know what to say, but it's not

fiction when we work on Windows. I think we can't change the fact that it's written in Java anymore, and C++ is necessary because there are some things that Java cannot do. Yeah, that's what I wanted to ask specifically. You said there's a performance consideration, but what other reasons do you have C++ for part of it?

At that time, please carry on, Julio. No, I was going to go further back in history, as you were. I was going to say that today we have some JNI code for file system operations, for example. Okay. But Java has gotten much better at that, so it's possible we could even replace that code with Java, NIO, native calls now. I don't know though. Yes, I was about to say that the first reason why we considered CETA passes is because at the time, Java was not able to handle symbolic links. But then it turns out that, for example,

internally we use extended X4 attributes, and I don't think Java can handle that, even today. I think somebody pointed out to me two days ago that it can today. Maybe Java 12 or whatever it is, so we can't use it yet. Believe.

There's another thing which is very important for Bazel, though, which is process control. Because if you run a build step and that build step is finished, or Bazel decides that it wants to terminate it, it's important that it be able to

terminate it, and it's important to be able to control what resources it has access to. For example, I personally did the to be able to control what resources it has access to. And, for example, I personally did the part that does process control on Windows, and that just doesn't work in Java. Yeah, yeah, you need C to do those things, basically.

I hate doing process control. Me too. I'm not too much of it. But basically, you have JNI code. You have JNI code. We have JNI code that's linked into the binary directly, but we have, as Lucas was saying,

this, what he mentioned about some process management, those are separate tools that we bundle with Bazel. They are native and they actually wrap anything that we invoke.

When you run GCC, we will invoke it through this kind of process wrapper thing that we have that makes sure that everything gets cleaned up or gets killed if it gets stuck or, you know,

anything. It kind of sounds like everyone here who's had to deal with cross-platform process control, you hate it. It happens that that's an universal problem of the carrier. I think I did Windows process control

wholly over the whole laton OSX. And I think there was recently a said about some very arcane details of Unix signals that we didn't quite get right. Oh, most certainly, yeah. I've used Q process a lot many years ago and that's supposed to abstract all those things away for you but I still ended up

having to replace core pieces of it that didn't work quite how we needed them to anyhow whenever I run into memory size issues or something that you cannot really easily do in Java,

I kind of find myself wishing that we did everything in something other than Java. But then I realized that if we had written Bazel in some other language, I'd probably have the same feelings with that other language too. Yeah. Okay.

Julio, we mentioned in your bio that you're actually going to be leaving the Bazel team soon. And you wrote this recent blog post about one of the last bugs you were working on. Just to tell us a little bit about this. Oh, sure. I mean, it's very relevant to what we were just discussing. Yeah.

And it may be interesting to your readers, but no, listeners, which is we had some very obscure bug internally where someone was changing the AppSale library. If you haven't heard about AppSale, it's a core library. Yeah, we've had Titus on here and talked about it, yeah. So they were trying to do an optimization that was

very important because it was supposed to save many resources. And the only tests that failed at Google, which are many of them, the only tests that failed were the ones for Blaze on Mac. And that turned out to be because this specific version of Bazel was trying to dynamically load a couple of shared libraries, a JNI plugins, and both of these libraries depended on AppSale separately. So we ended up having two copies of AppSale in memory and control and AppSale was trying to do

some deadlock detection and have internal state tracking for logs and control was jumping between these two copies of the library and things were not working. So this sounds right. Yeah. So what I was trying to do, which sounds very easy, is let's try to unify all the native libraries into one object so that we can reuse this shared piece of code, which sounds silly because if you were using dynamic libraries for everything, this wouldn't be a problem. But because we tried to statically link many things in our rules, that's why we ended up with two separate copies of UpSing in different shared objects.

There was no common central piece of code where you could deduplicate symbols. Yeah, and this was easy enough conceptually, but in practice

it wasn't. Especially because of this friction we still have between Bazel externally and Blaze internally. And I was touching the pieces where the two interact.

Those are the most annoying ones. There's more details in the link I shared. You can read in the notes later. I'll be trying something new. I've been at Google for a long time and I'm about to start something new. I'm not going to unveil where here, but you will know soon.

Okay. Very cool. One question I had. You're looking at future adventures. Yeah very cool. One question I had is, we've talked a lot about how Bazel works great for this gigantic monorepo that you have at Google. Is it a good build system to use if you're starting from scratch with a smaller project, or is it harder to configure, more setup required than something like Make or CMake? I think at the very simplest size

it's just as easy. I think that the difficulty comes if you want to use features that Make or CMake don't have. Remote execution is a good example, because then you need to think about what tools you use on the remote nodes, you have to figure out what to do when, how to do the same thing between different developers, and so on and so forth. So it's more like if you want to use a feature, you have to pay for it. We are trying to make it easy and straightforward though. One thing I would add is if you think that you will end up using more than one language in your project at any point, then the Bazel investment will probably pay off.

And when I think more than one language, it's not too hard to get into the territory. You have your C++ code for your binary or library, and then you have some shell scripts to test that. There you go. You have two languages already. How do you automate execution of the tests for each of these pieces? Right, Bazel gives you that very easily because you can combine these two different kind of artifacts and build rules in one project You can test everything at once, but if you end up with a language specific package manager or build tool It's very hard to integrate with other stuff, right? Anyway, Laptopless, C++, Rust, whatever you want. Rust, for example, has cargo, which is amazing.

But as soon as you want to integrate any other language, you can't. Interesting point. Yes, what I would like to add is that Bazel's integrated testing support is a big boon. Because if you say Bazel test and the list of tests you want to run, those tests get run no matter which language they are written in, the results get collated in a single list and you get all sorts of neat features, for example, automatically trying a flaky test or running tests multiple times if you think a test is flaky and so on and so forth. Yeah, we like to think of Bazel as a build tool,

but really, as I've heard a lot of people in the team say, it's a test tool, right? Testing is a super important part of a project, and Bazel's abilities for testing are pretty amazing. I think, while we're running low on time, but a question that I still have is, are there any publicly facing projects from Google that are big projects that use Bazel that we are all aware of? I don't know, is it used for Chromium or something like that? I think the biggest one is TensorFlow at the moment. TensorFlow, okay.

And there's another one I'm forgetting now. Although they're going to upsell, of course. Do you mean maybe Envoy Proxy? I don't know. I haven't followed that, unfortunately. Okay. I was also just curious, you know, since it became open source, are you aware of any large projects outside of Google that converted from some other build system to Bazel and what that process may have been like for them?

I think the example that comes to mind right now is Twitter. They announced in a mailing list right before coronavirus started, I think, that they were switching from Pants to Bazel. I don't know how that's going for them. It's a big transition probably for a large codebase. And I know other companies have tried, but I haven't. The Pants to Lukasz does better. So I want to mention a tool called Back by Bazel by Facebook here, which, while it's a separate tool, it's pretty much a spiritual sibling of Bazel. And when Bazel started out, we met with the folks over there pretty frequently to make sure that we learn from each other and to make sure that the two tools are approximately

the same. Jason, anything else you want to add before we let him go? I don't believe so. It was great having you both on the show today to tell us all about Bazel. Where can people find you online? My email address is lberki

at google.com and I am a bit of an online hermit because I don't have a Twitter handle. I don't have a Twitter handle. I don't have a significant social media presence. What else I don't have? You're maybe the smartest of us all here then. Yeah, I have too much of it.

So my online username has always been jmmv. That's two M's there. That's my Twitter handle. And also jmmv. That's two m's there. That's my Twitter handle. And also jmmv.dev. Yes, that's my blog page. You can contact me on my contact details there. Awesome. Okay.

Thank you so much. Thank you. Well, thanks for inviting us. And have a nice day, I believe. Yeah, thanks for having us, of course. Thanks. Thanks so much for listening in as we chat about C++. We'd love to hear what you think of the podcast.

Please let us know if we're discussing the stuff you're interested in, or if you have a suggestion for a topic, we'd love to hear about that too. You can email all your thoughts to feedback at cppcast.com. We'd also appreciate if you can like CppCast on Facebook and follow CppCast on Twitter. You can also follow me at Rob W. Irving and Jason at Lefticus on Twitter. We'd also like to thank all our patrons who help support the show through Patreon. If you'd like to support us on Patreon, you can do so at patreon.com slash cppcast. And of course, you can find all that info and the show notes on the podcast website at cppcast.com.

Theme music for this episode was provided by podcastthemes.com.