CppCast - Maintaining Large Codebases

Episode Date: September 15, 2016

Rob and Jason are joined by Titus Winters from Google, about Google's strategies to maintain a 100M line monolithic codebase. Titus Winters has spent the past 4 years working on Google's core C++ libraries. He's particularly interested in issues of large scale software engineer and codebase maintenance: how do we keep a codebase of over 100M lines of code consistent and flexible for the next decade? Along the way he has helped Google teams pioneer techniques to perform automated code transformations on a massive scale, and helps maintain the Google C++ Style Guide. News Visual C++ for Linux Update What's New in ReSharper C++ 2016.2 Exploring std::string C++, Short and Sweet, Part 1 Titus Winters Titus Winters Links CppCon 2015: Titus Winters "Lessons in Sustainability" CppCon 2015: All Your Tests are Terrible Sponsor Backtrace

This episode of CppCast is sponsored by Backtrace, the turnkey debugging platform that helps you spend less time debugging and more time building. Get to the root cause quickly with detailed information at your fingertips. Start your free trial at backtrace.io slash cppcast. CppCast is also sponsored by CppCon, the annual week-long face-to-face gathering for the entire C++ community. Episode 70 of CppCast with guest Titus Winters recorded September 15th, 2016. In this episode, we discuss the ReSharper C++ update and the internals of Steadstring. Then we talk to Titus Winters from Google.

Titus talks to us about what it takes to maintain a 100 million line monolithic codebase. Welcome to episode 70 of CppCast, the only podcast for C++ developers by C++ developers. I'm your host, Rob Irving, joined by my co-host, Jason Turner. Jason, how are you doing today? All right, Rob. Are you all ready for CppCon? Ready as I'm going to be. Yeah, next week, CppCon. Should be exciting. Yep, Sunday.

Are you ready? Maybe. Just going over my talks some more today. Okay, that's good. Talks. Yeah, you're going to have a busy week. Okay, well, at the top of every episode, I'd like to read a piece of feedback. This week, we got an email from Shehar Mike, or Mike Shehar, I'm not sure which it is.

And he wrote in, hi, Jason and Rob. I'm a C++ developer who also takes great interest in C++ beyond work. One day I was looking for a C++ podcast with the assumption that I'm the only one who cares about C++ so much that I actually listen to it off work. And then I found CppCast, which is exactly what I was looking for. And I listened to all the great interviews and updates you provide. I got to know the online community. And following that, I even opened my own blog. Thanks again for all the effort you put in.

I know I appreciate it a lot. I didn't even realize it at the time, but when I was preparing for the show, we actually pulled one of his blog posts as one of our news items this week. Oh, that's cool. Yeah, so his blog seems to be doing pretty well. Mike, thanks so much for the feedback. Thanks for listening to the show. And tell your friends.

Yeah, tell your friends. There's always people interested in this stuff, I think. Not just you. We'd love to hear your thoughts about the show as well. You can always reach out to us on Facebook, Twitter, or email us at feedback at cpcast.com. And don't forget to leave us a review on iTunes. Joining us today is Titus Winters. Titus has spent the past four years working on Google's core C++ libraries.

He's particularly interested in issues of large-scale software engineering and code-based maintenance. How do we keep a code base of over 100 million lines of code consistent and flexible for the next decade? Along the way, he's helped Google teams pioneer techniques to perform automated code transformations on a massive scale and helps maintain the Google C++ style guide. Titus, welcome to the show.

Hey, thanks. Glad to be here. Thanks for joining us today. So we had a couple news items to go through. Titus, feel free to jump in on any of these, and then we'll start talking to you about the work you're doing at Google and large-scale code maintenance. Okay? Sounds good. Okay, so this first article I have

comes from the Visual C++ blog. And we've talked about Visual C++ for Linux before. I think we talked about this with Ankit Asthanu, and he had him on the show a while back. And they just released a new update with a couple new features. So they now have a makefile project template that you can use. And it says it works with CMake, GMake, BashScript,

overrideable C++ compiler path. So if you don't want to use their version of GCC, you can point to a version of GCC that you already have on your machine. And they added some new debugger options. Right, Jason? Yeah, it's just so crazy to see Microsoft doing all this stuff. Yeah, it's definitely a crazy world we're living in right now with some of the things they're putting out.

It's good stuff, though. Yeah. The next one, also Microsoft-related, is a new version of ReSharper C++, which is JetBrains' tool for Visual Studio for C++ developers. And they put

out a bunch of new features. The big one here is Doxygen support, which can be really nice to have. Yeah. And it looks pretty powerful. You can be changing the parameters in your function, and it'll automatically be updating the Doxygen comments. So definitely a nice feature to have.

On top of that, many new bug fixes, better code analysis. Anything else you wanted to call out here, Jason? There's one thing that surprised me, that they let you specify what you want the default of a null value to be, where previously it was null pointer, now you can choose null or zero. Like, no. I'm guessing it's there

for people who aren't using C++ 11 or 14 yet? Yeah. But it's been five years. I mean, what date are you going to pick for getting with the program like 10 years yeah

it's a good question I'm guessing there's enough people out there though that warranted them making the change there must be people asking for it otherwise they wouldn't have done it I'm sure I live in New York and work at google new york and like when we do interviews we get a lot of finance people and it's gotten a little better in the last couple years but a lot of finance

groups seem to still be stuck pre-11 oh goodness yeah and you know i i don't understand like you have to have a plan yeah i was kind of noticing that they also added mocking support and i've never actually used any mocking tools have either of you guys well it's actually google mock so i'm guessing you're familiar with this titus yeah people that used to report to me are now the primary maintainers. Oh, okay. So it's near and dear. Okay, well, the next article is Exploring Standard String, and this is the one from Mike Shehar and the one who wrote us that feedback. And he's going through the GCC, the Clang,

and the Visual C++ version of standard string and taking a look at how it's implemented. And it's a pretty interesting article. You can see how memory allocation works, the size of the string. Not really a thing I really thought about, but it seems that each of the three kind of have very different implementations. Right, Jason? Yeah, and I think it's also maybe an important read for people

who have not heard of small string optimization before or are curious how it might be implemented to read this article. Yeah, do you have any thoughts about this one, Titus? Yeah, I really liked the analysis and i really liked uh that specifically included the old version for libstid c++ prior to the avi break yes i think that the details in the analysis like the the data is great um i'm always curious to get into the why things are done the way they're done and some of the story behind why old string was copy-on-write and why it's not allowed anymore. I think those are great and kind of important stories to be aware of

and to kind of internalize because they have impact on other aspects of programming that you might stumble into. Right. Right. And this last one, which I wanted to mention, was a course on Udemy, which I'm not too familiar with Udemy, but I think it's kind of similar to Pluralsight, only you can buy courses kind of a la carte as opposed to having a subscription. I believe so. Yeah. And this is an opposed to having a subscription. I believe so. Does that sound right? Yeah. And this is an introduction to C++ course.

It's called C++ Short and Sweet Part 1. And, you know, I haven't watched the course, but it's five and a half hours long, and it's completely free. And it's given by a professor named Jeremy Syke who works at Indiana University Bloomington. So if you're interested in getting started with C++, this seems like it might be a good way to go if you haven't gone through your own university. Yeah. I was scrolling through this and I'm like, this is interesting.

And then I noticed Jeremy. Do you know who he is, Rob? I'm not familiar with him now. He was the lead on the first concepts proposal. Oh, okay. That was determined to be more complicated than it needed to be. And that's why what we have today is colloquially

known as conceptss Light. That's because it's the trimmed down version of what Jeremy was working on. Interesting. I didn't realize it was the same guy. Yes. Yes, that's definitely a very talented developer to be giving this free introductory course. There's also an interesting side note to that, is that most of his research has been on

gradual typing of programming languages, which is the research that led to TypeScript, the Microsoft language. Right. That's the language your cousin used to work on, right? Yes, which is part of why I know part of Jeremy's history, but also because I met him at his C++ Now, C++ Now number two, I think it was, way back in the day. Okay. Okay. Well, Titus, let's start talking to you about what exactly you do at Google. Can you give us a description of what your actual job is at Google? So for years I was the lead for the C++ libraries team, which kind of started out very humbly. There's utility code in any sort of code base, and ours is no different. And a lot of the utility code was written by the original engineers,

people who kind of had grown along with the company into being very senior figures. And so something like five years ago, there was a proposal for, let's have a dedicated team to maintain some of those libraries so that our super senior people don't have to. And then from there, we realized, oh, hey, we're maintaining things like

our collection of string manipulation routines and caches and things like that. We should make sure that everyone knows that they exist instead of trying to rewrite their own because the Google primary code base is kind of a large mono repo. And so we got into trying to do advertising and advocacy and spread the word, make sure that people don't reinvent the wheel.

And that started snowballing into taking larger and larger roles of like, yeah, we kind of maintain and control the core pieces and need to modify the core pieces. And we need to figure out how to adopt things where we overlap with the standard. C++11 came out, and the story that we tell a lot is the one of we had our own internal owning smart pointer like most codebases do or did and we wanted to move to unique pointer and so we had to do some research on how do you change and commit changes of call it half a million, instantiations of that smart pointer type and make that into a unique pointer.

And then, you know, so we've kind of refined processes for large-scale changes, and along the way I started to help maintain the C++ style guide, and it kind of just continues to expand along those lines. So you say large changes. Let's get into some details. How big are you talking? How big is your code base? We've been saying publicly for a few years that it's upwards of 100 million lines. I don't think I can say more. Okay. So we can maybe assume that it's slightly bigger than 100 million

if you've been saying that for a few years, I guess. You can assume that. Okay. And the other stat that I think gives some sense of scale is we have somewhere upwards of 6,000 people that will make a change to a C++ file in this codebase this week. Wow. The internal program for mentoring new C++ programmers has about 3,000 people enrolled either to be mentored or on the waitlist.

So there's kind of a large population here. That's roughly half of the C++ developers then are being mentored or involved in mentoring? We've recently sort of decided that 7-day active isn't the right stat for measuring the population, but we literally came to that conclusion like two weeks ago, so I don't know what the better stat is. So somewhere between six and maybe ten thousand. But I'm kind of guessing. Okay.

Now, I thought I had seen an article or something recently that said that Google uses essentially one giant repository for all of this code. Is that? That is true. And basically everything builds from head. Okay. So you have like 6,000 people working on hundreds of millions of lines of code and one giant repository continuously. Yep. Okay. Now we've got this idea of what we're talking about.

Yeah. So when we make changes to like core library sorts of things um you know you we have unusual challenges in that most things of a scale that's actually interesting uh can't be done in a single change like you can't have that many files in a single change you certainly wouldn't be able to sync a change with that many files to head in order to get it submitted. You're saying because you would end up with a conflict every single time you tried to sync? Exactly. It moves to us.

So we have unique issues in the changes that we make. We have to figure out how to stage non-atomically. So you introduce the new interface and at every step change whatever callers you can until there are no more of the old interface, and then you delete the old interface. But it has to be safe to build at every moment. Wow. Yeah, there's a knack to that. You should start learning what things can be done and what things can't. So I'm just trying to understand why exactly you go with the monolithic repository, because there's so many different products that must be coming out of this repository, right?

I mean, it's not just for Google itself. I mean, there's Chrome and... So the open source projects aren't in there. Okay, okay. So Chrome is not one. Neither is Android. But Gmail and Search and all of our store stuff and the majority of cloud and, you know,

internal infrastructure, those sorts of things. It's absolutely crazy. The amount of things going on. Yeah, and kind of the reasons why we think the monorepo approach makes sense is because nothing else really scales. When you have a separate repository for absolutely every product, there's no central choke points that you can actually rely on. I wouldn't be able to make changes to core APIs without having linear scaling in the number of separate repositories that those APIs appeared in. And the problems just get very, very hard. Our ability to make change is predicated partially on the ability to have complete awareness, complete understanding of where an API is being used. You can search for it. You can index it.

You know exactly how it works. You know that there's exactly one copy of it. If you need to fix a bug, you need to do it once, not ten times. If you need to add additional security checks, you find the handful of places at head where you need to add those checks, and you're done. And you don't have to have dedicated content management people and contacts on every team.

It just doesn't scale well. Right. It makes some sense. I mean, I've certainly never worked on anything nearly that large, but I can see the argument. Yeah. So what lessons do you think you have that might apply to developers who aren't working on 100 million line code bases?

The one that's been very big for me, and it kind of touches on one of my talks at CppCon last year, is the notion of sustainability and the inevitability of change and being aware of the kind of the context and expected lifetime of what you're working on. I find that a lot of very novice programmers, like just coming straight out of college, they've never worked on anything for long. You know, maybe over the course of a semester, they worked on the same project. And maybe they have a hobby on the side of something that they've worked on for a couple years, but odds are not strong that it has users or other developers. And so you kind of look at the expected lifespan of code like that

as weeks, months, maybe a year or two. Most startups fail in 18 months, so everyone working in the startup industry, you can expect that that's, you know, what you're looking at. And it's a very different thing when you're working in a monorepo where, you know, it's hard to argue that Google is going to disappear in five years. And it's basically impossible to argue that we can get out of this mon repo in anything less than you know five or ten um and the result of that is like what you check in is probably going to outlist outlast your tenure

at the company for a lot of people um and that puts a different balance on a lot of the engineering trade-off like if you're just in a startup and you only have to last for 18 months, you should very rightly accumulate all the technical debt necessary to get to your next round of funding. Technical debt is a great metaphor. You borrow to finance short-term velocity, but you have to pay it off later.

Whereas if you're in an organization where you need to be looking at 10-year and 20-year horizons, you need to make sure that everything that's going to change over that period, you're capable of reacting to. And that's advances in the language standard and changes in tooling and changes in libraries that you depend on and changes in tooling and you know changes in libraries that you depend on and changes in hardware and everything else like change over a long enough time frame change is inevitable all right okay um can you tell us a little bit about some of the continuous integration systems using google um do you have your own stuff or is it something we might be familiar with? Most stuff at Google is custom, largely because our scale is so ridiculous that nothing else works.

Which, when I first got there, kind of drove me crazy because I didn't really internalize how much the scale was an issue. I was like, oh, why do we just keep reinventing everything? Then I realized, like, wow, yeah, scale. So yeah, we have continuous integration, mostly continuous testing that runs. There's a rather large amount of cycles in the fleet go to continuously running all of the unit tests for C++ and the other languages as well.

So when a commit's made, your system automatically runs all the tests that are available? All the tests that were affected, yeah. All the tests that were affected, okay. So when you're developing as a developer on your local machine, how modular is the code? How long does it take you to build the whole thing? What does the average person have to deal with? It really

depends a lot on where they are in the stack because it's very project dependent. Some of the big projects have painful development times. Probably on the order of half an hour, an hour to actually do a full build of things.

Okay. Most things for incremental builds are still within, you know, a minute, five minutes, 10 minutes. So your continuous integration environment, I'm assuming, then does incremental builds, does not do a fresh build with each commit? Right. We have something that's very much like Bazel that guarantees reproducibility of the build at every step. And once you have actually reproducible, there are no weird side effects. There's never a need to make clean.

Then it's much easier to commit to a world where you just have an incremental build. All right. And all of those distributed build caches. I'm sorry, distributed build caches? Yeah, because most people are synced to roughly the same point, and most files don't actually change. So a lot of the build artifacts can be cached and distributed, and Google is good at distributing and caching. So that's interesting.

So an individual working on a desktop in Google is going to have access to a cache of all the previous build files that other people have done? Yep. So does that kind of require everyone's working on pretty much the exact same build configuration? Largely. We definitely frown on people having elaborately different build configs. Okay. We specialize very heavily. So, you know, the compiler groups here are responsible for deciding which flags are appropriate to use in the build and to kind of limit it down from every team has control over absolutely every flag,

which ruins your build cache, down to there's a configuration for getting the debug build and there's a configuration for just build it as fast as you can, and there's a configuration for build it for opt, for performance. Okay. So managing the build times of large codebases and doing things like static analysis and continuous integration is something we've been

asked about a couple of times to talk about on the show. So it sounds like heavy caching is one of the main keys you guys have settled on. Yeah, not my area, but yeah, that is my understanding. I wanted to interrupt this discussion for just a moment to bring you a word from our sponsors. Backtrace is the debugging platform that improves software quality, reliability, and support by bringing deep introspection and automation

throughout the software error lifecycle. Spend less time debugging and reduce your mean time to resolution by using the first and only platform to combine symbolic debugging, error aggregation, and state analysis. At the time of error, Backtrace jumps into action, capturing detailed dumps of application and environmental state. Backtrace then performs automated analysis on process memory and executable code to classify errors and highlight important signals such as heap corruption, malware, and much more.

This data is aggregated and archived in a centralized object store, providing your team a single system to investigate errors across your environments. Join industry leaders like Fastly, Message Systems, and AppNexus that use Backtrace to modernize their debugging infrastructure. It's free to try, minutes to set up, fully featured, with no commitment necessary. Check them out at backtrace.io.cppcast. Just going back to monolithic codebases for a moment, I'm just wondering if you are familiar with maybe what some of the other larger software companies might be doing through some of your colleagues who maybe have gone to Microsoft or Amazon. Do you know if they're also using monolithic or do they use more

multiple projects? It varies and I think it reflects in a lot of cases sort of the corporate culture my understanding is that microsoft is fairly fragmented uh and amazon facebook are fairly monolithic um but it's been years since i had any uh deep conversations about that with colleagues. So my information could easily be out of date. Interesting. Okay. At the CppCon 2015 talk, you talked about automated code transformations. Can you tell us a little bit about

what types of transformations you've made and what tools you've used? Right. So the first sort of large transformation there was, like I was discussing, the scoped pointer to unique pointer transform. And this is we rely very heavily on the Clang AST Matchers framework, which much of the development for that

comes from internal Google tools teams, but it's all done in the open. And with the AST Matching framework, you basically are specifying patterns in the C++ abstract syntax tree, tongue twister. And with matching against the AST, you can specify, like, hey, when you find a match

in the AST of this form, here's a snippet of code to run. And then within that snippet of code, you can ask, okay, so give me the location from the source of where this match actually occurred and then you're down to the realm of, okay, so here's a call to the thing that I'm talking about. Let's change the text.

And the framework gets more advanced all the time and the set of matches that we build advances. So we have, we work on tools for more general, like replace this method or rename this class or find all calls that look like this

and change them to a different format of call or reorder the parameters or whatever. And so we're kind of slowly building a suite of more or less off-the-shelf tools to perform some of the common refactoring tasks. One thing that we're working on right now, historically our code base has had the weird quirk of most people just type vector and list and map instead of std vector and std list and std map and that kind of harkens back to 10 years ago we took a gcc update that was the

first time that namespace stood was required and we didn't know how to adapt to change at the time and had to take the easy way out of just adding using declarations for the standard types or all the standard names that we were using at the time so that the build would still work after the toolchain update. Now we're going through with client-based tools and actually just running rename class on map to std map and vector to std vector and et cetera, et cetera, to just try to get rid of that little weird technical debt. Because we find that in the rare cases where we try to use code outside of our normal build environment, the weirdness there tends to bite us.

It makes it hard to share code and things, so we're just paying that down. And that's all more or less standard tools for us now. So all these transformations and processes you're writing, the concepts are all available to anyone using libclang-related tools, right? Yes. And a lot of the refactorings that we do, if we know that we're going to have to do it ongoing, they just get added to Clang Tidy. And occasionally we'll just get fed up with unnecessary string conversions.

And, you know, you call a.cstring for a function that accepts a string. And so we'll just strip those out. And there's a Clang tidy for that, I believe. And some of the libclang-ast-matchers tooling is, I think, upstream. Certainly we're working on making sure that more of that is. That's cool. Yeah, I was wondering about that, how much of it is accessible to us normal people. We want to see more of this happen.

We want it to be less of a weird alien thing. So I think there's desire to do that. It's just occasionally tough to get around to it. So you said they often migrate to Clang-Tidy. Do you start, do you write

them in C++? Do you start with the Python API? It's almost all C++ from the start. Do you want to talk a little bit about automated tests and unit tests that Python API or? It's almost all C++ from the start. Okay. Do you want to talk a little bit about automated tests and unit tests that you run at Google? I would be happy to talk about tests. So one of the questions that people ask when I talk about refactoring is,

how can you do that? And you're changing things. We don't like changing things. Things break. And that's kind of true until you change them enough. And then they kind of become more supple, but more flexible. But it is kind of critical for us to have unit tests basically for everything. And one of the things that I find, especially new developers, they rarely have a background in testing.

And almost always when I first talk to them, they feel like testing and even code review are processes to tell them that they're bad at their job. And that is completely the wrong conceptualization. Tests are a proof that your code is correct, which is nice, but also a proof that all of the change I made down in the low levels hasn't broken you. And without that proof, we're all just flying blind. And it's not a formal proof by any means.

There's plenty of things I've changed that don't cause a test to break that might still be wrong. But in practice, we find that it works well enough. with we have policy that says if all of us librarians, if we make a change that breaks your team's release and you didn't have a test for it, it is not our fault and we will not roll back. We jokingly call this the Beyonce rule because if you liked it, you should have put a test on it. But it's actually really important because this uh kind of puts the right uh you know values in place of you know if you just want to throw something together you can just

throw something together but we have no need to like keep it running um So there's trade-offs. Engineering at scale like this is very much a community. There's a lot of social community engineering aspects to it. I can see that. Were there a lot of growing pains in

getting everyone to accept this amount of unit testing that you do? Yes, although mostly the battle for unit testing happened before I got here. From my understanding, the battle for unit testing was maybe circa 2007 when they started drawing hard lines on no more change goes in without a test. And I'm very grateful to the people that fought and won that because we would not be able to do the things that we're doing now. Like, not a chance.

It's not even a matter of technology. It is entirely the matter of being able to execute it with some notion of safety. And even the first times that we just made big changes, changing, you know, scope pointer or changing our string split routines, things like that, you wind up finding weird dependencies on behavior that you never thought. Like, it's certainly not documented that that was a behavior you could rely on. But internally, we have one of my colleagues, Hiram, has made more changes to the code base than absolutely anyone. And on the flip side, I have reviewed more changes to the code base

than absolutely anyone. And Hiram discovered over the years that with the sufficient number of users of an interface, it doesn't matter what you promised in the interface contracts, all observable behaviors of your class or function or whatnot will be depended upon by somebody. So we call this Hiram's Law. And it's really kind of fatalistic, but it's true. I remember one time early on when I was just starting on the team, just alphabetizing the includes for a very central file.

And we have a kind of global monitoring system that exports monitoring that can be viewed in production, things like that, counters and trackers and things. And one of the tests broke far, far away from where I was making the change, because they had very foolishly written the test to depend on the full text of dumping that counter. And it included the file name and line number of where the counter was defined. Like, guys, this is not okay. Like, you can't, the tests can't depend on that.

So we've been, we fought the battle for you will have tests, but we are still working on you will test well. So on the topic of, you said all observable behavior will be depended on by someone, essentially. I got that right. Do you then, Do you find yourself extremely limiting the public interface of your classes

because you just don't want to deal with people relying on things you didn't intend? We generally don't go that far on it, partly because we're capable of changing things, but most of the time we don't

need to um so we kind of the the the application of hyrum's law is really when change is necessary uh be prepared to be surprised okay so you don't you don't take a proactive approach to it it's more of a just don't be surprised. It would be too hard to do. And to some extent, the standard tries to do the same for some things. It is undefined behavior to take the address of symbols out of the standard has the right to add new overloads and to change the type that you get back from taking the address of something. Most people don't know that or think about it, but the why there is actually critical. Like, every time that you go take the address of pushback,

you are making it harder for the standard library to potentially change and improve interfaces. Interesting. So you're already at CppCon 2016, right, Titus? Are you going to be talking this year? I am not talking this year. When I was thinking about it back in the spring, there was nothing I found that was critical for me to get out in the public.

So I have some early plans for talks next year, but we're not there yet. So this time I get to just kind of relax. Just enjoy the conference? Yep. Sounds nice. Okay, well, is there anything else you wanted to share about, you know, getting started with unit testing or, you know, just other horror stories about large code bases that might be interesting? Oh, I could go on.

As far as testing, like, I really cannot overstress how much testing I think is the next frontier for software engineering to grow up. I regularly tell the story that if you go back 10 years, it was not surprising to encounter groups that weren't using version control. And now they're out there, but when you bump into them, you're like, whoa, dude, you're weird. And I honestly think that the next 10 years is going to kind of do the same thing for testing. Because it's important to know that your code actually works.

And it's not a velocity like it doesn't harm velocity to sit down and write tests or try a test first even Your designs usually come out better when you write tests first your velocity definitely goes up when you spend some time writing tests the conceptualization that it's just you you know, weird, futsy overhead is just kind of baffling to me, just from everything that I've seen. Occasionally, a team will be like, oh, we're on a sprint, we have to meet this deadline. And so we're just gonna throw all of our practices out the window. And I have never heard a single story of that going well you can't just stop being an engineer like engineering is about having good practices because it works not just because some guy said so um otherwise there wouldn't be engineering

best practices um And testing is evolving into, yes, this is kind of a critical part of being a grown-up and responsible software engineer. And I'm really looking forward to seeing what we're all able to do once we have that

net of safety, that safety net and that set of checks in place. Change is much easier to affect when you know that you didn't actually destroy the world. Right. Okay. Do you have any other questions today, Jason? I do not

believe I do. Okay. Well, Titus, it's been great having you on the show. Maybe we'll run into you at CppCon this year. I hope so well thank you so much for your time today thanks guys thanks for joining us thanks so much for listening as we chat about

C++ I'd love to hear what you think of the podcast please let me know if we're discussing the stuff you're interested in or if you have a suggestion for a topic I'd love to hear that also you can email all your thoughts to feedback at cppcast.com I'd also appreciate if you can follow CppCast on Twitter and like CppCast on Facebook. And of course, you can find all that info and the show notes on the podcast website at cppcast.com. Theme music for this episode is provided by podcastthemes.com.