Algorithms + Data Structures = Programs - Episode 31: Strategic Decision Making & More (Part 3)

Episode Date: June 25, 2021

In this episode, Conor and Bryce talk to Chandler and Patricia. We talk about strategic decision making, how to disagree, one-way vs two-way doors and so much more.About the Guests:Chandler Carruth leads the C++, Clang, and LLVM teams at Google, building a better language with better diagnostics, tools, compilers, optimizers, etc. Previously, he worked on several pieces of Google’s distributed build system. He makes guest appearances helping to maintain a few core C++ libraries across Google’s codebase, and is active in the LLVM and Clang open source communities. He received his M.S. and B.S. in Computer Science from Wake Forest University, but disavows all knowledge of the contents of his Master’s thesis. He is regularly found drinking Cherry Coke Zero in the daytime and pontificating over a single malt scotch in the evening.Patricia Aas is a C++ programmer with a “thing for building browsers”. She works for a company she co-founded called TurtleSec where she teaches courses in Secure Coding in C++ and does consulting and contracting. She has been a professional programmer for 16 years, and started off her career working on the original Opera browser. Since then she has made embedded products at Cisco and another browser at Vivaldi. When she has time she works on her own open source (pre-alpha) Chromium/Blink+Qt based browser called TurtleBrowser.Show NotesDate Recorded: 2021-06-05Date Released: 2021-06-25ADSP Episode 29: From Papa John’s to Google (Part 1)ADSP Episode 30: Google, Interviews, Leadership & More (Part 2)ThinLTO Clang DocumentationTeresa Johnson - ThinLTO Whole Program Optimization - Meeting C++ 2020 Center KeynoteGuts, Part Three: Having Backbone – Disagreeing and CommittingOne-Way vs Two-Way DoorsIntro Song InfoMiss You by Sarah Jansen https://soundcloud.com/sarahjansenmusicCreative Commons — Attribution 3.0 Unported — CC BY 3.0Free Download / Stream: http://bit.ly/l-miss-youMusic promoted by Audio Library https://youtu.be/iYYxnasvfx8

There's an episode of the TV show Futurama where the protagonists discover that cats are actually aliens, conquering aliens that have come from another planet and conquered humanity into caring for them. Welcome to ADSP, the podcast episode 31 recorded on June 5th, 2021. My name is Connor. And today with my co-host Bryce, we continue part three of our four-part conversation with Chandler Carruth and Patricia Oss. On today's episode, we talk about strategic decision-making, how to disagree, and so much more.

If you haven't listened to parts one and two, they will be linked in the show notes. I highly recommend you listen to those ones first, as this is a continuation of that conversation. So I'm just curious, should we try and get one of the points here is, is not overcoming resistance to a directional change. Yeah. What does that mean? Like, can you give, put that in an example, explain how you would go about that? Yeah. I mean, I think this is, so, so first off, let's, let's, let's make sure we understand what we're talking about. We're talking about times when there's, you know, there's there's a strategic decision or a directional decision that needs to happen. And you have you're suggesting and you are proposing a particular direction.

And that is not the direction taken right and you still need to like you need to like we're in the kind of disagree and commit phase but it's not about like doing a single thing it's about like a very large scale directional shift and and i've seen this a couple of times and i've i've had this happen to me a couple of times um so the particular like one of the particular sharp examples here is there's a technology that I do not personally kind of like. It doesn't seem elegant to me. And it's called ThinLTO. This is a variation of link time optimization developed by Teresa Jackson and David Lee and a bunch of other people at Google. And personally, I don't like the way it works.

It seems like a messy workaround. And so I advocated really hard that we should actually invest in understanding why. But the goal of it is to make link time optimization scalable enough to use broadly on very, very large C++ applications, huge C++ applications. And LTO normally, as it normally is, just doesn't scale very well. And so you can't use it for these applications. And thin LTO scales very well. I had argued that we should actually try and tackle the underlying scaling issues of LTO rather than taking the thin LTO approach. I thought it was much more clean.

I think there's a lot of really exciting untapped potential in that space. Basically, every link time optimizing compiler that we had access to, which was basically one and a half LLVMs and some early stage work in GCC at the time, was single threaded, right? And built around link lists. And so like has the worst cache locality you can imagine.

And so it doesn't scale well, which is kind of unsurprising. And I was like, we need to fix those problems before we try and kind of change the fundamental model to make it scale. But Teresa and David had a great idea for how to change the fundamental model. And it didn't require changing all of the internals of the compiler it didn't require you know making things you know multi-threaded and efficient all this stuff and it still scales and the decision was to basically go with their approach it was it was and at the time I really disagreed with this I thought this was was just the wrong thing to do. And it was kind of hard because, right, like I had to help them out.

I was bringing up LLVM at Google, having brought up Clang as the front end. I was bringing up kind of the LLVM backend for actual compilation. And they were working on thin LTO and needed my support in LLVM. And that was hard for me. I struggled at first. And largely, I think finally, we got to a much better place where I found ways that I could kind of contribute to enabling them, rather than kind of getting in their way. But it was difficult. And the funny thing is that, like, in retrospect, right, like, if today me got past me and Teresa and David Lee's kind of different proposals, I would absolutely pick thin LTO. So as much as it's less elegant, it's also dramatically less risky. And at the time, I felt like I really didn't see this risk that was associated with the direction I was pushing for. But in retrospect, it actually makes a lot more sense why we ended up going that direction. I

think it was the right call, even though personally, it felt really like, oh, this is a good technology. And it also worked. And for me, the biggest part of this kind of disagreeing and committing at the strategic level was, you know, I had a loud voice in the Olivium community. And it was important to actually acknowledge, like, this is working. And to do that very loudly. And so, you know, making sure, like, Teresa could get up on a panel to talk about LTO. So I got invited to a panel. I was like, you should probably also bring Teresa up

because like, you know, I advocated for a different strategy of LTO and hers is the one that actually got built. So maybe we should not, right? Like, and doing that is, you know, it can feel bad, but that's the important part

of kind of disagreeing, committing at a strategic level is that you actually have to own, you actually have to own the strategy at the end of it. And because it's a strategic thing, right, it's not just like you're going to let the patch go in, right? You have to contribute to the new strategy, right? You have to help debug things.

You have to help with the planning. You have to, like, figure out how the piece of, you know, LVM that I'm working on can actually work with the thin LTO strategy, how to integrate this stuff. And you have to pull yourself on board with that, even if it feels like the wrong strategy. But your negativity towards the concept probably spread to people around you as well. Because we are tribal are oh i'm sure you know we

are tribal in in in how we work we are in our groups in it and oh of course so did that cause any friction but when you were trying to to i'm sure i'm sure it did and like i can remember specific places where it did because there were people people who really thought I was right and I should have been able to make this decision. And one of the interesting things, and again, this is a great observation, because when you want to actually overcome this kind of having a direction that you don't agree with being decided on. You actually have to undo that negativity. You can't prevent it. Like, there's no hope, right? People are going to, as you said, this is a natural thing. But you have to figure out how do you undo that.

And one of the ways that you undo that, for example, is by going and doing the work, right? Do the groundwork, the dirty, grungy work to actually help out the direction you didn't want and let people see that because that helps people then understand that like, oh, okay If Chandler's okay with it then I guess it's okay Right, like

and you have to be an active participant in kind of undoing the tribal stuff There will be people who will feel an intense loyalty to you. And so therefore, yeah. I also want to be clear. I don't think I handled this particular case perfectly. Part of why I'm bringing this one up is that this one is probably one of the rougher cases.

And I think there's still kind of ruffled feathers and it still is difficult for people who are on different sides of this to kind of work effectively. And we all lose as a consequence, right? Like, that's not a good outcome. Is there a way that you could have handled it differently that could have maybe given a better result?

I should have been much more vocal earlier. So there was a lot of lag time between kind of the decision getting made and when I was able to kind of you know do some of this legwork to support it um and I think that really uh amplified a lot of the problems um and what would have been better would have been to have kind of very immediate like decision made and immediately show signs of like helping implement the new decision um but but for me like when i i read these things it seems it seems like an upward thing towards management that i have to get on

board with this thing uh because the decision has been made and and you know so that the management or whoever's above me will see that I'm a team player but and and that doesn't resound much with me but but what you're saying is more that I need to show these people who are loyal to me and who respect me that you know I'm okay with this and we should be okay with this together and we should you know yeah that makes much more sense to me and it's it's wild the effects this has like like this this you know uh that delay which i i still really regret to this day um uh right it hurt team morale right it caused you know parts the team to drift apart right how many missed opportunities were there because we let this friction – and I say we let because I was part of this, right?

We let this friction develop. And it's got to be a lot. And that's, you know, it's a regret. There's a – a part of my soul is sad that I'm bringing this up. But it is a good leadership principle. So there's – at Amazon, there's these 14 LPs, which stands for leadership principles. And Amazon is famous for, you know, really promoting these across culture and whatnot. And one of them is disagree and commit. And it's actually, it's actually a pretty good one. Some of them I don't, I don't agree with all 14 of them. But this one, it's basically like, there's going

to be times where there's different directions that you can go, either technically or strategically. And they're really big on data-driven design and stuff like that at Amazon. But at a certain point, a decision has to be made. And basically, the LP talks about how once that decision is made, you can say, I don't agree with this decision. I think it's, I prefer this one, but we're going with this one. And once that decision is made, it's supposed supposed to be like, everyone's, everyone's on board with

it, commits to it, and they're going to make that successful and work. And, and there's an opportunity to sort of state that, like, I don't agree with this final decision. But once that's, that decision is made, it's, it's time to, you know, go forward. And yeah,, yeah, it's, it's definitely not easy, but like having that as an LP at a company, like as, as a sort of a tenant, like I definitely have worked at companies where those disagreements happen and then they just sort of like, they just linger. And, um, it's not even clear, like, uh, you know, how do we come to the decision? And so they just sort of, it goes on and on and on. And it's, I think it's a great sort of tenet to have that, like, you know, there's inevitably at any company, there's going to be disagreements. But at a certain point, you just got to pull the trigger.

And everyone's got to contribute to that direction at that point. And, you know, maybe a year down the road, you'll figure out it wasn't the right decision. And you can reassess at that point. But for that year, when you're getting to that point, it doesn't help to have only 50% of the team that believes that that was the right decision to be really all cylinders ahead on it or have 50% not all cylinders ahead. That's not helpful. You're going to realize the decision was the wrong one if it was the wrong one faster if everyone is just helping out and i want to be very clear about something i having a tenant is important but taking it back to patricia right the questions like like for interviewing someone in a leadership role

the the particular key thing here is that people in this world need to really show that they are operating by that tenant. And that's even more important than the tenant itself. We have the tenant at Google and it still happens because whenever there's a leader that isn't showing they're going to live by this tenant, it starts to cause like it starts that entrenchment and that i i think we are i think it's a lot about the human condition we are often very proud people and we also have lots of loyalty and and having these things will make it difficult to say okay i lost and also i'm not only okay with losing but I'm enthusiastically on board with you. Yeah, it's really hard. It's a great way of putting it.

So some techniques that I've been trying to practice to help me overcome that human reluctance is to change the kind of frame of mind of what my goal is. I'm not usually a huge fan of corporate trainings, right? Sometimes they're real questionable. But there's one corporate training that I really love. Even though most of it's garbage, it has this one catchphrase that I keep coming back to, which is, you know, what do you really, really want? And then they clarify, no, no, no. What is the one thing that you really want? Like, you must choose one thing and focus on that. And when you apply that here, the interesting thing you get to is what you really want is a decision.

Not your decision, but a decision not your decision but a decision and so by focusing on the fact that you've made a decision you're making progress you basically channel your enthusiasm for progress as proxy for the enthusiasm of the decision you don't particularly like right and this this kind of lets you get yourself enthusiastic yeah and then apply it to something that you're not super enthusiastic about because okay so get a proxy realize yeah yeah and it's all about picking well what do you really want do you want your decision or do you want to have a decision because you can't always have both so you have to pick which one you care about most once you do that it's much easier to

be enthusiastic about not having your decision because that's not what you're enthusiastic about you're enthusiastic about making it through the decision getting a decision making people like start moving on in some direction right um and and this for me was really helpful in part because i noticed something. And a manager of mine, you know, years and years ago told me this. And I didn't get it. It didn't make sense at the time. But he pointed out that it is much cheaper in so many cases to let people make the decision they want to make than to convince them to make the decision you think is right.

And then, to your point, Connor, eventually, if it is wrong, they'll figure it out. It can be so much cheaper to let people make a bad decision and then learn what the right decision is than to try and prevent them from making a mistake, or trying to avoid the wrong decision just just make the bad decision it'll be totally fine we'll figure it out later um and that can be so much lower cost um especially at larger scales i i yeah yeah i well i have a baby example from when i was like a super junior um i was at opera and we we were working on the desktop browser and we had this guy who was basically, I don't know, I guess in retrospect, I would say he was like sort of like a creative genius, but he was also a coder. And so he had created all sorts of things. And so basically a lot of what was opera was basically his brainchild um and so and then he comes and he was like he has this brilliant idea in his like his opinion is brilliant and all of these people that you know worked with him for a long time they you know he was like super here they were like

this is brilliant and and i thought it was dumb i was like like, nobody's, this is silly. Like, are we going to do this? This is a really silly thing. And I was so wrong. I was like the wrongest in the entire world. Because what, I don't know if anyone remembers, but way back when, in the early 2000s, when you opened a tab, it was white.

There was nothing there. it was just a white rectangle in your face and his idea was let's put like big icons and things there for your favorite websites and he called it a speed dial and it would have like the nine like a speed dial and he would even connect it to see if you tap the right like number it would open that but you could also click on them and so you had like a three by three matrix where you could put in all of your favorite websites with like a picture and everything and so when you opened a new tab you would see the speed dial and you could just click on where you wanted to go i was like this is silly this is stupid it's like five I just I was so wrong I was I was absolutely completely terribly wrong about this entire thing and and and for me personally

this was huge for me because and it's colored like my entire thinking for the rest of my career because i i learned that i can be fundamentally completely wrong about something and have no idea like and because i think we have this tendency to believe in ourselves like i you know i think this is dumb and i will all data i will find will reaffirm my my position and i would kind of hold on to it no matter what happens but actually sitting down and and like accepting the fact that this was brilliant and and it has been copied by all browsers ever since you know it's like 15 years later, this is what you do.

But so it was brilliant. But having that gave me the, I don't know, I won't call it humility because it's not humility. It's a healthy skepticism to my own opinion. A willingness just to be wrong. Yeah. And see that good things can happen even when you're wrong.

And just like, okay, so this is what I think. I might be wrong. There might be a fundamental flaw here where I'm just totally wrong. And if I am, that's okay. Yeah. And that's like a superpower. If you can actually go walk into a room and be convinced by someone else and like really, truly convinced that I think that what you said was right and what I said was wrong, that can build like mountains. Like it's massive.

And it's such a small thing. I do this, I freely admit I struggle with this. That's why I cheat. I cheat by focusing on, you know, progress over. Yeah, but I think that's a good one because I think that's something that, you know, you can do because it hurts to be wrong. It sucks to be wrong.

It really does. It's interesting this is coming up because literally just on the last episode, Bryce remarked the exact same thing that his telltale sign of really bright folks is the ability to like once they've realized that they're wrong just to like unabashedly like change 180 degrees and start 90 degrees and start walking that direction as if like they never at any point previously thought otherwise like they'd oh i'm wrong okay that's fine with yeah yeah and yeah something something you were saying earlier and this sort of goes to the your example patricia's i i'm not i actually don't think this is an amazon thing but i learned it in amazon they have this policy of like one way and two way door decisions and that like if it's a two-way door decision meaning that you can enter and then leave um those are the ones where it's a lot easier to just like you know disagree

and commit and we'll find out and that's the thing is like designing that sort of nine three by three icon if it's a terrible idea at the end of the day you just you just trash it sure there's a sunk cost there but like there's there's very very little cost to like reversing that and there's certain decisions where it's one way if you're going to make this big change and those are the ones that you really have to spend a ton of time up front you know figuring out is this the direction although chandler you're saying maybe not even that one no no no like i think i think that's that's that's actually a critical flaw the goal shouldn't be to spend a ton of time up front figuring out the right decision when you have a one-way decision the goal should be to engineer the door to open the other direction yeah the goal should be like what one one-way

decisions are never good um like what one of what the the classical example of this is standard C++. One of the big problems that standard C++ has is that pretty much all of our decisions are one-way decisions. And we would have a lot less stress if we could find ways to have that not be the case. This is something that I see in API design all the time. A lot of people feel like you know an api is a one-way decision because once you publish it you have to support it forever and and so so then a bunch of people invest incredible amounts of time trying to do good api design and to avoid mistakes and and my perspective has really been warped here to a different thing where it's like i mean sure invest some amount of time there but you almost want to time box that very sharply and instead invest all the rest of the time in getting out

of the mistakes that you will inevitably make um because one of these things is has a different scaling function on it than the other one right if you if you invest a bunch of time making a good decision, that's one way. Great. But the next one, right? Like it doesn't, none of that investment pays off for the next decision you have to make. Right. Whereas if you invest a bunch of time making the door two ways, all of that time you invest pays off every time you get the decision where that door applies. And so it's a tremendously better scaling function and so you time box the non-scaling investment right and put all the rest of the

energy into into changing the the problem space that's a great point it's just it's basically like from finance or business it's fixed versus variable cost like you you do the two-way door and it's variable and then that just distributes over every single decision you make in the future which yeah i've actually that's a it's variable, and then that just distributes over every single decision you make in the future. Yeah. Actually, that's a great meta point. Like, Amazon, you should update your book. Update your stuff.

But there's another kind of other subtle problem here, which is the time spent before a decision is made is the most expensive time. So one of the interesting things about spending your engineering time on the doors instead of on the decisions is that you make the decision more rapidly. So you timebox your investment in the decision. You make the decision rapidly. And that has its own value. And you're not paying for an unmade decision the entire time you're investing in doing the door. You're just making bad decisions. Can I take us down from down from metaphoric to concrete like what is the door yeah absolutely here so so like classic example let's take apis i think c++ apis are a great example here right

we can we can try to do good api design but the the thing we actually want to invest most of our time in is the ability to change the API when we get it wrong. And so we build refactoring tools, and then we build large-scale refactoring tools, and then we build systems for running large-scale refactorings across a monolithic code base, or across all of the shards of your sharded code base. And then you tune that system so that you can run them more rapidly and at lower cost. For example, at Google, one of our big costs is we're lucky we have a monolithic repository that we can handle, but we have to do code review. And so maybe we're making 10,000 refactoring changes.

How do you code review those? So we've actually built tools and infrastructure to smooth the code review cost and to bring that cost down, build tools to smooth the testing cost of rolling out these changes, right? And what we're doing is we're, because you almost never have true one-way doors. What you do is you have a differential cost function based on the direction, right? That it costs us so much money or time to go in and it costs us, you know, so like this massive multiplier if we ever want to go back out. What you're actually trying to do is you're trying to drive that cost down over time.

And we very specifically just attack the economic problem of replacing an API when we discover a mistake. And every time we make that cost lower, right, this makes the kind of risk of each API decision lower. Because we're like, well, maybe we're wrong. And so we can still apply some time boxed amount of, like, there's still a differential here. It's not free, right? But now we can have a much more rational time box around like, well, we've consulted the experts, we've thought about the consequences of this API design, to the best of our ability, it's going to, like, we think this is a good one, let's try it out, and if it doesn't work, we'll replace it, right? And even for a particular API, we, like, you can, you can turtle this all the way down, right? So for a particular API, past a certain point,

rather than trying to theorize about how the programmers will react to this API and how readable or not readable it will be. If we're particularly uncertain, we'll actually invest in being able to roll back that particular API. So we'll have, you know, an allow list early on for who gets to use it so that we can avoid it, you know, being everyone on day one, right? And we can kind of have some beta testers and get some rounds of feedback. We may put special things into the API just to make it easier to change if we discover we're wrong. And again, the whole thing here is just about decreasing the differential cost so that we can make the decision and not worry about it, and then learn and then make a new decision when we get new

information. Thanks for listening. We hope you enjoyed. Stay tuned for part four, our final part of this conversation, and have a great day.