The Changelog: Software Development, Open Source - You are not Google/Amazon/LinkedIn (Interview)

Episode Date: August 4, 2017

If you find yourself chasing shiny objects and squirrels all time, you should 💯 listen to this episode featuring Ozan Onay (President of Bradfield School of Computer Science) where we discuss his recent blog post entitled _You Are Not Google_ which was the #1 link in Changelog Weekly - Issue #159. This show is full of wisdom and advice for every developer out there.

Bandwidth for Changelog is provided by Fastly. Learn more at fastly.com. And we're hosted on Linode servers. Head to linode.com slash changelog. This episode is brought to you by GoCD. GoCD is an open source continuous delivery server from ThoughtWorks that lets you model and visualize complex deployment workflows with ease. GoCD helps you automate and streamline your build, test, release cycle

for reliable continuous delivery, promote trusted artifacts, see how your workflow really works, deploy any version, any time, run and rock your tests, compare builds, take advantage of plugins, and so much more. Head to gocd.org slash changelog to learn more. From Changelog Media, Thank you. president of Bradfield School of Computer Science, about his recent blog post, You Are Not Google, which was the number one link in ChangeLog Weekly issue 159.

If you're not subscribed, head to changelog.com slash weekly. This show is full of wisdom and advice. And given Oz's background with teaching software developers, we had to get him on the show to talk through the details of this post. So Oz, you wrote an article recently called You Are Not Google, and it resonated and reverberated across the software developer link sphere or whatever that is. We all read it. We all like to talk about it. Why did that resonate so much with so many people?

Look, I was actually a little bit surprised. I thought there'd be an undercurrent of, you know, old people, let's say, who nodded along and they're like, yes, finally, somebody is speaking truth to the young ones. But it actually, yeah, did surprisingly well across the board. A lot of people reached out afterwards being like, you know, I've had this conversation with my manager or I inherited this code base or whatever. We're really struggling with exactly the thing that you're talking about. And I'm glad you put it out there. So I think we're just getting to the point now where there's been so much excitement, so much promise with some of these technologies. People

have actually had time to implement them and see them in practice. And they're starting to get that like voice in their head that says, maybe this is too much. Maybe this is not the right thing. And so I think, you know, a lot of people have actually spoken and written about this before, but just the timing, I think, is such that people are ready to hear this message. Yeah. Well, we have short memories as software engineers and so it's nice to hear things, even if you've had that thought or read that before,

bring them back up either to a new generation of people who haven't thought about these things or to those of us who have and forgotten that principle and moved on. I feel it's a key thing to keep being reminded of things like this. To even go back to reread books that were pivotal to you. Or go reread or be reminded of things like this in your career to sort of jolt you back into reality. Like, oh, stop chasing shiny objects.

I think it's got to be a counterforce against the marketing machine of of hype and new technologies right like there's a lot of money behind things like mongo uh and so you've got this like constant force online you don't even realize they're inventing acronyms and they're like they're sponsoring conferences and it kind of it's just kind of subliminal that there's this um active paid force to get you to buy into the new technologies. But there's nobody who's putting money behind Postgres and sponsoring conferences and got a marketing team inventing flashy acronyms. So we need that reminder just as a counterforce to capitalism in a way. I was at OSCON London, and there was actually a Postgres company in the vendors area.

So there are people out there doing different things. But yeah, there's voices that are louder than others. Or more well-funded. Sure. Before I get too far into the weeds here, why don't you just give us the gist of this article? I think a lot of it is right there in the title, you are not Google. But give us the high level breakdown and then we'll go into the details of your ways of fighting against this and we'll go from there. But give us the gist. Yeah, so the gist is that there are some amazing technologies out there that are great for

only a tiny, tiny fraction of companies. And we see that they're great, but we forget to consider, are they great for us? And this is a theme across computer science and software engineering. It's true for newer data stores, distributed data stores like Dynamo and its legacy, Cassandra, Rehack, and so on. It's true for large-scale data flow engines like MapReduce and its legacy, Hadoop and Spark, and so on. But also true of

other things like software engineering practices, service-oriented architecture, where it's been amazingly successful in a very specific context. That very specific context is not your context. And so we've just ended up as a community pretending like we do index every website in the world. Or we do have tens of thousands of engineers whose teams need to be split up and interfaced. So the post just pulls together some of these trends that I've been seeing a lot of. Some of this overexcitement about these technologies and these practices and just threads them together like that. This is a question that I posed, I think it was with James Pierce of

Facebook. Coming from their side, especially as we covered the open source ecosystem and seeing Facebook open source its tools to lots of coverage, to lots of interest, and to lots of people start to use them. And the question that I posed to him at the time was, do you guys feel a responsibility for putting out there tools that may not solve other people's problems, but because of the massive interest and because it's Facebook or because it's Google or whichever company, they receive an outsized portion of developer mindshare. And he said that's something that they think about for sure. And they try to address it by way of documentation and giving talks and explaining where they're coming at this problem from. shift back onto the individual developer, not on the company that's open sourcing Cassandra or putting out their manga or whatever, but to us to actually from the other side, not get swept away by the hype. Yeah, totally. I mean, you've got to expect that companies like

that act in their own self-interest, right? Like they're not going to open source something unless they think they're getting value from that, whether that's being a magnet for engineers, getting just good PR generally, inviting contributors and having, you know, offloading some of the work of maintenance, flushing out bugs and that kind of thing. They don't do it out of the goodness of their hearts. Sometimes it's just for retention, right? The engineers there, they want to have their work seen publicly and they want to get the recognition for that. And so, you know, Facebook or a company like that will support that, you know, to keep their engineers happy. But the objective is not to solve your problem. You've got to solve your problem. It's great that they provide these tools for you, but the onus is on you to read the paper, read the docs,

read whatever it is that's available. Obviously, it's open source, so read the source code if you can and really stop and think about your problem. Consider the other technologies that might do as good a job. Really be honest with yourself about what you need and why you're making this decision, and then pull the trigger. You know, you can't just trust that

because it's Facebook, it's going to be good for you. Probably if it's good for Facebook, it's not good for you. We just recently shipped a show with Gerhard Lazu where we talked about the real world situation of deploying Chainsaw.com. That episode has received a lot of praise, Adam, because it was focused around a real-world problem.

It wasn't just tools. It was about the why and the what and the how. And Gerhard really brings just a level and logical method to the way he does things. One thing that I've been cognizant of, Adam, I'm curious if you are, with the changelog itself, because as we cover open source software and the people that make it,

do we sometimes contribute to the hype? And how do we not just be, because we like to cheerlead, we like to root people on. We like to root people on. We love to see success. And sometimes I wonder if that's something that we have done, which is to just make more noise and less signal. You ever think about that?

I think that's a great thing to think about because there's times whenever I feel we cover things because it's our duty in our position and our responsibility to share the news. Right. And the news is what's happening. And it's similar to Twitter in a retweet. Did my retweet of that mean that I agree, disagree? You know, does it, do I represent, does it represent me or my feeling or did I retweet it because I wanted you to also see it?

And I kind of feel like it's on us to sort of share and it's on the audience to, as Oz is saying here in this, in this article is like, Hey, you've got to read the docs. You've got to read the fine print. It's us bringing some of the sifting through, curating what's happening out there, and to a degree disseminating it, but not to every finite point. And I feel like that's sort of our game is that focus. Yeah, I think we need the news news we need to know what's uh

what's out there what's um what's just arrived that we don't know about uh but if y'all if the news bringers can give us a bit of the historical context i think that goes a long way so you know some of uh some of the people i've spoken to they don don't realize that Cassandra is based off of Dynamo. So, you know, just for the sake of your listeners, this guy, Avnesh Lakshman, who worked on Dynamo at Amazon, moved to Facebook and really re-implemented Dynamo at Facebook. And that's what became Cassandra. Some changes, you know, he got to work on the same system twice. So some, you know, you might say improvements, but it's very similar to Dynamo, which also means that it's very well documented. There's a great paper. And so when somebody

delivers news about Cassandra, which I guess is not news anymore, but there's going to be Cassandra's legacy as well. When someone delivers news about Cassandra, giving the context of, hey, this is derived from Dynamo. Dynamo was created specifically so that Amazon could have a high write availability shopping cart because they lose money if you can't write to the shopping cart, but it's not such a big deal if the shopping cart is inconsistent if you see your item twice in the cart. As soon as you hear that, that that was the reason for Dynamo's creation, that they published a paper on it and you can read it and learn a lot about the rationale and the context, and that became Cassandra, well then the news about Cassandra

is less newsy and more like, okay, this is now an open source version of this thing that worked really well for Amazon. So if I have a problem like Amazon's problem, I can use Cassandra. Right. So, you know, obviously that takes a lot more work. And maybe news is going to stop at like a one or two sentence historical background. But just being like, hey, hey, shiny new technology. Cassandra is great because, Cassandra, it's great because it scales. That's less helpful than we as providers and reverberators of news could be. There's two sides to this, really. We've got our quick, poignant Twitter slash weekly email that we ship out,

and then we have deeper, more personal, more human, I guess. Contextual, yeah. Contextual with the podcast. And I think there's times when we hit, let's say we news jack something or we hype train something, you know, just because it's on everybody else's minds and it makes sense for us to cover it, we're not exactly advocating, hey, you know,

because this shiny new tool is shiny and new doesn't mean it's the thing you should choose. We're doing our best to come as interested, curious technologists, developers, and hopefully

providing context to other people's choices and what fits them for their problems. Yeah, the real danger is the cargo cult mentality, as you point out, Oz, in your article. And your call, so we're going to go through, you have an acronym of your own. Talk about marketing hype, Oz. Come on. Unfat.

Unfat. Which we'll go through the finer points of that. I think there's some great advice in there, even though I got some qualms with your acronym itself, because I can't help but bike shed. I think what your overall call here, besides just to point out that this is a phenomenon inside our community, which is problematic,

as people end up with the wrong tools for the job and don't find out until much later and it's much more expensive to reverse that problem. But your overall call is for critical thinking. Or really, you just say, I mean, that's the, I guess that's the T in your own fat there is think. And so I guess my question to you is, what is it about us as a software engineering community

that leads us so, we're so susceptible to the new shiny, and we don't put critical reasoning, and of course I'm speaking generally here, so if you're a critical thinker and you always make the right decision, I'm not talking about you, but maybe it's just me, I'm easily just take the shiny new choice without putting it through that rigorous thought process. What is it about developers that makes us this way? You know, I don't think we're actually that bad. I think we do strive to make good choices and to be thoughtful. But we miss the mark a lot of the time, particularly the more junior engineers, the people who are, well, you know, I say that, but I still have this problem and I still need to be vigilant about it. So don't take that as me saying, hey, you youngsters, you're the ones who are struggling to do this. I do this as well. But there's one thing I've observed about the software engineering community, which is that we love fast feedback loops. We love hacking and getting the feedback. And that's, in some contexts, excellent.

That's something that we can use as a process to do really good work. Just a little bit of input, get some output, have a little REPL going, some quick feedback, hot reloading, whatever. So this kind of thing is great in some contexts. But in other contexts, you actually just need to sit back, turn off the computer, get a piece of paper, be thoughtful, and really reason about the problem. And that switch from fast feedback, fast input, see your results quickly, get that adrenaline rush of building something directly

to, hey, let's slow down, let's take notes. Let's question what we're actually doing. We have something that's working, but is it the best way that we could do this? It's kind of counter to that fast feedback loop thing that works well for us in other contexts. I would wonder if it's similar to, and this may be very provocative, if it's similar to an addiction. Because i wonder if you can connect something like this to maybe something like where people are addicted to instagram or addicted to the the feed the next thing coming where it becomes uh the reason why you do it is

less about wise choices as a developer and more about our actual minds as human beings is that we get this rush you use the word adrenaline rush i wonder if it's connected to something that's far above simply being a developer if it's just a human flaw that's above my pay grade right there i think we're really lucky to have a craft where we can be totally engrossed in it uh where we can get that feedback where we have joy uh from. And it kind of stops feeling like work at those times, which is amazing, right? I'm sure you've had this experience where you've got this big problem to solve. You sit down, you start writing, and then you look up,

your tea's cold, your coffee's cold. It's like nighttime, you're hungry. You totally don't know what happened for the last 10 hours. That's an amazing, amazing feeling. But sometimes that's not what we're supposed to be doing. Sometimes it's not just sitting down and hacking on something and getting that physiological experience of doing this, like rock climbing or tennis or something. Sometimes we actually need to be more aware and less in the flow. We need to slow down. We need to counter our first instincts. We need to question ourselves. And the best engineers can do that and they can switch between those.

And the rest of us are working up to that standard. While we're speculating, I'll throw another thing into the ring. This is something that I've thought of with regards to this particular problem. Perhaps it's part of the revolt against waterfall methodology, the agile movement where it's like head west, young man. It's like let's just get going and we'll figure out as we go. And we found out that that's a better way of building software than thinking through every possible thing way up front for six months and then being done with the design phase and moving on to the build phase. Because six months ago, we didn't know what we needed. And we realized that as you build software, things change. It's in motion as you're building it. And so that perhaps leads to, well, let's just get going. I'm just going to pick a tool and I'm going to build it. And so that perhaps leads to, uh, well, let's just get going. Like I'm just going

to pick a tool and, uh, I'm going to build it and then, you know, we'll, we'll figure that out when we get there. And, you know, as, as with most things, like the, the, the true, uh, best choices are in the gray areas where you want to move fast, but you know to slow down to think as well, right? And you can save yourself a lot of effort by putting some thought and some preparation and still doing agile software development, so you don't have to just fly into everything. I do feel like there is this push to always be moving. Motion creates emotion.

Emotion shows progress. fix it along the way you've got a race to to to do so why not just get in the car even if it has no wheels we'll put them on during the race kind of feeling you know and it's like well we needed four wheels not two and you're halfway through the race and everybody else is you know already finished because they slowed down enough to think how many wheels do we need? And I feel like that's, you're right, where it's like motion is sort of the the anti, where it's

you're always forced to go forward and forward is progress. Have you guys seen Rich Hickey's talk, Hammock Driven Development? Yes. I have not personally watched it, even though I know it's one of his greatest hits as that's true it's documented by rich hakey's greatest hits on jane.com oh it's it's totally

it's totally worth it it should definitely be you know maybe number two or three it's got some great ones but hammock driven development is up there even just for the name right like you've got hammock driven development you're like hmm what else is there driven development what is this an alternative to and um you have this amazing visual as well of uh this uh senior software engineer someone really respected at the company who's you know got his uh sprint planning points or whatever he's about to start work for the week and try and get his points and the first thing he does is string up a hammock, just like sits there. Nice.

Maybe a couple of hours later, comes down, makes a coffee, goes back. I mean, this is how a lot of good software gets written, through thinking, through first thinking about it. And Agile doesn't leave that much room for that or at least it doesn't encourage that uh you need to fight to make room for yourself to think before you you know psych yourself up for the week and get your sprint points uh and so you know waterfall by default uh encourages that it encourages the pre-planning and obviously it has a lot of other downsides and that's why as a community the we've swung the pendulum away from that.

But, you know, now the flip side is that it's up to us to really stop and think. Coming up after the break, we talk about unfat. This is in Oz's words, I promise. It's a dorky acronym for you to follow the next time you find yourself googling some new technology to build or rebuild your architecture around. We break down each letter of the acronym. We talk about its clear intent for humor. But more importantly, Oz shares some serious wisdom to consider when evaluating new technologies. Stick around.

This episode is brought to you by Datadog. Datadog is cloud-scale monitoring that lets you track your dynamic infrastructure and applications. It brings you visibility into every part of your infrastructure, plus APM for monitoring your application's performance, dashboarding, collaboration tools, and alerts that let you develop your own workflow for observability and incident response. Datadog integrates seamlessly with all of your apps and systems from Amazon Web Services, Slack, to Docker, so you can get visibility in minutes. Go to changelog.com slash datadog to get started for free. Also, our listeners get a free Datadog t-shirt when you

start your trial and create your first dashboard. Once again, changelog.com slash datadog and get started for free. And by Bugsnag. Bugsnag improves the task of troubleshooting errors by making it more enjoyable and less time consuming. For example, when an error occurs, your team can get notified via Slack. You can see diagnostic information on the error. You can identify the developer who committed the code

and Bugsnag's integration with Trello, Jira, GitHub, and many other collaboration tools makes it easy to assign and track bugs as they're being fixed. We have a special offer for our listeners. Head to bugsnight.com slash changelog. Try out all the features completely free for 60 days. Once again, bugsnight.com slash changelog. So the acronym you came up with was unfat u-n-p-h-a-t and it says your article says the next time you find yourself googling some cool new technology to build or rebuild your architecture

around i urge you to stop and follow unfat instead. I'll just lay out the words here, the brief synopsis. We go into the details. I'll tell you, the one that enumerates is the one I struggle with here. Anyways, understand the problem. The N is enumerate multiple candidate solutions. That's a stretch, but I get it. The P is read the paper if you find a candidate solution. The H is determine the historical context, which you referenced in this conversation as well.

And then the A is weigh advantages against disadvantages. And then finally, the T, as we mentioned previously, is think. How did you come up with this list and uh this this acronym you got i'll be honest this is uh pretty tongue-in-cheek and uh the way that i came up with the acronym was to first think of the dorkiest acronym that i could and then fit everything else to it so i was really pushing for uncool uncool was was was tough because the two was that would have been good though yeah and on fat i mean i just needed to massage it a little bit and enumerate was one of those things that i messaged but well it's funny because fat phat

is that still something like is that still in the zeitgeist i don't know i mean i know that translate well too you know i i actually had to ask a millennial about that. I was like, if I say fad, do you know what that means? And he was like, yeah, I've heard that word. I think you can use it. So I went ahead with it. And here we are. That's so funny.

I asked a millennial. That's how you have to know. Well, you know, I had a couple of people help me edit this. And one was really doing copy editing. And the other one was like, uh, helping me, you know, empathize a little bit with my, with my younger readers. So that was his input there. That's okay. So you started out with the dorkiest way to do it, which means that you're, as you said, tongue in cheek too.

So you're not trying to be overly serious. You're trying make a point but memorable yeah in a memorable way that's like you know hey come on this i don't really know how do you how do you mean by that like it's okay to be unfat you don't have to be fat phat phat i feel like i have to say that every time since we're audio only i kind of have to go back in my own mind because i'm you know i'm 38 i have to think about like was is fat phat that's fat that was like saying that's cool so right saying unfat is saying like uncool just to give everyone context you may not be they're probably scouring to like urban dictionary to to get to get in a context that is like this is saying right i think ch think Chris Tucker said it best on either. It was either on,

uh, money talks or what was the movie he did with Jackie Chan? Um, I mean, it wasn't Friday to help me out here, guys. Uh, yeah,

I'm, I'm, uh, I know the movie. It's comedy slash. Yeah, there was, it was,

there's a sequel. Oh, it's going to be one of those shows where people are emailing us. I can't. Oz, you can't help us out on this one? No, it was with Josh E. Chan. Rush Hour. Rush Hour.

Thank you. Oh, that would have killed me. And he said PHAT, pretty hot and tempting, was the way that Chris Tucker described fat. You remember that, Adam? Probably not. So we re-acronymed it. I didn't realize it was originally an acronym.

I think it was just a statement of what's cool. It was just a word, kind of an appropriation of the word F-A-T. And I think he then backronymed that during the movie. I don't have the full history in front of me, but I do have that much. So he said it meant pretty hot and tempting. But now you have unfat, so bring us back around here.

Unpretty hot and tempting. That's good context. It goes back to that what Oz said before, which was give something some history, right? Give something some context. And I think that's just an interesting way to unravel this and bring some humor to it as well. Yeah. But what you're calling for here, first of all, is to understand the problem.

And one of the things that you state there is that people tend to think in solution domains and not in problem domains. Can you unpack that for us? Yeah. So when I say problem domain, I really mean your problem. So your business or your project or your customers really thinking about what it is that makes this your problem and not somebody else's problem. That's really the problem domain, the facts and context around that. The solution domain is the set of tools that you could use or the architectures that you could use,

really what it might look like, all the candidates of how you might solve this. Now, people think that spending time in the solution domain and considering, hey, should I use language A or language B, that's what the main decision is about. And you're gonna end up there, sure. But most of your time should really be spent asking questions, understanding your own problem, probing that as much as you can. And then out of that, you'll be surprised how frequently the solution will just fall out.

Where you're like, oh, so we expect to have this happen. We expect to have 10 000 customers use it this much every day over this period of time and uh every right needs to persist uh well you know then there's one solution for that and it's this so uh i i very rarely see people spending too much time thinking about the problem and i i very frequently see people spending too much time thinking about the end solution, what the technology may be at the end. If you start to go down that path as well, it's just a trap. You start Googling, you start reading articles, there's a debate on Reddit or something, you get drawn into that whole thing. And they don't know about your problem, only you

do. So really, if there's one thing that you take from this conversation or this article, it's understand the problem. Just spend the time, ask the questions, dig into it. Everything else should flow much more naturally after that. Yeah. So next up, you say enumerate multiple candidate solutions. So not just your favorite tool of choice. Now I start to get conviction on this one because when it comes to data stores, I just tend to reach for Postgres. And because it's general purpose, I'm generally okay. But perhaps I'm being lazy in that regard. How many candidate solutions is sufficient and why is it such an important aspect of

being unfat? Oh, gosh gosh you're struggling huh i'm struggling i like it though i hate it uh so how many are sufficient i don't know i i would challenge you to at least think of one and at least like honestly give it a bit of a whirl because the temptation is always like, I know this first thing, this default thing, and therefore it's always the best. And I'm not saying you need to think of five necessarily, but at least like yank yourself out of that confirmation bias,

that prejudice that you have for the thing that you first thought of and just look at it from one other perspective uh maybe after that you're going to think about a another one or another one but if all you do is temporarily yank yourself out of this like um deer in the headlights kind of fixation with your language or your operating system or your whatever it is data store um that's great so maybe with postgres you're not thinking, hey, do I use MySQL instead? It wouldn't make sense. But maybe you're thinking, hey, do I really need to actually persist this data?

Is that really a part of the problem? Could I store it in a file? Could I store it in memory? So maybe it's that kind of thinking instead that really gives you the different perspective. Do I really need to solve this problem? Can I just call somebody up and talk to them about it instead? But maybe at the end you use Postgres.

I don't know. It's just something to pull you out of the default path. Any examples where you enumerated the multiple candidate solutions, as you mentioned here, and you were very thankful for doing that task, that discipline? Yeah, so I have a story in there, and the company will... I will not name the company,

but this actually happened with them. So they were using Kafka. The first design of their system was not very good. And they responded to that by really over-engineering the second system. And so it was Kafka and Samsa and all these really excellent technologies that operate at a way, way larger scale than them. And really through a conversation with one of the engineers at the company, we ended up with a design that would have, well, you know, more of a traditional relational data store, but which could have honestly been somebody writing into a book. And that's actually the design that I push for. So I actually push for the data store being somebody who receives an email and physically writes it down,

maybe in a couple of books. So this kind of thing. Yeah, redundancy. Or maybe, you know, you have it in a spreadsheet and a physical book. And I don't think they went for this design ultimately but uh so maybe i'm not exactly asking answering your question of um when i was personally thankful for it but um i just mean to to advocate for something means that you must have had some

sort of reward from doing the discipline so i'm just wondering what was some reward you personally experienced from it yeah i mean know, just the satisfaction of ultimately being right. I mean, maybe that's petty, but I live for that stuff. Nothing wrong with wanting to be right. Yeah, must be right. So I think, you know, a lot of the time comes down to just talking yourself down from the ledge. Really, you get excited about something.

Oh, really excited about functional programming. Clojure is a really well-designed language. I'm going to use Clojure for this project. And then one path is to go down that and really feed off your enthusiasm and write your system in Clojure. And the other one is to say, hey, but I've actually been writing Python for five years and it'll be fine

if I write in Python. I will, as a company, do better. People will be able to understand me. We'll know how to deploy it if I write in Python. That's the kind of situation where you look back and you think, oh God, I'm glad that I didn't

go with my first instincts on that. So given the next point, I have a question for you, Jared. Yes. And since you're using Postgres as your example here. Yes. If we follow what Oz says,

he says to at least give one additional candidate solution to look at and enumerate over. Right. And then once you've chosen that, let's assume you chose Postgres. And so point three is consider that candidate solution

and then read the paper if there is one. And I just Googled Postgres paper and found a 36-page document from the University of California, Berkeley. I know it very well. You know it very well, I'm sure. So this is my question. So have you read this paper,

The Implementation of Postgres? That's funny you ask that because I was just thinking as we come to this third point, I was doing so well up until this point. And maybe I'm coming out here as not a great developer, because when it comes to reading the paper, I'm trying to think of any papers that I've read with this particular goal in mind of vetting a tool, where I've gone and said, I'm going to read the paper. No, I never read that Postgres document, to answer your question.

So I'm just wondering how many people, maybe I'm the only one here, but how many people will actually do this one? And I don't know, is this aspirational, number three, read the paper, or is this, should we all be doing this every time? Look, I think if it's a brand new technology, or if it's like five to 10 years old, yeah, you should read the paper. If there's like one key paper, you know, if you can go and read the Dynamo paper or something, read the paper,

because you're going to get so much useful context out of that. It's just going to, I mean, so, you know, context is I teach databases. I have a lot of students who use these technologies and then they read the paper for the first time with me because I assign it as reading. And so I see that. I see their eyes light up and then say, oh, really? This is, you know, Amazon uses it for that. They want high write availability. Well, we're using it for, you know, for high read availability. This explains why we're getting this inconsistency.

They read the paper, they get the context, and immediately they switch on and understand it at a deeper level than other people at their company who have been using that technology for longer but who just don't have the context. So I think for a newer technology, particularly if there's one key paper, the MapReduce paper or something, that's fairly straightforward to read. But something like Postgres, because it's older and it's got a really long legacy, that's harder and you need a little bit of a guide, like a trail guide.

Googling and finding a paper, maybe it's good, maybe it's not. But if you've got someone to point you in the right direction and say, okay, Postgres has a long legacy, really it's based on System R in the 70s. There are some key ideas. Well, I shouldn't say based on, a lot of people would be angry with me if they heard that. But a lot of the key ideas in these traditional relational database management systems are from System R in the 70s. And so those are well documented. And there are some famous papers there that really get, you know, if you're trying to understand the optimizer, you know, there's one paper there that's still required reading in most databases courses. It's about the Selinger optimization model. It's a Selinger paper.

If you want to understand how Postgres really optimizes your query, you want to start by reading that paper from the 70s. And there are a couple of key papers like that, but they're hard to find, right? You didn't Google Selinger optimization model. You you Google Postgres paper. So, you know, having a little bit of guidance there helps a lot. Hopefully, you've got senior folk at your company who can point you in the right direction. If you say, hey, I really want to understand this at a foundational level, hopefully they'll point you to those kinds of resources and not stack overflow questions. But yeah, it is tough for sure need you need discipline uh you need some spare time at work ideally you need to be at the kind of workplace where you can say hey i'm going to

read this paper for a couple of hours to get a better sense of um what's going on here and for people to be cool with that i would argue too that you shouldn't have to say hey i'm gonna i think you should maybe that's that's too, but you shouldn't be treated like a child where you have to ask for permission to say, let me read a paper to get more knowledge on what we're going to do. I mean, I don't know. But you don't get any sprint points for reading a paper. That's true.

Well, I think we're back to the old waterfall agile. We're forced to produce and producing this code, or commits at least. This might be a good time to mention the papers we love repo on GitHub and that community, because what I was thinking there, Oz, as you were talking is it would be great to have a centralized curated place where you could just come and say, okay, when it comes to Bigtable or when it comes to MapReduce, this is the paper and it's right here.

And I was thinking, yeah, I've heard of papers we love and so I was looking it up and actually that's close to what they're doing. They have a data storage section and this is BigTable database at Dynamo. So you can find all those papers in one centralized place curated by a community. And so that takes out some of the legwork

that may otherwise prohibit you from finding the best or the canonical paper for this particular subject or tool. I'm glad you brought that up because they deserve a shout out. Great community overall, particularly New York and San Francisco. Incredible organizers. Really would encourage folk to go to those meetups if there's one in their city. Great community, great turnout. There's always a thoughtful speaker focusing on one paper. But then the people in that audience are going to be the folk where you can ask a question like that. Hey, I'm trying to understand this thing.

What would you read if you were me? Or what would you be thinking about if you were me? It's a totally different community to the standard meetups so you know big big ups to to them yeah for the listeners if there's an easy thing you want to do right here right now while you're listening you can actually tweet at love a paper and that has their repo link in it so if you just want to say hey heard this on the changelog, tweet that. At least point to any friends you have in the right direction. And we'll obviously include a link in the show notes too.

Absolutely. Oz, moving on in your checklist here, the fourth letter, the H, is determine the historical context. And I feel like that keys into number three, the paper. Because really, when you read the paper, you're probably going to tease out the historical context. But the idea there is what you've said previously, where if you found out why Cassandra was such a key tool that was abstracted out because of their necessity for high availability rights, well, now you know why they built it the way they did. And now you know whether or not it fits your need or not. Yeah, absolutely. If you can't find it from the paper, find it somewhere else or just dig and ask those

questions for yourself. Like, you know, why did Google build MapReduce? Like, what problem was it solving? What hardware were they using? How much were they paying for it? And how much was that an issue? And how much of that fits? That totally changes the equation.

You know, whereas if you just Google, like, is MapReduce a good tool for this job? You're probably going to find the answer, yes, somewhere. Somebody said that and you're ready to read it. Well, maybe just a good moment to add a little self-plug. Another place that's great for historical context is podcasts. So find a podcast about the topic. That's one of the things we do. When we just recently had a show about Kubernetes,

and we asked those kind of questions, like why was this born inside of Google? Why was it open source? What are the reasonings behind it? And so that's a great way to get historical context if you can't find it elsewhere. And heck, if we ever ship that transcripts feature, Adam, you could even just go read.

Right here on the show, you want to... We do. We need to get that out there. I just got all depressed. Oh, man, don't do it. Don't do it. Listeners, we have transcripts for since episode 200 of the changelog. And so that's a lot. That's like 54 plus shows at this point. Maybe more since this show is probably more like episode 260 or something like that.

So long story short, we have full transcripts. We just haven't shipped the feature yet. It's because we have other problems we're trying to solve. It is. And that's okay. That's okay. We'll just keep telling ourselves that until we ship it. That's the bandit.

That's what makes it okay. Next two points here, just to finish up your acronym before we take another quick break, is weigh the advantages against the disadvantages. Do you want to expand upon that, or do you think that's self-explanatory, Oz? Look, it was mostly to get the A in there. But, you know, really, there are going to be trade-offs. Really, the main thing that we do as engineers is decide which tradeoffs to accept.

And sometimes we're honest about that and sometimes we're not. So to keep going with the Dynamo Cassandra example, you trade off consistency. Maybe you want consistency. Probably you want consistency. Probably you want consistency. And so just being aware of that quid pro quo is what I'm saying there. You get the advantages, but that comes with disadvantages. I like the second half of that, though, which was determine what was deprioritized to achieve what was prioritized. So it makes sense, advantages, disadvantages, but in that

context, it's about determining what was more important versus what wasn't as important. And does that align with your goals or your problem? Well, it probably goes back to, who was it? Was it Fred Brooks' book, There's No Silver Bullet? We're always looking for the panacea, the perfect solution, the silver bullet to solve all our problems. And what we find out is there aren't any. And everything is a trade-off,

and that's what engineering is. It's picking the correct set of trade-offs for your problem domain, as you laid out, Oz. And so absolutely, if you're going to prioritize something, you have to deprioritize something else. And so knowing those things before you go into the tool is just think, man. That's number six, think.

I would almost think this is like part of the course, though. Obviously, you want to think, right? Did you feel like you had to drive that one home, Oz? With an exclamation mark. Yes. You know, really, I want the whole article to just be think with an exclamation mark, and that's it, just that one word.

But I don't't you know i just i ran that that by somebody and he was like yeah that's not going to do well just an article with one word so i flesh it out a bit but that's the that's the core thing here really i just made the title you are not google is much more uh than just think good go well i'm glad i went with uh this then but uh uh you, this is the main thing. And many of us have said this. We do need to say it because we see thoughtlessness a lot. Even we're reminding ourselves to be thoughtful as well

when we say this. Our writing code is not really about writing, right? Or writing a book is not really about writing. Thinking is the thing that we do mostly, like mostly we're paid to think. And eventually that gets translated to running code. But the fact is, when you look around, most people are jumping to the implementation. Most people are jumping to a technology choice. Most people are jumping to a technology choice. Most people are jumping to their way of writing code. We're not thinking.

The thing that we know works in software engineering is thinking. Everything else is contextual. Everything else, there are trade-offs. Thinking is the one thing where you can reliably

get better results by doing. That's awesome. Oz, thanks so much for taking the time to write this post, to come on the show and share so much of what you know about software development, all the wisdom you've shared. So really appreciate your time, man. Thank you. All right. Thank you for tuning into The Changewell this week. To learn more about Oz and the work he's doing at Bradfield School of Computer Science, check out bradfieldcs.com. Also, I mentioned at the top of the show how to subscribe to Change Law Weekly. Make sure you do that. Head to changelaw.com slash weekly. That's where we share everything that hits our open source radar. Special thanks to our sponsors this week, GoCD, Datadog, and our new sponsor, Bugsnag. Also, thanks to Fastly, our bandwidth partner. Head to fastly.com to learn more.

We host everything we do on Linode cloud servers. Head to linode.com slash changelog. Check them out. Support the show. This show is hosted by myself, Adam Stachowiak, and Jared Santo. It's edited by Jonathan Youngblood. And the awesome music you've been hearing is produced by the mysterious Breakmaster Cylinder. You can find more episodes just like this at changelog.com or by subscribing wherever you listen to podcasts. Thanks for listening. Thank you.