The Changelog: Software Development, Open Source - The foundations of Continuous Delivery (Interview)

Episode Date: July 2, 2021

This week we're sharing one of the most popular episodes from our new podcast Ship It. Ship It launched in May and now has 8 episodes in the feed to enjoy...it's hosted by Gerhard Lazu, our SRE here at Changelog. In this episode, Gerhard talks with Dave Farley, co-author of Continuous Delivery and the inventor of the Deployment Pipeline. Today, most of us ship code the way we do because 25 years ago, Dave cared enough to drive the change that we now call CI/CD. He is one of the great software engineers: opinionated, perseverant & focused since the heydays of the internet. Dave continues inspiring and teaching us all via his newly launched YouTube channel, courses, and recent books. The apprentice finally meets the master 🙇‍♂️🙇‍♀️

Hey everyone, Adam Stachowiak here, Editor-in-Chief of Changelog, and today in the feed we're sharing one of the most popular episodes of our new podcast called Ship It. Ship It launched in May and now has eight episodes in the feed to enjoy. It's hosted by Gerhard LeZou. Check it out and subscribe at changelog.com slash ship it in any where you listen to podcasts. Here we go. Hey, how's it going? I'm your host, Gerhard LeZou, and you're listening to Ship It, Here we go. else is an implementation detail. Today I have the privilege of chatting with Dave Farley, the inventor of the deployment pipeline and co-author of Continuous Delivery. He's one of the few that had a profound and permanent impact on my professional career and most likely yours too. We talk about what a good continuous delivery pipeline should be, why there is no template for the ideal

pipeline just yet, and why you should subscribe to the Continuous Delivery YouTube channel. We also discussed the changelog.com deployment pipeline. After this conversation, I will always assume that I'm wrong, strive to learn quicker, and make it my ship admission to chat with Elon Musk. Dave likes this idea. Big thanks to our partners Fastly, LaunchDarkly, and Linode. Our bandwidth is provided by Fastly, learn more at fastly.com, feature flags powered by launchdarkly.com, and we love Linode. They keep it fast and simple.

Check them out at linode.com forward slash changelog. applications. InfluxDB empowers developers to build IoT, analytics, and monitoring software. It's purpose-built to handle massive volumes and countless sources of timestamped data produced by sensors, applications, and infrastructure. Learn about the wide range of use cases of InfluxDB at influxdata.com slash solutions, network monitoring, IoT monitoring, infrastructure, and application monitoring. To get started, head to influxdata.com slash changelog and click Get InfluxDB. Again, that's influxdata.com slash changelog. We are going to shift in thing, getting things out there.

And something happened around 2010. There started to be a shift in 2012, 2013. There was an acceleration of just Git pushing and the code wouldn't get out there. And I thought that was amazing. Like, why haven't we been doing this all along? I mean, what else do you need just to get it out there for the user to tell you, does it work or doesn't it work, or you're missing this?

The quicker you can get to that point, the better off you are. And even if you make mistakes, that's okay. How do you learn if you don't make mistakes? So don't try to not make mistakes. Try to make them so quickly and fix things so quickly that no one even notices. By the time they notice it's a problem, you fixed it and it doesn't exist. So I think you, Dave, had something to do with this because around 2010, right, you published this book, you co-authored this book with Jez Humble, which was called Continuous Delivery. And even though, hand on heart, I haven't read the book, but everything that you capture in the book, I'm sure I have practiced for more than a decade. And I cannot think of working any different. So how did you come up with the concept of continuous delivery? I'll start off by admitting I'm very old and I've been doing this for a long time. And so mostly how I came up with this is by doing it wrong in lots of weird and interesting ways first and finding out what didn't work.

I had a kind of formative experience, several formative experiences. But one time I remember in the late 90s building some reasonably complex software for an insurance company. And we were supposed to deploy it by writing a manual script that somebody would then take over and execute. And the system wasn't configured in the way that we expected. We didn't know what the configuration system was really. So that didn't work very well. So I remember me and a friend spending two days stood up in a server room somewhere trying to manually install the software and I was thinking, there's got to be a better way of doing this. Later on, I worked on a project for a point of sale system when I worked for ThoughtWorks in the early 2000s. And we were doing extreme programming at some scale. At the time, we thought that it was probably the biggest Agile project in the world.

There were about 200 people working on this Agile project in three different continents. In those days, it sounded really weird because Agile projects in those days were just small projects. And we were kind of trying ideas out. And so we started to get more disciplined in our approach. We started seeing the things that were going wrong. And we started to introduce more of the deployment automation, more of the configuration management, infrastructure as code, better approaches to automated testing, and starting to formulate

basic deployment pipelines out of those. And it was during that period that we started playing with the ideas and pulling something together and thought that we had something. The book itself came out of a notion that there were a bunch of us doing different things on different projects in ThoughtWorks and we thought that we were onto something. We were starting to see patterns that worked, and we were starting to apply those to the project. So we would go in having a better

sense of what to do to begin a project and have some success with it. And so we thought we'd kind of write a book that was initially meant to be a series of essays. And there were a bunch of us that said, yeah, we've got some stuff to say and started talking about it. Actually, when it came down to it, only Jez and I did any writing. So that's why we ended up writing the book. And the book morphed into something else. It certainly, it didn't start out being called Continuous Delivery, but it morphed into something else. And both of us, I think, were a little bit wary of thinking of this at the time as I now do think that it's kind of a methodology. I think it's a way of approaching software development in its own right. And I believe that it's an engineering practice. I think that

engineering in the sense of amplifying the impact and talents of the people that are making the changes. As you said, software development is weird stuff. And one of the really hard things is knowing how well you're progressing, knowing how good your ideas are. And so you want to be able to get those out into the hands of users quickly and efficiently so that you can learn from that and adapt and change. And I think you put it perfectly when you were describing it in terms of, we want to give ourselves the freedom to make mistakes. We want to be able to start off.

I am a popular science nerd. I love reading about science and physics in particular. And I think that we can learn a lot from the fundamental philosophies of science. I don't mean kind of six sigma accuracy in statistics or anything like that, but just applying kind of scientifically rational thinking. You know, start off assuming that we're wrong rather than assuming that we're right. Test our ideas, try and falsify our ideas.

Those are better ways of doing work. And it doesn't really matter what work it is that you're doing. That stuff just works better. And certainly the ability to move quickly, make small changes quickly, observe the impact of the changes so that you're in effect controlling the variables, you're limiting the scope, the blast radius of mistakes is a fantastic way of making progress efficiently. One of the things that I am obsessed with at the moment is watching Elon Musk and SpaceX build starships

to go to Mars and blowing them up in Texas. Because, you know, that's how you learn. That's how you do great engineering. I think we're onto something here because I think continuous delivery is an approach that allows us, facilitates that kind of thinking and that kind of approach to software.

You've made so many great points, but I have difficulty tracking all the things that I want to mention to all your points. I mean, there's just so much. So let me start with this. First of all, thank you very much. You have no idea how big of an impact your approach and your teachings and sharings had, not just on me, but on everyone I know. The stuff that you do and the stuff that you've been promoting for decades now have been part of me in many different ways. So for example, I was also into physics when I was in high school and I was convinced that I'll go to university to study physics. But then when I was in Puget Sound, the University of Puget Sound in Tacoma, there was the Max Planck talk, 100 years from Max Planck.

That was a fascinating conference. But I discovered the Macintosh and that changed my life. Yeah. So I was so good at making mistakes and learning from them and deploying and figuring out what doesn't work that people like they said, can you do my website? And can you host my thing? And can you like, oh, I have like, how do I do emails? I don't know how to do emails. This was like the early 2000s. That's when I started doing these things properly. And I realized, well, this approach of trying to figure out if it works, it's so good, you don't need anything else.

Keep learning, keep iterating, keep improving. And that's all there is to it. Nothing else. And continuous delivery is so fundamental to this approach of working that everything changes and you don't want to go back. I mean, I can't imagine. Absolutely. That's one of the things that I've observed. First of all, thank you for saying thank you. That's very kind. But I've had the privilege of working with some great teams over the years, and I have yet to see one that has adopted continuous delivery in the way that I would recognize it that ever wants to work in a different way.

So one of my other formative experiences was I was involved in building a very, very high performance financial exchange while I was in the middle of writing the continuous delivery book. And that was an exercise in genuine engineering. We were doing some really hard stuff, some really difficult stuff to be able to build this ridiculously efficient software system. But we were starting with a blank sheet of paper. I was the head of software development and I was sort of dipped in the middle of this continuous delivery thinking because I was in the middle of writing the book. So we built the organization from the ground up as a continuous delivery organization. And the commonest message that I still get from the people that I worked with on that project is,

oh God, I miss what it was like there. If they've moved on, they've gone somewhere else. One of my friends now is in New Zealand and he's working there. He regularly grumbles to me that if only we were doing those things. People don't want to go back to a different way of working. This stuff works better. And fascinatingly, we're gathering data to back those sorts of statements up for science nerds like you and I. You shouldn't just trust what we say. We should also gather data and you should try that for yourself and all those sorts of things because if we're right it's a reproducible thing it's it's not some kind of magic yeah i love that starting from i i'm wrong yeah let me figure out what right looks like yes and if you always assume that you're wrong even with all your experience and everything you know you will never be wrong

that's like very weird because you'll figure what right is. You don't know what right is. It changes all the time. It's contextual. And most importantly, it's the people that you work with. They always change. You replace a team member, you have a new team. Someone leaves, someone joins, you have a whole new team. So how do you stick to those principles and how do you promote those healthy principles that everybody respects, abides by, and then magic happens. And I think you were capturing a little bit of that magic with your friends, right, that you used to work with, and you became so much more than co-workers. That's amazing. Yes, certainly. And I think that's certainly the difficult problem. We're technologists, and so we often get, you know, lured by the technology and your joy of discovering the Macintosh and all that kind of stuff.

But the hard parts, I know it's trite, but it is true that the hard parts are the people parts. It's not that continuous delivery or its practice are particularly difficult. In fact, I would argue the reverse. I would argue that one of the lures of this way of working is that it's a much simpler way of working. But you have to put some work in and you have to think about things differently and you have to discard some of the baggage from previous ways of thinking about things. organizations to do that these days I make a decent living with helping people to try and make that change. But it's incredibly difficult. And one of my proudest boasts is that in the organization that I just mentioned, where we built the exchange, it was an organization called LMAX. Still exists, still trading, still running these

exchanges around the world, built on our technology. And the culture is still fantastic. Funnily enough, I saw a job advert for developers for LMAX past my event horizon today. And I was reading about it and I was smiling to myself because I probably could have written that job advert seven years ago when I left. Because, you know, the culture is still there. The behavior is still there. And that's continued. So we started something that the group of people that were there at the beginning, that's been durable in that organization. We established a development culture that has been not only lasted, but has been communicated to subsequent work generation of developers and

other people working in that environment, which I think is fantastic. I'm incredibly proud of that. It is difficult to get people thinking differently, to change their minds, to jump out of the ruts of their old habits and jump into a new way of thinking about things, which is these days is a lot of the thing that I get pleasure from is trying to help people just think about ideas differently. So would you say that the people influence these practices and the people contribute to how these practices work? Or is the inverse true, where the practices influence the people to behave in a certain way and then sustain these practices long term?

I think it's both. I think it's a combination. I think it's a little bit too trite. So we've always said things like what it takes to build great software is you need great development teams. And that's true. You need good people, but it's not enough. I've worked with some genuinely brilliant developers building bad software. And that's not their fault. A bad process will break good people every time. And so there's more to it than only that. And this, again, is one of those things that why I refer back to science quite so frequently. Often developers and development teams and organizations are somewhat disdainful of process because they assume that software is a heroic exercise carried out by

geniuses toiling against the code man team. But you could say the same kind of thing about science. Science is this terrific endeavor, a human activity carried out by fallible, mistaken human beings. But by organizing their thinking in a certain way, they eliminate whole classes of errors and biases that are built into us through our biology. And if you want to do engineering, which I would count as an application, so practical science to a practical end is the way that I think of engineering. If you want to do engineering of that form, which is what I've come to think of what we do as when we do the sort of stuff that I'm talking about, then if you want to do that, then you're going to have a better outcome. You're going to improve your chances.

It's no guarantee. It's not going to make a bad development team great. It's not going to make a bad development team build world-class software. But it's going to improve the quality of their work. It's going to amplify their talents and their skills to an extent that they can do better than they would do without it. And that's true of world-class developers too. One silly example, one of my good friends who I worked at Elmax with, he was the CTO, I was head of software development, he's Martin Thompson. And I regard Martin as

at least one of the best, probably the best programmer that I've ever met. He's genuinely brilliant, really, really talented guy. And I've known him for a long time. I've known him for many years. We first met in the 90s. And he's a bit younger than me. And he was a young man then. And he was very, very good then. But I taught him to do test-driven development while we were at LMAX. And he and I think he's a better developer now than he was before. There are some of these techniques that however good you are, or however bad you are, can improve you. And if we were to be able to identify something that we could class as an engineering discipline, then I think it would have that kind of property, which is really what I'm talking about. I think we are

odd in software development circles, in that we tend to take terms and change what they mean. I think in nearly every other context that we can think of outside of software, if you use the term engineering, it means the stuff that works. In software development, we've turned it to mean something else, usually something more complex and something that we don't like very much. I think words have some power. And I think that I like to use a reasonably strict definitional approach to parsing things to be able to form ideas. And if we think of engineering in the terms of the practical application of science,

then it ought to work for software at least as much as it does for anything else. And we have a bunch of advantages in our favor too. We've got one of the most powerful experimental platforms that happens to also be exactly where our software lives, a computer. First of all, I really like that you point out this distinction between engineers, software engineers, and developers, because it's a very important distinction. People don't even think about it, and they use the terms interchangeably, but they mean very different things. Now, I think we can

have a show just about that, whether we are software engineers or software developers. So let's park that there, recognize it for what it is and move on to the other thing, which I, like when you mentioned this, Martin being the best software developers, engineers? Yeah. Software developers or engineers? Both. Both.

Okay. Okay, great. So being Martin is like best software craftsman, software person. TDD made him better. Yeah. Before TDD, what made him so good in your eyes? There were lots of things. He's a very smart guy, which helps.

It's not enough, but it helps. One of Martin's great talents is he's got a laser beam focus on simplicity. And one of the things that I learned from developing software with Martin is I've really strengthened one of the tools in my toolbox, which is focusing on separation of concerns. So Martin is absolutely brilliant. He sees the least piece of code that is doing two things and immediately he's pulling it apart to try and separate those two things

so that each piece of code is focused on achieving one outcome and then he's growing it from there. And he's always had this – Martin's code is almost like reading prose. It's readable. It's modular. It's cohesive. It's just nice code. And it's also blisteringly fast.

Martin's one of the world's experts on high performance and concurrent systems. And he's widely recognized as such. The people on the Java team occasionally ask him for his advice about how to speed things up and that kind of thing. He's well respected in the industry. But the thing that I value most is focus on separation of concerns as a driving force in the design that he applies to code. I kind of had a more informal use of that kind of technique.

My design skills were pretty good. I'm a decent coder. I'm not a bad developer myself. but Martin was always so focused on it and I've picked that up now. And now I'm always looking, oh, could I pull something apart here? And my code is much nicer as a result. I can definitely see how the TDD would have enhanced that property or that aspect because it forces you to focus, right? I mean, you could be focused or not as you write code, but when you write TDD, if you do it properly, if you start with red, right, which is always the first one,

do it red, go green, then refactor. And I even have a video on this, and I don't like, we'll introduce this a bit later, but that's, I think, fascinating to see how simple it is if you really think about it. And one of the things, again, I'm jumping here between things, but there's, again, there's like so many things I want to talk to you about is how it took you a really long time of thinking, working in the space of extreme programming, agile, I'm sure it's a big part of that, continuous delivery, test-driven development, to not only hone your skills, but also share your skills and share your knowledge with everyone that

wants to listen or is interested in these things. So in a way, it's not just the focus, I would say, but also the consistency and the perseverance to stick with it. I mean, it's been decades and you've been sticking with the same thing, right? Yeah. On the same hill. It's like, I mean, sure, you're sharing it with others. I know that Jez Humble has a same hill, maybe a different hill. The point being these things stand the test of time and focusing on that one thing long-term is what, I don't know, recognition, admiration, thankfulness, respect. That's how you build them. Success, whatever you want to call it. It's all related, I think. Okay. That was a very nice story. Thank you for that. That's a pleasure. This episode is brought to you by our friends at LaunchDarkly, feature management for the modern enterprise,

power testing in production at any scale. Here's how it works. LaunchDarkly enables development teams and operation teams to deploy code at any time, even if a feature isn't ready to be released to users. Wrapping code with feature flags gives you the safety to test new features and infrastructure in your production environments

without impacting the wrong end users. When you're ready to release more widely, update the flag status and the changes are made instantaneously by the real time streaming architecture. Eliminate risk, deliver value, get started for free today at LaunchDarkly.com Again, LaunchDarkly.com. Again, launchdarkly.com.

Now, as we come back, I would like to dig a little bit into the technology that you used back in the day. So the specifics around the CI system, the CD system, the programming language, the frameworks, how that used to work, if you had any project tracking tools or how you would organize work in the days. And I would like to dig a little bit deeper into those specifics. Sure.

Time it took, which cloud provider you used, if any, where you would run these things and how they changed over time. Yeah. So you're telling us some very good stories, Dave, about your time at LMAX, that you were very fond of, some great people that you've worked with, that you've been in contact ever since. And you were like part of that family in a way, right? Like your work family, maybe your continuous delivery family, whatever you want to call it. I would like to dig a little bit deeper into the specific stack technology

that's used at the time and also how they changed over time. So in the early 2000s, what did the technology stack look like? The programming language, the framework, the CICD, how did you organize work, that type of thing. I'm more than happy to talk about tools and technology. I should begin though by caveating it because I'm not very technology driven, which is weird for a technologist to say, but I think that the tools are secondary. I think that I value design and design thinking more than I value the tools. And I think you apply that in different tools. Having said that, so my background was largely in the C family of languages. I did a lot of programming in C in the early days, C++ later on.

During the early 2000s, building the point-of-sale system and stuff like that, I was doing a lot of work in Java, sometimes C Sharp, bits of other things, Python, played with Ruby slightly. But mostly the early days of continuous delivery were mostly on Java projects. One C-sharp project that I can think of. And so we were mostly using the technologies around at the time.

In the early 2000s, there weren't many tools, certainly no continuous delivery style tools. So in the early 2000s, we just started doing continuous integration, really. At least it had become popular. So people had been doing continuous integration for a long time. So I was doing some version of that in the early 90s when we had continuous build, but it was all just done in shell scripts. Tools to manage a build process and those sorts of things didn't come along until about 2000. And the first one was a thing called Cruise Control, which is an open source project from ThoughtWorks. When we built the Exchange LMAX, we started off using Java and Cruise Control,

the starting point for building out our deployment pipeline. And we built a very sophisticated deployment pipeline with different instances of that, using mostly things like ant files as the glue between the stages and those sorts of things to encode more complicated bits of glue between the different pieces. We did a lot of development of sometimes reasonably sophisticated tooling of our own. So we built our own deployment mechanism, which was similar in some ways to something like Chef or Puppet. It ran a little agent on the server and the server

called back to some master repository and pulled down changes and deployed them for us. We were doing early things with kind of infrastructure as code so i remember we wanted to be able to version control the configuration of network switches and the only way that you can configure the network switches was through a firmware admin console web-based firmware like you're getting a home router or something like that so we wrote a little domain specific language that we could program the configuration of this thing in, which then kind of was back-ended by, I think it was Selenium or something similar. Yeah, that's right. Would then drive the web app to poke the values into the router.

So we were doing a lot of messing around with those sorts of things, very ad hoc, very implementing those as we needed. And as I said, we did some fairly cool, fairly sophisticated things. One of the things that one of my colleagues, Mark Price, wrote at LMAX was still the best version that I've seen ever of a test distributor. So we were managing a fairly large set of infrastructure to be able to get our feedback fast enough. We had one big repo.

We put everything in one big repo, and then we could build and test and deploy everything together. And we could be more certain then in our changes. It meant that we didn't have to worry too much about how loosely coupled. We wanted good design to make sure our designs were loosely coupled, but we didn't have to have them independently deployable, the pieces, so we could test them together first. So we did that, and that was efficient.

We ended up with a network. When I left, it was about, I think it was about 48 different server instances that ran our continuous delivery infrastructure and a dynamically managed grid or compute grid to evaluate these things, which Mark Price wrote the software called Romero to be able to manage all of these different instances. So we did a lot of various tooling of that kind. We weren't, in that project in particular, we weren't very big consumers of other people's software to some

degree. We wrote a lot of stuff of our own, partly because of the performance demands. Most of the third-party software that we had wasn't fast enough for what we were trying to do with our exchange, so we had to write our own. So we wrote our own collections, for example. Okay. A hash map in Java for each entry at the time creates five objects.

So you've got five garbage collection problems for every item that you add to a hash map in those days. So we wrote constant memory footprint hash maps and stuff like this so we could go fast. So we did a lot of stuff at different levels of abstraction, very low-level technical detail stuff to bigger picture things. That's really fascinating to capture the context in which these ideas came to be.

Because while, you know, different people may have had similar thoughts, you, first of all, you thought works was a consultancy. I imagine the time, right? So that's how we start. That's what I know about ThoughtWorks. And so not only you had to come up with these ideas, but you also had to build the tools, which didn't exist. Open source, I think, was only just getting started.

This was like early 2000s, right? So it wasn't really a thing. Git, I think, was only just starting around that time. GitHub didn't exist, by the way. And we know that's what an important place that is for open source. Twitter didn't exist. Facebook didn't exist. A lot of the platforms open source. Twitter didn't exist. Facebook didn't exist. A lot of the platforms that we have today didn't exist.

And the CNCF, the Cloud Native Foundation, didn't exist either. So you were like, I wouldn't say a void because that sounds negative, but like you were like in a big sea with no islands, with no towns, nothing, no harbors in sight. And you had to figure these things out.

And that was really challenging. Not because, you know, it was not invented here. It was not that syndrome. It just didn't exist. And the communication was very different at the time as well. So that must have been very challenging. And even so, you build those things, you ship those ideas, you ship the code, and many people benefited in so many ways decades after you started. Like we're 21 years, 21 years later, we're talking about how this started and how relevant it is. And it feels to me like the whole world in a way or the other, the whole software world is revolving around the principles that you set down then. There's a bit more history to it than that. So, you know, there were people doing good stuff before us and open source had been

around as an idea. It wasn't as big as it is now. There wasn't as much choice as there is now, but you know, Linux was around, you know, Linus Torvalds had released that as an open source project considerably before then and so on. A lot of the ideas that we were building on, so ThoughtWorks was an interesting place. There was a brief spell. I feel privileged to have worked for ThoughtWorks at a time that was very exciting. And ThoughtWorks in London in particular, I think was not quite on the same scale, perhaps, but it was almost like an agile Xerox park. It was a place where some interesting fundamental ideas were kind of introduced. As I said earlier, agile at a big scale, agile in a more commercial setting. We were doing it because we thought we could make software more quickly

and better quality software using these techniques, and so it would have a commercial advantage in part. These are the reasons why we're doing some of these things. BDD, continuous delivery, mocking, these are ideas that came out of that office in Thought working there at the time. So that was an exciting place to be. We were consciously experimenting and playing with new ideas and trying to find better ways of doing things. The software industry had been through what I think of as a fairly rough time of trying to industrialize it through the late 80s and 90s, applying techniques that people thought would work to make it more effective and productive,

and they didn't. And the agile movement was a bit of a reaction against that, I think. What I'm trying to say is that we were building on the shoulders of giants. People came that did stuff before. Continuous delivery, I think of as second-generation extreme programming. It's extreme programming, but just with some other ideas added to it and that help you get there a bit easier, more easily, maybe in some ways. But if you're doing extreme programming, you're not doing it wrong. That's a pretty good starting point. I see a lot of the new systems, for example, Argo CD. That's something which fascinates me right now, how it takes the concept of a pipeline

to like a new level and you have workflows. You have a programmable API, which is the Kubernetes in this control plane where you define these custom resources and then things happen. All these relationships emerge between them. Event sourcing, that's like another big, big thing, which maybe it's not as popular these days. I don't know. I'm not too into it, but I keep hearing it. It keeps coming up. But there have been so many CI, CB systems that appeared in the last five years,

which seem to have like exploded recently. There was Drone CI, there was Circle CI, there was GitHub Actions, which wasn't a CI to begin with, but it became one over time. And all these other systems, I mean, Jenkins, I think that came after Cruise Control, I remember. We switched to Jenkins at LMAX.

So we refactored our pipeline to use Jenkins later on. Interesting. So there was like this transition and now we're like in the era of, I think it's almost like the third one, which is the cloud native one, where you have so many projects. I'm not sure whether you've looked recently

at the landscape, at the CNC of landscape, where you have all those projects. There's so much things there and you can't even keep up with all the updates. That's how many there are. Nevermind, try them. It's impossible. There's not enough, you know, days in the week, days in the week yet, hours in the day.

You know what I mean? So I'm wondering how did the cloud native landscape shape your ideas of continuous delivery? Was there an impact of that or was that happening in parallel? What influence, if any, do you feel coming from there? I think that the gestation of the cloud was kind of in parallel with the starting points of continuous delivery. So we published our book in 2010 and that kind of put a name on these practices, I suppose. And, you know, people these days talk about continuous delivery and continuous integration and continuous delivery and all these sorts of ideas. I think we helped to popularize that through our book.

But we were doing it for several years before that. And certainly in the early days, the cloud wasn't around. So all of the projects that continuous delivery began with weren't cloud projects. I think that the cloud makes some of the continuous delivery thinking more obvious. I mean, you'd be absolutely insane to be an organization like Amazon or Google and to manually configure your service. It's such a bizarre idea. It's just laughable. You couldn't do that. You're going to automate that unless you are crazy.

And so ideas like infrastructure as code just seem obvious in the cloud. And they didn't always seem obvious to other people. People thought we were strange when we started automating those things and making our servers in our data centers more repeatable and reliable. There's that kind of stuff. If I'm honest, I don't think that the cloud had a huge impact on the kinds of projects that I was working on during that time, certainly up until after the continuous delivery book came out. Of course, like everybody else, it has an impact on me now in the way that I think about these things and the way that I advise my clients how to approach solving problems.

I am, I suppose, an old school developer. My formative years in software development probably predated the cloud. The cloud is obviously a good thing. It gives us lots of opportunities. It commoditizes compute. But to my mind, it's not a fundamentally defining thing. So it certainly changes some of the dimensions of design that I would care about, the way that I would think about design. So one of the things that seems to me that changes is the economics of the design from an architectural

point of view. In the old days when we were doing stuff in our own data centers, we were probably more worried about managing storage because storage was expensive. And that's largely become commoditized. And the price per byte of storage has kind of just dropped through the floor with the introduction of Moore's law and the introduction of cloud-based services. The big difference with the cloud now is that the unit of cost is really around compute. Certainly, if you start thinking about serverless things,

so that ought to change the way in which we apply design thinking. You know, for example, why bother normalizing data anymore? Why not just shard it out and make everything separate to optimize the compute cycle? You could do that. You could have processes sucking in data and just allocating them out in a way

so that they're all more parallelizable. And I think that's kind of interesting that it has those sorts of things. The easy access to be able to spin up some resource, some compute resource or storage resource or whatever else is fantastic. And the ever-raising of the bar of abstraction that the cloud services add is kind of interesting.

And I think there's much more to come. I mean, one of the things that you were talking about, the continuous integration and continuous delivery tooling, I don't think we're there yet. I think there's more to do. I would like to see tooling that's more opinionated. I would like to see tooling that just, you know, bang, gave me a deployment pipeline. If I follow the rules, it's just going to run my unit test, run my acceptance test for me and deploy it into production.

That's doable, it seems to me. We could do that. I'm hoping to see the continuous delivery cloud vendors do more of that kind of thing. A bit more opinionated. One of my favorite technologies, we were talking about tech earlier on, is Gradle for build systems in the Java space.

And one of the things that I always loved about Gradle is that if you don't care, if you just are willing to buy into its model, you can write your build script in one line. You can just say, I'm doing Java, and it will do it for you. It will compile the Java file. It will run the tests. It will do all of those things for you, all of those things for you.

But if you want to override almost any behavior, you can do that too. It's a whole programming language built on top of this well-designed domain model for builds is what Gradle really is. I like opinionated software. I like opinionated software that says, do it like this. And if you don't like it like that, it gets out of your way really quickly. I'd like to see more tools like that because I think there's a tragedy of the commons kind of thing goes on a little bit. If everybody has a choice, everybody's rediscovering everything from scratch, and I think we ought to be able to build

a little bit more things. And the cloud is one of those things that is doing that in some context. It's not doing it enough to my taste for build systems. So I am currently in the middle for a piece of work that I'm working on to demonstrate how to build deployment pipelines, building a little sample application using GitHub Actions. It's all right. It's nice. I quite like GitHub Actions. It's okay. But it's too fiddly. I'm fighting with trying to get my Docker images

to communicate with each other so I can run my acceptance test and these sort of things. I'd like something that just works. I'm not really interested in that part. I'd like something that just works if I want to do something simple. That's really interesting because I have seen, so these trends have been emerging

and I think build packs came closest to that where it would automatically detect your application and it would know detect your application and would know what to do what is the build step what is the run step what is the the package step so it had like this stuff like like built in i think heroku were the ones that made it popular cloud foundry that was like the enterprise version of that i know that there is like other newcomers renda is one of them fly i, it tries to do something similar. The point being there are some good concepts,

but I don't think they're standardized. The one concept that was able to be standardized was the Kubernetes API. And it's amazing because it's unified API and you can have almost anything via the same API. Do you want, for example, a VM? Well, you're programming in the same API and it spins up a VM.

Do you want a SQL instance somewhere in some cloud? It's the same API. You can start in arbitrage. You can be starting some really, really clever things. There's even like a control plane which controls all the Kubernetes deployments, all that DNS, your CDN, all the things. So I think that is fascinating.

Could we have something similar for CI, CD? And it's like the pipeline, this concept of a pipeline. I think we should. But I don't think there's any one clear winner as to how to approach. It's just YAML, really. And that's okay. I mean, Tekton CD, that's trying to do something.

It's using the same Kubernetes API to declare your pipelines and your inputs and your outputs and how that works. But you're right, it's still fiddly. It's almost like we need the next building block. And I think this is something that happened with the cloud. You had this fixed compute before, which was your CapEx. You'd buy the hardware, you'd invest it, that's it. You spend the money, so you have to use it. But it was very difficult to increase, or very slow to increase. Then the cloud came and you could have almost use it. But it was very difficult to increase or like very slow to increase. Then the cloud came and you could have almost like infinite capacity.

Do you want a thousand CPUs? Two minutes later, you have them. Do you want, I don't know, petabytes of SSD storage? You had it. And then storage wasn't an issue anymore. But it was difficult to scale that down to zero. And that's where serverless came. Serverless, like you can have an infinite capacity that you run for milliseconds and then it spins down again.

And that's like a very interesting take. But you're right. How do you declare the pipeline or the thing that kind of controls all of these things that need to happen? Because shipping the code out there is just part of it. You have tests, as you mentioned, right? You have the builds, you have the dependencies you need to resolve. And that pipeline would be really big. So if you had to imagine it, it'd be massive. How could you declare it? And I think there's like a lot of variance in pipelines. There is, but if I'm honest, part of why I'm complaining and being a grumpy old man about this is I'd like people to take my opinion. Right. Okay. Same as everyone else. Yeah, yeah, of course. But one of the things that I wish people had picked up from the continuous

delivery book that they didn't was that the book outlines a pattern for what a deployment pipeline is. Jez and I each wrote equal amounts of that book. I started off writing the beginnings of the pipeline bits. He contributed on top of it. But when I wrote the pipeline bits, what I meant is that I think this is the starting point for a pipeline. So, yeah, absolutely, you vary from there. But I think it's a bit like patterns. So I think that if you want to write software, what do you want? Well, you want fast feedback during the development phase to confirm that the code that you're writing is the code that you think

it is. And then you want to be able to check that that code works as a system, is deployable, is configured correctly, delivers value to customers. And so there's a separate, different focus of testing that you need to establish that. And then there might be other things that are optional, but maybe performance testing, security, whatever else. And so you could imagine, so my minimum deployment pipeline is pretty fixed. You start off with a commit stage, which gives you fast feedback, asserts your coding standards, runs all of your unit tests. If it succeeds, it builds a release candidate. That release candidate is a deployable thing. You deploy it into an acceptance test environment.

You run a bunch of BDD-style acceptance tests against it to check that it's deployed correctly, works correctly, does what users want it to do. And then you can deploy it into production because it's deployable. That's my minimum deployment pipeline. I would pay money. I would pay my own money to be able to have the push of a button for a Java project or a Python project or a C-sharp project, whatever it was that I was doing, and I can see no reason. In fact, I have built that.

I've built that internally in organizations in the past. I must confess, when I started my own business and working for myself, my ambition was to earn enough money to pay me to have enough time to write some code so I could open source this model like this. And I earned enough money, but didn't have enough time. I think that worked out really well because what you did manage to do was spend a bit more time on those videos that I have alluded to in the past. And that is actually how I came across like, oh, Dave Farley. I've heard that name. Continuous Delivery. Okay. I've heard of that book. I haven't read it, but I've heard of that book. And now I know I have to read it. Based on what you've said, there's some very important information there, which I need to get. So that's the first step. The second step is you are able to capture some concepts in very simple terms, in very good terms. And these concepts stood the test of time.

So that's out there. That's super valuable. And it will continue being valuable for many years to come. I'm sure of it. If someone's listening to this and wants to do this, that would be really interesting. What would a GitHub Actions pipeline look like, for example, that resembles Dave's ideal pipeline? And I think in your videos, you even have that graphic keeps coming up. Do you have a specific course or book that talks more about that pipeline? I do. So the continuous delivery book that we've been discussing talks in broad principles about continuous delivery. And deployment pipeline is kind of the core of the book, but it's not all the books about. I have another book that was released this year on LeanPub, which is more of a focused manual on how to create deployment pipelines and this pattern

that I'm describing, really, I suppose. So what are the key stages as I see them and what should those be doing? It's a pattern. So you would expect it to evolve over the time and to morph into different shapes. So you take it parallel and so on. But there are some, seems to me, some fundamental things

is that you want fast feedback on the kind of technical quality of your work as a development team. And then you need confidence to know that your software is releasable. And the latter involves tests that are more expensive to run. So you need to think about this as kind of a machine, a parallel computing algorithm, if you like, so that you can kind of trade off getting the fast feedback and then moving ahead in confidence that you're likely to get good feedback from later stages. So thinking in those sorts of terms helps you model it. So I think that's a good pattern and I would make that the starting point.

It's almost like a template for pipelines. There was a template that captured these important elements that need to be present. And then from that, it's almost like an IFC. And then from that, you have the specific implementation, which is for maybe a specific CI. And then you have variations of that implementation in whichever CI it is. Yes. And there are key stages that I would expect a system of any complexity would probably want to parallelize and grow to get fast feedback. What people took, I think, from the idea of continuous delivery was when people think about deployment pipelines, I think what most people probably interpret that to mean is basically a build script that can deploy stuff at the end.

And it's much more than that. There's more to it, to the model than that in my mind. This is one of the things that I can be definite about. I'm usually not definite about things, but in this case I can because I invented the term deployment pipeline so I can be definitive about what I meant when I said it. A deployment pipeline goes from commit to releasable outcome. That's its job. If at the end of the deployment pipeline you've commit to releasable outcome. That's its job.

If at the end of the deployment pipeline you've got more work to do, it's not finished. It's not a deployment pipeline. The objective of a deployment pipeline is go to commit to releasable outcome. It doesn't mean you have to necessarily push the thing into production. That depends on the business, whether that makes sense or not. It makes sense to go frequently, but you don't have to.

But working cellular software is always in a releasable state, is the way that I tend to describe continuous delivery. The deployment pipeline is the thing that determines releasability. So that's what we're trying to get to. And if you're thinking about that, so what does that then take? Well, it depends on your system. But at a minimum, you want to know that the software works, does what you think it does as a developer, and is deployable and does what the users want.

As an absolute minimum, you must want to answer those questions in some way before you deploy into production. So that's my minimum starting point. That's where I would start my template from. And then you could optionally plug in bits that would do performance testing, host where you put your performance testing. So part of this is really to address the problem of how do you get organizations to start buying into this? And what you want to do is that you want to make it really easy to do the right things or what you think are the right things and possible to do other things. That's my approach. And so I would like it to be absolutely trivially simple that if you're

willing to accept some small constraints on the way in which you organize your code, then I will be able to build it for you, deploy it for you, run all of the unit tests because you put them in the right place. And then once I've deployed it, I'll be able to run all of the acceptance tests because you put them in a different place. And if all of those tests pass, I'm going to give you something that you could deploy into production. And that's pretty trivial and doesn't really constrain you very much. All you're asking is, tell us what you want to build.

You know, plug your build script in here. That's fine. Build the piece that you want. But tell us where the tests are and we'll report them to you. And now, as a developer, I'm going to press the button, get my template out, start building my project against that template. And my deployment pipeline just starts functioning.

Instead, at the moment, with all of the technologies that I've tried so far, I pretty much have to go through that exercise every time to set something up. This episode is brought to you by our friends at Square. For our listeners out there building applications with Square, if you haven't yet, you need to check out their API Explorer. It's an interactive interface you can use to build, view, and send HTTP requests that call Square APIs. API Explorer lets you test your requests using actual sandbox

or production resources inside your account, such as customers, orders, and catalog objects. You can use the API Explorer to quickly populate sandbox or production resources in your account. Then you can interact with those new resources inside the seller dashboard. For example, if you use API Explorer to create a customer in your production or sandbox environment, the customer is displayed in the production or sandbox seller dashboard. This tool is so powerful and will likely become your best friend when interacting with testing or playing with your applications inside square.

Check the show for links to the docs, the API Explorer and the developer account signup page, or head to developer.squareup.com slash explore slash square to jump right in. Again, check for links in the show notes or head to developer.squareup.com slash explore slash square to play right now. Okay, let's do this. My assumption is that the pipeline that we use to push changelog.com updates out is wrong. So my assumption is wrong. This is what a pipeline is, and I would

like you to tell me what your thoughts are on the pipeline. The pipeline, whenever there is a commit to the GitHub repository, it pulls down the code. It runs a build. By the way, it's an Erlang-based project, Elixir-based project. Cool. So it has to fetch some dependencies, compile the code. So that's the first step.

And then it fans out into two other steps. One of the steps is to compile all the assets. And these are the static files, the CSS, the JavaScript, all those things. And the other one is to run the tests. The tests are mostly unit tests, but also some integration tests because it does use a database, you know, so it's like some hundreds of tests. It doesn't take too long to run, but a few minutes later, it gets to the last stage. If both stages pass, it fans in. So this is a small fan out and fan in. And the last one is to build the artifact that is deployable. And in this case, it's a container image. All those things put together, it takes maybe up to 10

minutes to run. It can be a bit slow, but we won't get into that. The point is that the pipeline ends at publishing this artifact to a repository. And this is like an artifact repository. In production, we receive notifications, oh, there's a new artifact. And the production system, in this case, is Kubernetes. It knows how to pull the latest version down, how to do blue-green deploy, and there are checks which make sure that the new version actually works. It connects to the database, the health checks pass, all those things pass, and then it gets automatically promoted to be the new version. All this happens within 15 minutes. A lot of it is just like slow workers. Anyways, it's a lot of free

infrastructure there, especially on the build side. But within 10, 15 minutes, every commit goes out into production. Yeah. What are your thoughts about this pipeline? I'm assuming it's the wrong one based on your description. I think all of the things that you said, I can't be too critical of it because it's working so I'm nothing if not a pragmatist let me critique it nevertheless yes please so I think what you said I think the implications of what you said is as a developer I don't know that I'm ready to move on to something new until after about 10 minutes when you've run all of your tests okay that. That seems a little bit slow to me. I suppose it depends how big or complicated the

code is. And part of the reason why it's slow is that you're conflating different kinds of tests. So the reason why a deployment pipeline is called a deployment pipeline is weird. And it's all my fault. I'm a software computer nerd. And what this reminded me of when I came up with the idea was instruction pipelining in Pentium processors. So when I say a pipeline, I don't mean a straight line. I don't mean what I mean is an instruction pipeline. And an instruction pipeline in a Pentium processor was a branch prediction algorithm.

So at the point at which you come to a branch in the code, a Pentium processor will start three threads of execution, three processes internally. It will start evaluating the statement in the condition that you're interested in. And it will also in parallel start executing what happens if that condition is true executing what happens if that condition is true and what happens if that condition is false. And then once it's finished evaluating the condition, it will discard the computation that wasn't useful. So it's made progress. It's made progress in parallel with carrying out the conditional. At the point at which a developer commits a change, my recommendation for a way of working is that you

sit and you wait for the results of the tests. And at that point, what I'm looking for is a high level of confidence that if all of those tests pass in the commit stage, then everything else is going to be fine. If my tests pass, I'm going to move on, I'm going to start working on something new with about 80% confidence that 80% of the time, all of the rest of the tests are going to be okay now. Now I can afford to run more slower, more complicated tests because I'm making progress in parallel with executing those tests. And 80% of the time or better, all of those tests are going to pass because I've got high confidence because I'm doing the fail fast thing of testing things. So I'm going to run very fast, very efficient sort of tests in the first stage, in the commit stage. It's going to be focused on

really technical evaluation of what we're doing. Then I'm looking for the deployability of that. So one of the things that you said was that the first time that you actually deploy the software is in production. Yes. It goes straight into production, yes. Yeah. So how often is that a problem? Does it ever cause a problem or does it always work?

Always works. Then I can't critique it. For other kinds of software, though, I want to test the deployment of the system. I want to test that that works because it changes over time. If you introduce a new service or something that's different, then you're going to be evolving it over time. And so I'd like to be able to evaluate those kinds of things too. I'd like to be able to test the configuration of the system. How does it work if I run entries in a thread pool

or whatever else it might be? I want to test all those sorts of things too. So I guess partly it depends on the consequence of things going wrong, how far you take that. These days, I make a living as a consultant advising usually large companies on how to improve their software engineering practices. And one of the companies that I worked with was Siemens Healthcare. So they're building machines that can kill you if they go wrong. These are medical devices in hospitals. And so there are chances that you don't want to take with that kind of software. So you want to be more thorough in

your approach to evaluating those kinds of systems than I would for other kinds of systems. So it probably does, it does vary. So I must say, I can't really critique your approach very well because it sounds very good. I mean, it's light years ahead of probably what whatever average means in our industry. Talking to you, this was really good because even talking to you about how it works i realized why certain layers are so slow why does it take 15 minutes and it's not the tests the tests run in maybe 15 seconds the tests are really fast but it's all the caches of dependencies of pulling things down of running updates of compiling things uh when we run this in the CI, there's a queue. So your jobs may be queued for

maybe 30 seconds or a minute. And you have multiple jobs, you have containers, you have to pull down images that may or may not be on the node or on the host where they run. And all those things, like the cache misses, can mean 30, 45 seconds, which in a big scheme of things is not a lot, but they add up because you have so many layers. What is the impact of something not working? Well, when we deploy into production, the reason why it's slightly slower is because the first thing that we do is we back up the database before running the migration. So there's a full database backup every single time a new version starts. We back up all the assets to S3. So if we lose everything, that's okay. We can restore the whole thing in 30 minutes.

And in front of the app, it's a monolith, by the way, and we didn't have time to discuss about microservices in monoliths another time, I'm sure, is that we have a CDN. I have a good video on that topic. I know you do. We will definitely discuss that next. And in front of the website, there's a CDN, which serves all the content cached. So if the origin is down, if the app is down, that's okay. Everything is cached worldwide.

Yeah. they see the old content, but it doesn't go down. So the uptime is always a hundred percent because it's never down, right? It's distributed across the whole world again. And all that, there's like a complexity in the system, which makes certain things slow. But anyways, I mean, I would love to talk more about this, but we're running out of time. I mean, it just shows how much we have to talk about. I would really like to talk about your YouTube channel next. So what made you start your YouTube channel? By the way, for those really like to talk about your YouTube channel next. So what made you start your YouTube channel? By the way, for those that don't know about this amazing is my favorite YouTube channel right now. It's called Continuous Delivery. It's Dave's new YouTube channel. And week on week, every Wednesday, he publishes a new video. It's

one of the best videos, tech videos that I've seen. They're short, 17 minutes, 18 minutes, but there's so much information there. I highly recommend you check it out. So Dave, what made you start this YouTube channel? The simple answer is it was coronavirus. Finally, there's a positive. So it's something that I kind of had in the back of my mind for a long time. I am approaching the end of my career. I've done a lot of interesting things.

I am opinionated, as you can probably tell from my conversation about software. And I think that the teams that I've worked on have found some things that are worth spreading and worth hearing hearing at least. You can dismiss them. You can disagree with them. That's absolutely fine. But I think when I'm being grandiose, which I sometimes am, when I'm being grandiose, I think that we are on the verge

of discovering what engineering for software might really mean. That is in the same sense as Elon Musk blowing up starships in Texas. It's experimental. It's about learning and discovery and trying out ideas and focusing on the skills around that kind of thing. I think that if people just did that, then they would find a dramatic mind change, experience changing improvement in the way in which their experience of building and delivering software. You genuinely can build better software faster doing these sorts of techniques, I think. So sometimes I err on the side of being too prescriptive about some of these things, possibly. But I wanted to start talking about those things.

And I've been talking at conferences for some years, working as a consultant for some years, helping people to do this kind of thing. And in the back of my mind, it'd be nice to play with a YouTube channel one day. The coronavirus happened. We were in lockdown. And at the time, I was traveling around the world constantly as a consultant. And that kind of fell off a cliff. I was at home and I thought, wow, what am I going to do now? And so instead of writing software, I should have built the system that we just described. Instead of doing that, what I did was I started a YouTube channel. And that's been fascinating, engaging, delightful experience on the whole.

Sometimes some of the comments are not quite so delightful, but usually they are. Mostly they're lovely. And I've had a fantastic time. I think it helped keep me, my wife, and my son saner than we would have been otherwise through the process. We have released a video every week at 7pm on Wednesday since the start of the pandemic, and we haven't missed one yet. Now, I have to thank you again. Again, I find myself thanking you so much because those videos, they were like a breath of fresh air. There's like so many videos, obviously on YouTube,

it's massive. It's like, I think for me at least, and for our families, like the new TV, where we use YouTube way more than anything else. Netflix is there, Apple TV is there, but YouTube by far. And I don't know how it happened, how I came across your videos, but they were so refreshing. They were simple, they were to the point.

And it's not just me. If you look at the comments, the more positive ones, that's what the majority is saying. And it's like the me. If you look at the comments, the more positive ones, that's what the majority is saying. And it's like the way you capture these principles and the way you convey them is so good. And it's like so simple. It's like, yeah, it makes sense. Like at the end, after you watch a video, you're like, oh, I want to try this out. Like it just makes you think.

And I'm sure that some of the information that you convey, it will not hit home until a few months later, or maybe even a few years later. There's like, it's simple, but there's so much there. And my favorite one, you keep mentioning Elon Musk, by the way, if you know him, or if someone that knows him is listening, I really want to interview him because I think he's the embodiment of shipping it. He's literally shipping the human race to a whole new level right i mean i'm so fascinated by him so my favorite video is the space x and software engineering how to learn on your youtube channel the link will be in the show notes by the way now there i try to limit myself to three this was my

top the other one is how to build quality software fast that shipped yesterday i mean if you're paying attention they're videos that you're just publishing which are top so they're getting better in my mind why ci better than feature branching i would love to talk to just about this i'm a big believer single branch push straight into main master however you want to call it i would recommend main the main branch uh if you use git power to you if you something else that's okay too as long as you have a single branch you continuously integrate you continuously deliver that's the place that you want to be in because you're trying to learn and you will be wrong even when you think you're right so better think you're wrong right

and start thinking that you're wrong and it will be good trust not me trust dave because that's what i've been saying okay try it out yeah that's the best one um and what's wrong with the state of devops that's the one that i want to watch again because that's another very good video we don't have time to talk about like the specifics but if anything it's almost like i feel like we should have another interview and we're just finishing this one so i'm not sure how that's going to work, but I would definitely like to get together again, maybe this year. And if not, next year is fine as well to have another, like do another check and see how it's going right now. You had like 53,000 subscribers or 54,000. That was yesterday, by the way, it changes day to day. So let's see how many subscribers we'll have next time but i when i started watching you had like 5 000 6 000 and then just exploded so yes yeah the response has been

positive i hope you're pleased with it because i'm very pleased with this youtube channel and thank the pandemic that it happened right i mean it's a weird thing to say but it's the truth if it wasn't for it we would never have this YouTube channel. Yeah, well, it doesn't compensate for the bad things, but it's been a lot of fun and a lot of pleasure out of making the videos, but also engaging in the comments and talking to people about ideas, which is fantastic. It's all that any of us can do. I'm interested in your selection. They weren't the ones that I expected, to be honest. So they're not the most popular ones on the channel, some of the ones that you've mentioned, but they are ones that I like. I was slightly disappointed by the take up of the SpaceX video because I thought that was a good video. I liked that one. Based on what I was saying earlier,

I was saying that some of the things that you share, I don't think people realize how valuable they are until maybe a few months or even years later. And I mean, I think it depends on experience. It depends on what you value, but I see like, for example, spec SpaceX is such an important thing. Tesla is such an important thing, not the things that they do. It's how they approach it, how they're able to build. I mean, that's what fascinates me. And I know it fascinates you too, because you mentioned it in the videos. So which are your favorite videos? Oh, last week's, of course. Last week's, okay. That was a good one. That's how to build. No, that was this week's. Which one was last week's? I just meant always last week's. Oh, I see. Okay. Oh, I see.

No, there are some that I'm proud of the early ones i think there were some good ideas in the early ones but my editing skills have improved significantly and that my equipment has improved a bit it's still not very professional but it's good enough now that it's not going to it's not going to make people run away screaming i like the spacex. The microservices video in which I talk about the problem with microservices is a good video. I was pleased with the last week's video, which was CI is better than feature branching, which is just talking about an informational system.

I'm trying not to do it in an emotional way. I'm trying to do it just based on information and just thinking about two pieces of information in two places that are both being changed, you know, copies, they start off as copies, they will diverge. And the longer the time they will diverge, you know, the greater the divergence, therefore, the more work to put them back together again. That is incontroversially true. And so continuous integration, continuous delivery is about trying to minimize that time trying to shrink that time down so that you're taking less risk with the changes so there's ideas like that

which i enjoy and i enjoy trying to find a simple way of describing sometimes complex ideas i think that's a very good thought to end on because it's a very profound one. I think people need to think about that. The simplicity in complexity that I think everybody should strive to look for. Martin Thompson, I think he was a bit of an inspiration there as well. So keep improving. Be wrong. Start being wrong. And maybe you'll be right.

Who knows? Nobody knows. Check out David's YouTube channel. It's really good. It'll be right who knows nobody knows check out david's uh youtube channel it's really good it'll be worth your time trust me and david's been a pleasure uh thank you very much for making the time and i'm looking forward to the next one thank you great thank you very much it's been fun that's it for this episode of Ship It. Thank you for tuning in. We have a bunch of podcasts for developers at Changelog that you should check out. Subscribe to the master feed at changelog.com forward slash master

to get everything we ship. I want to personally invite you to join your fellow changeloggers at changelog.com forward slash community. It's free to join and stay. Leaving, on the other hand, will cost you some happiness credits. Come hang with us in Slack. There are no imposters. Everyone is welcome. Huge thanks again to our partners Fastly, LaunchDarkly, and Nenode. Also, thanks to Breakmaster Cylinder for making all our awesome beats. That's it for this week, see you next week. Game on.