PurePerformance - Bad Software Engineering killed Cyberpunk 2077 Release – What we can learn from it with Dave Farley

Episode Date: February 8, 2021

If you are not a gamer you may have never heard about Cyberpunk 2077. If you are – you may know about the challenges during their latest release.Dave Farley (@davefarley77), Co-Author of best seller Continuous Delivery, has been an engineering large and complex systems for decades. His work helped elevate our industry around Continuous Delivery and DevOps. In this episode he shares his learnings from failed projects like Cyberpunk as well as his own latest experiences around that picking the latest technology might be fashionable but is not always the smartest choice.To learn more about Dave check out Continuous Delivery website that also links to his YouTube Channel hosting some of the episodes he was referencing in the podcast.https://twitter.com/davefarley77https://www.amazon.com/Continuous-Delivery-Deployment-Automation-Addison-Wesley/dp/0321601912#ace-g9859629705https://www.continuous-delivery.co.uk/https://www.youtube.com/channel/UCCfqyGl3nq_V0bo64CjZh8g

It's time for Pure Performance! Get your stopwatches ready, it's time for Pure Performance with Andy Grabner and Brian Wilson. Hello everybody and welcome to another episode of Pure Performance. Actually, it's the 125th episode of Pure Performance, according to my count. That does not include all the perform episodes and one-offs, but the 125th actual episode. So whoever's listening is here for a milestone. And obviously one of the people who's with me, always is my co-host, Andy Grabner, who's a little hairy today. Hello, Andy.

How are you doing? Well, it's unfortunate that we don't record the video for our podcast, but yeah, I'm actually kind of proud for my facial hair. After two weeks, I finally grow a beard and the color is also reflecting the color outside because it's snowing right now in Linz and I get a little whitish beard here. So. Nice. Yeah. And I could have asked how you're doing and you could have responded. I don't know. It's been a little hairy here. Crazy, you know, but 125 episodes. It's amazing.'s three three years almost no we've been going for four years i think in fact um my wife's cousin's husband ted who's a fan of the show i'm not sure if he still listens

regularly but if you do hey ted um their mom makes us a count my wife and i calendar every year with our birthdays and all and on one of them it has uh the the anniversary for pure performance so yeah it's been going on for quite a while now yeah it's uh you know it's great even though it's a challenging it was a challenging year last year but uh we're we're strong and we're back in 2021 we do our best to make the year a good year yep and we have great guests as always yeah Yeah. And talking about guests, it's a great segue. I was actually really excited that I was invited to speak at the conference called Scaling Continuous Delivery. And the keynote speaker of that conference is our guest today, Dave Farley, who is talking about scaling up. And this is

really, it's an honor, first of all, to speak at the same conference that he's speaking. And then also, I know, Dave, you're always busy, I assume, because you do a lot of things. And therefore, it's really great that you find time to get on this podcast. Now, for those people that, for whatever magical reason, have escaped what you're doing or what you have been doing, especially for the continuous delivery community, maybe you can quickly introduce yourself to our listeners. Sure. First, thank you very much for inviting me

onto the 125th podcast. And I like your beard. I think it looks good. So, yeah, my name is Daveave farley i'm a software engineer by profession software developer by profession i spent most of my career building what i think i was largely kind of the complicated end most of the systems that i built have been large scale distributed systems in the last 30 years or so and um i was author i'm also the co-author of a book that's became quite successful called continuous delivery with jess fumble and i think if people have heard

of continuous delivery and not me i'm probably part of the reason why they've heard of it because because i think continuous delivery was a very successful book in that respect um what i do these days is i work as an independent consultant and offer my the windmill that i am tilting at is that i am trying to figure out how i can help people do a better job of building better software faster and i've got a youtube channel and i write books and do conference talks and act as a consultant and all of those things really are focused on trying to do that thing yeah very cool and you are very humble in in what the way you you explain uh your impact on the industry and you can see there's a word punt on chess humble you're very humble right i guess i'm not sure if this actually makes sense i'm not native speaker in english but i just make up things sometimes but um so dave

i have to say that when i did the research to this podcast initially i thought well scaling up sounds like an interesting topic that's the the title of your talk it continues delivery but then you also pointed me to your YouTube channel and I started watching it. And there was one episode recently that you recorded and I think that became hugely successful. It was around the cyberpunk release. And I think there, I really thought it was very interesting. The things that you explained that you walk through, especially around planning challenges. I thought that was an interesting point of what can go wrong with planning that then leads to catastrophic failures

like that release. But maybe instead of me repeating what I've learned from your show, especially on the Cyberpunk release, can you just quickly fill us in? How did it happen? And then what did you learn, especially from it?

Sure. I think there's a couple of different angles from that. So as you said, the video on my channel was surprisingly successful. My channel was doing well. It was growing better than average. I've been doing it since the pandemic started, basically. A channel on YouTube focused on software engineering in the context of kind of DevOps continuous delivery, those sorts of practices, is primarily what it's about.

But I thought that when the cyber, the problem with the Cyberpunk 2077 release was announced, I thought it would be interesting to just try and analyse that from publicly available sources and just if I could kind of see what I thought went wrong. And I came up with some things, and really I used the case study of the cyberpunk failure to look into what was observable about the way in which they practiced software development,

which is what I mean about when I talk about engineering. So one of the things that I've learned, so the video ended up being remarkably successful. We were just chatting before we started. When we started out recording the video, I was chatting to my wife and my son, and we thought that maybe we might get 10,000 views on this video. It ended up at the moment it's at 424,000. And I've gained significantly more subscribers to my channel and all of my other videos are being watched more, which is all great from my point of view because I want to get my message out there.

So the Cyberpunk video was interesting. One of the things that I thought was really interesting was the feedback that I got. So one of the bits of the commonest bit of negative feedback that I got in the chat section for that YouTube video was that it wasn't the engineering that was at fault, which was the theme of my video. It was the management.

And I think that's missing a trick because when I talk about engineering and thinking about engineering, engineering is the process of creating something. It's not the thing itself. And I've got, so if anybody's interested in my channel, I've got a great, what I think is a really good video coming up tomorrow, which is about looking at the positive example of engineering, SpaceX

in this case, and what we can learn from it. I think engineering is a really interesting discipline and an interesting challenge, and we don't really do much of it in software development. So my thesis is, how do we learn to be a bit more like engineers, applying scientific rationalism to solving problems in software? The take on the planning thing from the cyberpunk point of view is what seemed very evident to me looking at it from the outside and from publicly available sources was that they'd attempted to fix time and scope in planning. So there's the classic planning triangle, the iron triangle, sometimes people call it, of

planning scope, and people usually say resources, but they mean usually human beings. So, you know, you can go faster by changing one of those axes, you know, or two of those axes is usually the argument. But you can't really go faster by adding more people we know that we've known that since the 1970s fred brooks famously said you can't make a baby with nine in a month with nine women you know there are limits to how parallel you can go in solving a problem um so so just adding more bodies or more people to solving it doesn't work. So that leaves time and scope. So you can either work to a fixed time, in which case you don't say what's in it.

You work so that your software is always releasable constantly. And then at the point that when you hit the deadline, you release whatever it is that you've got at that point. Or you fix scope. You have a target for what it is that you've got at that point. Or you fix scope. You have a target for what it is that you want to achieve. You build that thing, and only when it's done do you say, I've got this thing, and then you release it. That's interesting.

That's the approach that Apple tend to take with their products. Mostly, the vast majority of Apple products aren't pre-announced. They try as hard as they can to keep all of their stuff secret so that it gives themselves the freedom to get stuff wrong. There have been rumors, for example, about Apple Glass for at least the last three years, their VR, AR product, which has been in gestation for a very long time. And all of the signals are that they're working really hard on it. But there's nothing publicly announced because

they haven't got it right yet. And we won't know about it until they think they have. So these are really different ways of working. And I think interestingly, contrast, my thing is this scientific rationalism is to you know engineering is about you know facing up to reality and dealing with reality as it is not trying to invent some kind of magical thinking you know your wish of what you would prefer reality to be and the reality is is if you're planning a large creative act that is a reasonable description of software development, you can't fix time and scope. That's just stupid. So you've got to figure out which one you want to do

and you want to work within the constraints that that choice offers you. So let me ask. My previous job, their approach seemed to be to reduce or shorten the timeline and add scope based on contracts they signed after the project started. So I think that's not a good thing to do. It's not a good thing. How did that work out for them? Well, you know, I always go back to the Parts Unlimited release. I think it's this stupid way.

And you can understand it. You can completely sympathize with the desire. It would be lovely if I could accurately predict how hard a problem in software was going to be and when i'd be able to give you the solution that would be lovely if i could do that i'm pretty good at software development if you'll forgive me boasting but i don't know how to achieve that that's not possible so instead i'm going to work to to optimize either time or scope and either one of those i can do i know how to optimize either time or scope. And either one of those I can do.

I know how to do either one of those. But, you know, there are trade-offs. And I think that's interesting because I think if you look at engineering, my thesis, my thing, my passion is to how would we get to improve the quality of engineering, and I mean that in the strictest definitional terms, in software. And so I'm currently writing a book on this subject, so I'm quite thoughtful about what engineering is at the moment. And I think if we started to think in those sorts of terms and face up to the realities, then in engineering,

there's nearly always trade-offs. If you're building an airplane or a space rocket there's a huge trade-off between the strength of the structure that you build and the lightness of the structure because any flying machine lightness is a key attribute you know it needs to be both strong and light and so you can't you know the the the stronger that you make it the heavier that you make it that the heavier that you make it, that means the more proportion that you need, that means the more fuel that you need, that means the stronger you have to make it,

and there's this nasty negative spiral. And engineering is nearly always like that. There are nearly always these balance, these different values that you have to trade off against one another. And I think that's important. And time and scope is kind of one of one of those classic trade-offs that we need to we need to think about and worry about and face up to rationally to do a good job my preferred approach in general is to work so that my software is always releasable even

from day one from day one i'm going to try and reproduce something that's releasable it's not going to be useful yet but conceptually that feature is complete to production quality with no more work to do that's what i mean by releasable and then i'm going to do that and do that and do that until i've got enough stuff that's going to be useful to people and then i can release it yeah now let me ask you a question then i assume with the cyberpunk release the reason why they had obviously a trouble with time is because there is obviously after engineering, there's other people like the business, the marketing, that probably

it's a big production. They probably have made announcements and promises to their user base that they will have something available by a certain time. I'm sure this takes also time on that creative side to get this part of the business to get things out.

And so my question is, we've been, I think over the years, really try to get good from the DevOps movement to figure out how we can do this continuous delivery, having always something that is deployable, automating as much as possible. But it seems we have not done a good job in including the business in the whole process so that they also understand that we are we are basically we have a moving target right because

there's always also problems that's coming along the way yes we try to do our best but obviously they also need to be included in this continuous process and also need to get more flexible but then the question is how flexible can everybody be in the value creation chain that goes all the way out to buying ad space? I don't know. Nowadays, we don't need to print CDs anymore or ship them, but still, right? I mean, there's something in once the software is done until it's in the hand of the end user and there's a lot of other parties involved. So my question is, is there anything we've learned from DevOps to extend it to BIS? And I don't want to say BIS DevOps now, but I'm saying BIS DevOps.

Is there something we've learned on how we can also include the business side so that overall we're not just becoming better in delivery, but really in bringing things to the user so that they are not disappointed because they were promised something that we couldn't deliver. But I mean, hopefully I make my point clear. Yeah, I think that there is, and profoundly so. I think that if you look at the organizations that we would all kind of think of as examples of world-class performance in you know software whatever that means you know whatever in whatever context um I think that one of the things that characterizes them more than anything else is that they don't look like other organizations

they don't look like regular organizations and i think there's really really important profound reasons for that forgive me for being slightly philosophical for a moment but i think that one of the interesting aspects of the impact of the information revolution on our society is the way that it's challenged existing structures and institutions and thinking. And that's absolutely true of business thinking, business structures, big, you know, organisational thinking. I think that through, so in the early part of the 20th century, there was a lot of work done to kind of formalise production processes, with kind of production lines and mass production and all those sorts of things.

That really, really, you know, it wasn't invented in the 20th century, but it really came into play at the early part of the 20th century, those techniques. And so that became kind of the defining characteristic, the default go-to place that people thought about when they were trying to structure an organisation. What happened with the information revolution and software software is different software software presents a bunch of challenges and that you don't face when you're building widgets building widgets produces a bunch of challenges that you don't face when you're building software

and so the same techniques don't work for software as work for building widgets um and so i think one of the biggest mistakes is that we've tried to force fit widget building techniques to building software simple example we don't have a production problem if i i i have a cup of coffee in front of me and the design of the cup is kind of interesting and all that sort of thing, perhaps. But the hard part of building a cup like this is the manufacturing, the production engineering, to be able to mass produce these to a certain quality, to a certain price. Imagine all of the logistics to get the materials just in the right place so that you can build these cups by their thousands or maybe even millions.

You know, that's always the hard part of the problem when we're building physical things. We never have that problem writing software. We can build the world's most complicated system and at the push of a button, unless we're stupid and do something dumb, at the push of a button, we can almost for free clone all of that stream of bytes

that represents that system. So our production costs are effectively nil. Our problem is never a production problem. And that has some profound implications. It changes the way that you think about it. Now, going back to thinking about business organizations, if you wanted to be effective as a business then the way that you should do it is that you kind of think what you want to do

and then figure out about how you're going to do it and then organize yourselves to be able to do that thing do you know work that way and that's how almost no traditional organizations work. The way that nearly all traditional organizations work is that they have an organizational structure that was created and embodied decades ago, and then they work within the constraints of that organizational structure. That's not what Amazon do. That's not what Google do. It's not what Netflix do or any of these big organizations. They do things differently. So I tend to use the terminology. I tend to not use the terminology DevOps kind of for this reason

because for me at least it narrows my thinking too much. If I'm talking about DevOps, I'm thinking about the relationship between dev and ops, and that's deeply important. And I love the DevOps movement and the people involved and the ideas. I just don't like the terminology. My preferred terminology is continuous delivery because I think that's a broader concept. So if we think about continuously delivering value into the hands of users, what does that take? It's not just about the software creation.

It's about the flow of ideas, the prioritization of ideas, the choice of ideas, the ability to go into production, see that an idea is bad, say we're not going to do that anymore, and do something different. All of that, for me, is part of this continuous delivery model and so if you want to optimize for that i i i make a large part of my living consulting with usually reasonably large organizations and what i say to them is that if you want to be good at this stuff you've got to take into consideration technical performance. You've got to be really, really good technically. You've got to take in cultural performance.

You've got to be thinking the right way, applying the right kinds of, the right mindsets to solving these problems, and organisational performance. You need to structure organisationally to minimise coupling, reduce the dependencies between different parts of the organisation. I don't want to be sitting waiting for you to finish a piece of code or vice versa in order to release we want to be able to you know release more effectively more independent all of those kinds of things and that's what it takes those are the kinds of ideas that it takes so i think that deeply

profoundly this changes the way that you think about not just tech but businesses and i i've been doing a series of talks to small groups of cto kind of people uh recently which is about this this kind of idea of economies of speed how do you you know how do you maximize your advantage as a business by going faster so i like what you just said with the decoupling. Have you read, I'm pretty sure you've read the team topologies. Are you aware of that book? I also thought that was pretty interesting on how they were also relating, obviously, the way you structure your teams,

that this also then have an impact on the architecture. I think this is a common non-entity anyway, that we often make the mistake that our software architectures reflect the business structure and obviously if we have a business structure that was inherited or invented decades ago then obviously we won't get out of the of the thinking and we will just have a software that reflects the architecture I am I first I love I love the teen topologies book i think i think it's fantastic and if i if i forget remind me to come back and talk about platforms

platform team but but i think it's i think it's a great book but the um um i'll just if you'll forgive me just plugging my channel again i've got a video on building big software with small teams. So decoupling. I am a self-confessed techie nerd, and so I view the world from that perspective. And so one of the things that I think is absolutely fascinating if you think about software and information systems, you know, at all, is that it's kind of true of everything some of the stuff that we think about as being software things are really just information

things and that's true of the world the universe you know physics kind of thing it's some of this stuff's quite deep so one of the things that i think so i think what one of the kind of thing it's some of this stuff's quite deep so one of the things that i think so i think what one of the kind of ways that i try and capture that is i think there are two genuinely hard problems in computer science one of them's one of them's concurrency and the other one's coupling and concurrency is mainly difficult because of coupling. So coupling is this hugely difficult problem. And that's just as true of organisations as teams and teams as it is of the software that they build. So we want to be able to focus on building software that is appropriately coupled and gives teams the freedom

and the independence to make progress in their part of the system. But we don't want it to be so coupled that they are having to, you know, it's fragile and if they make a change over here, it breaks something over there. All of that stuff's really important. And I think we often don't talk enough about the challenge of managing. I'm kind of old.

I've been doing this for a very long time. And I've come, I think, if I was to try and get, you know, capture the thing that I've learned most profoundly is that good software development is nearly all about managing coupling. At every level of granularity. It's true of the finest detail in writing a bit of code and building an enterprise system or or an enterprise to be able to build a system yeah because decoupling obviously decoupling

in the end also gives you resiliency right if you decouple that means it gives you resiliency on the finest level of element or entity that you build if you do it right yeah and i think if we then spend this further from the technical side, how we build our architectures and we can also decouple the outcome of what we build with the business. I think that that's when we are truly there where modern software organizations are. That's why I like so much what you said earlier. And maybe again, a shout out to organizations that are still trying, that are still struggling and trying to figure out how can

they become true software organizations. In order to become a true software organization, I guess you have to, or how did you call it again, an information organization, that you really need to look at those companies that have been doing it from the start really well. The Googles, the Netflix of the world, they not only develop differently, they operate, the organization is differently structured. They release differently. They make business announcements differently, they plan differently.

And that's why a lot of organizations that are now in the need to transform into the digital age, they will only be successful if they throw away a lot of the stuff that has worked for them over the past decades. But now it's a different age. We are in the digital age, we're in the information age, and we need to rethink everything from end to end, from the value creation process. Yeah, I think that what we are talking about here

is incredibly challenging for an organisation because it is a genuine paradigm shift. It's a genuine changing perspective. And the hard part of a paradigm shift is that it means that the rules of the old paradigm no longer apply. So going back to the stuff that we were talking about cyberpunk 2077 and all of all of that and planning um the the agile paradigm can help you to work towards the dates but it's largely optimized at being able to do the

kind of continual drip drip drip drip drip releases that allow you to always be releasing things. And so it's better at managing, it's most effective, I think, most efficient from an organisation's point of view, I think, when you are just working optimally to release frequently because that allows you then the shortest distance from idea to something useful. I think working to a time, you can use agile practices

to hit a date and all of that kind of stuff. But why would you want to do that if you could have it sooner? It's kind of less optimal. And that's kind of a bit of a mind hack for many organizations. They want to be, oh, yeah, yeah, but we've got to have this release date. Going back to the thing that you were saying about, you know, what about the marketing campaigns and all that? It's a choice.

The most successful organization in the world, business in the world, doesn't do that. So it's a choice that we have, that an organization has. It's a tool that we have that an organization have it's a tool that we choose and we can decide how to apply that tool yeah and so if i if i can quickly repeat what i think i'll just learn if you are focusing on scope that actually makes you also think about how can you break down a big let's say end goal that you have, like best software in industry XYZ in the world? How can you break it down into small increments and then basically define small scopes and then continuously release that? And then at some point you'll see, hey, say we are

halfway there and it's already, you know, it already provides value to our users. So let's start some marketing rumble about it. But yeah, I think that's... Yeah. I'm curious in the gaming world, is continuous delivery a viable option in the gaming world? I mean, I'm not a huge gamer. I know there's story-based games

and then there are things like Minecraft and I don't know if Second Life is still going or things like The Sims where there's really no point to the game except to just walk around and explore and hit things, right? Where a lot of other games actually have a story, have goals, you have to do X, Y, Z and all. So with this idea of continuous deployment, continuous development and all that, how does that fit into the gaming world or how could they adapt or adopt to better take advantage of this because

i can't imagine putting out a viable game when it's not done because then people will be like yeah it's kind of cool but now i played through this mission it was a half-assed mission i'm not going to go back and do it again now that the other features are there i have any thoughts on that considering you know in context of this yeah yeah absolutely so so so the first thing that i should say is that a confession i haven't written uh what in the 1980s we would have called a video game probably since the 1980s or 90s it's chip challenge i wrote a few that's how i started out writing space invaders and stuff like that for microcomputers but so i've done those sorts of things, so I know what it takes. I've done quite a lot of graphics programming,

so I know what's involved in that, but I've never been involved in writing a kind of modern AAA game, so I can't really talk to that. Practically, there are examples of modern AAA games that were developed with continuous delivery, and the developers that worked on those teams shout about it and say how wonderful the experience was compared to regular game development. So practically, it's possible. I want to think about it, though, from a slightly different angle, is that I'm not sure it's very different to any of the software.

It certainly has some different challenges. But let's think for a minute of something else that's kind of not as graphically intensive as a game, but graphically intensive, a rich web application of some kind. One of my friends, Gojko Adzic, he has a product that does mind mapping. So it's all visual. It's called MindMup, and you just can draw mind maps on a web page to represent your ideas and grow your ideas and all that kind of stuff.

And it's very successful, and that's developed using continuous delivery. It's not as graphically intensive as a game, but it's graphically intensive. And the way that you approach those kinds of problems is so first you want the architectural separation so you're you were talking about the different kinds of behaviors that you have within a game there's going to be what you might think of as the domain logic the the the uh the simulation of a game world in which you that you're going to inhabit um and from you know from that you're going to be rendering pixels so if you can kind of you know and that's not

really significantly it's different in scale but it's not really significantly different in the division of behaviors from writing a regular web page. And you've got a DOM and you're rendering pixels on a DOM and those sorts of things. So the sorts of techniques are more challenging at the scale of a game, but not fundamentally different. So we could imagine building, you know, our game simulation, you know, our virtual world in abstract, which you must do to implement a rich 3D game,

and testing it in simulation without painting the pixels. You could test that all the logic was coherent and that when you killed somebody, they were killed. When you hit something, it was hit. And all of that sort of stuff, you don't need to actually paint the pixels to figure that out. And then you could imagine testing the rendering part of it separate from that that's that's mostly going to be the province of these days of you know unreal or you know one of those kinds of big engines something

like that anyway so you can imagine you know testing the the logic outside of the pixel painting that's that's one route but even. But even then, you could still think about techniques for applying this kind of thinking. And it would be challenging and it would be difficult, but the difficulty in the games industry, I think, is first the economics. I think there were some unpleasant things. If you look at some of the stories that come out of the company that builds Cyberpunk, they were what they call

in the games industry crunching for over a year, which meant working kind of 60-hour weeks and all of this kind of stuff for more than a year. That's not a pleasant working environment. That's not going to come up with something creative if you're doing something like that, working like that. So there are those sorts of challenges, which are too common in the game industry, I think.

But also I think there's a lack of belief that they can use these techniques. I think that if you went in thinking that that's how we should work, then you'd find ways to solve those problems. Part of my background is working another place where people said it wasn't possible to do this kind of development. Part of my background is in low-latency finance, high-frequency trading systems, building ultra-performance systems.

And we built one of the world's highest performance financial exchanges from scratch using these kinds of techniques different set of challenges right but but i don't think in terms of the scale or the complexity of the system there's any difference so they're different challenges but they're comparable challenges in some way i think so i think that there's some kind of deeper principles. You start off thinking that if I'm writing software, I need to know that the software that I'm writing is the software that I'm thinking that I'm writing.

So you need techniques like test-driven development, you need techniques like continuous integration, and you need to be working to a completed, you know, releasable thing, and that's continuous delivery. Building an exchange is picking up on one of the other things that you said was that building an exchange is rather similar to a game like that in that you're not allowed to really release

half an exchange because the regulators don't like it very much and the people that put their money in don't like it very much. Continuous delivery is working so your software is always releasable. It doesn't mean that you have to release. Continuous deployment is if you deploy it into production frequently and you get different lessons from that. It's valuable.

It's really useful to do continuous deployments. It's not necessary to get the value from the technical quality of the things that we produce, though. Continuous delivery gives you the technical quality. Continuous deployments allows you to get feedback on whether your products are right. Great point. Hey, Dave,

I think in the beginning I said scaling up your talk at the upcoming Scaling Continuous Delivery Conference would be something great to talk about. I assume that looking also at the abstract of that talk, we covered some of this already because you actually talk about Fred Brooks book, the mythical man month and so on. Now this is why I want to pivot over a little bit in, in the time we have left because I also listened to some of your other talks.

I know you're a big fan of the state of DevOps report. However, in the last report, you had some points where you were not, you know, we had some, some critical feedback, let's say that way, some good positive feedback. But there was a term in there, and you reminded me earlier, if we have time, let's go back to this, which is platforms and platform teams. Because the State of DevOps report also says,

because I'm using this also in my work a lot, it says organizations to really become successful have to think about building platforms that enable their engineers with self-service capabilities around different disciplines, whether it's around CICD, whether it's around standing up environments,

around monitoring and alerting, auditing, all these things. And they're all run into challenges like time is a challenge, technical skills are a challenge, the lack of challenges like time is a challenge, technical skills are a challenge, the lack of having things standardized is a challenge. Now, you especially and many others around you have been promoting all of these things over the last couple of years.

I mean, why are we still figuring out how to do this right? Because it seems a lot of organizations are still struggling. What can you tell them? Also maybe reference a little bit the platform aspect because I think you have some opinions there. Why are we still in the state of talking about continuous delivery and educating people? I think this is a complicated thing to change.

If you think about what it takes to be able to effect the kind of change that we've been discussing so far this afternoon, it's all-encompassing. It changes the way that we think about structuring organisations, the way that people working in those organisations work. An individual working as a developer or a tester or a product owner or whoever they are, it's going to change the way that they work and think about their work.

That's really, really difficult for human beings to do. That's a really challenging thing. So it's going to take a long time. It is happening. So my perception, I'm the victim of my own filter bubble, do that's a really challenging thing so it's going to take a long time it is happening it's so my percent i'm i'm the victim of my own filter bubble like everybody else but but my my perception is that it's significantly happening a growing fraction of software is built using these techniques and there's really really good reasons why it's because it works better than anything

else that we have been able to know or understand that you know that we for the first time in the history of software development we have data that we can point out to say if you do these things you are you are more likely to get a positive outcome we've never had that before really in terms of real sociological data that's why i care so much about the State of DevOps report and critique it. Now, there's a whole bunch of things that are problematic to achieve that. I think there are some things in our industry that are challenging. I think that our industry has not done a very good job of learning. We've not, you know, we've made some kind of haphazard progress

but i mean i'm writing a book at the moment which i'm hoping to will be released later in the year about the discipline of software engineering what engineering means for software and as part of that book i'm it's kind of a slightly jokey exercise, but I tried writing a simple CRUD application using modern frameworks and technology. I picked Angular and I just wrote a CRUD application. And then I wrote exactly the same application in tech that was available in 1996. That allowed me to use Java servlets,

Java HTML, CSS, JavaScript. And I reckon that the old version is probably about a quarter of the code of the new version. Now, there's things that the new version gives me. There's things that the new version gives me that the old version didn't, for sure. There's certainly improvements in the platforms that have happened over the time.

HTTP and HTML is a better standard now than it was then. All of those things, I'm not trying to prove that. But I'm just trying to highlight in the book that the learning that we think, the progress that we think we've made isn't always where we think it is. Yeah. And part of that is that we tend to make decisions on the basis of um if you'll forgive me if the audience will forgive me sort of fashion uh rather than a real

decision let me just a little aside this week's video on my youtube channel which i commend you to go and look at is using spacex as an example of great engineering and what we can learn from spacex spacex um made a decision in 2019 to switch from building their starship from carbon fiber to building it out of stainless steel they did that on the basis of um some engineering decision making so they first looked at the cost. It was costing them $300 per kilogram to build it out of carbon fiber, and it was going to cost them $3 per kilogram to build it out of stainless steel. Then they looked at the temperature difference. At above 300 degrees centigrade, carbon fiber starts to soften a little bit and it loses some

of its strength and so for heat shielding for re-entry and all that kind of stuff they needed to keep the surface of the spaceship below 300 degrees stainless steel 1500 degrees and then at cryogenic temperatures carbon fiber becomes more brittle at cryogenic temperatures as is aluminium the other alternative. Stainless steel doesn't. When's the last time you heard a software decision that sounded like that? For anything. When's the last time you heard anybody choosing,

making a decision about what they were going to do in software that sounded even vaguely like that? We don't do that kind of reasoning. We don't do that kind of thinking. And the other reason why I value the state of DevOps stuff so highly is because that gives us a measure, a basis on which we could start to do that. The measures of stability and throughput that is the foundation of the state of DevOps report and the Accelerate book

give us that benchmark in which we can evaluate those kinds of ideas. What's the quality of the stuff that I produce and how efficiently do I produce it? That's pretty good stuff. And Andy, that's a topic that comes up quite often with our guests is the idea of, you know, don't pick the latest technology because it's the latest technology. Think about, as we even said at the very beginning of this episode, is what is it that you're trying to accomplish and then fill in the gaps from there? And that's how you're going to pick your platforms. Everyone's like, oh, Kubernetes and serverless. Well, why?

Yes, there's a lot of great benefits to it. How do they apply to what you're doing and how are you going to take advantage of them? But then let me ask you a question. This is just a theory now on my end. You used the word fashion, right? It's a fashion that people go to Kubernetes and serverless and everything. Might it be also psychologically easier

to make a popular decision where you know everybody goes to because you know if you mess up, you were at least part of a group and all of them made the wrong decision. It's like whether you like the analysts, yes or no, the software analysts that say, this is the tool that you need to buy in the enterprise

because they're on the top right of the quadrant and obviously you'll decide for that because as an executive you want to make a decision and if you fail you can at least blame it on somebody else is that also maybe part of the thinking where you say well we don't have to i mean the one of the reasons why maybe so many people go towards the latest and greatest because everybody's hyping these technologies and so everybody thinks this you have to be there because that's the future and sometimes you actually are much better off with the technology of the past and to and to be fair mostly it's not the future occasionally it is you might hit on the right thing that is the future but mostly that you know there are lots there are more dead ends the other thing that we're not good at as a result is that we're not good with discarding the bad ideas.

There's still people, you know, I don't want to give your podcast flack, but here's some clickbait for you. There's still people writing C++ arguing that Vi is the right editor. How on earth is Vi ever the right editor when there are tools that can, you know, syntactically analyze the code that you're doing and trace so you can do a rename. Do a rename in Vi. You've got to use grep. It's ridiculous. That's also why I think you brought up a metric

earlier where you said the code that you wrote in Java was only half, but I think that's only one metric. You need to obviously not just look at one metric. In the end, it's about efficiency, how efficient are you with code changes and all that stuff. But yeah, I get your points. There's clearly some things we can all agree on that the future is better than the past, but then... So my thing is, what would it take to allow us to make decisions more rationally?

So I think you're absolutely right. In the 1990s, there was a very common phrase. I think it probably predated the 1990s, which was you never got fired for buying IBM. In a big organization, if you just chose IBM, even if it went wrong, nobody would blame you because IBM was the default answer for most things. And we do the same thing. Everybody know, everybody at the moment is building systems that are microservices. One of my friends is Sam Newman, the person that wrote the book that defined microservices. And he says, don't start by building microservices. That's dumb.

So, you know, we need to be more thoughtful about these things. We are an industry that tends to work on fashion and soundbites, and we need to move away from that. If we want to make decisions that really have an impact on the effectiveness of the organizations in which we work and the products that we create. And that's what organizations like Amazon and Google, they take it more seriously than that.

It's not just about we're more choice in that sense. They're going to do a little bit more. They're going to apply a little bit more of those cost-benefit analysis and not jump on the latest fad necessarily. I have one last thing I wanted to actually get your feedback on and kind of get a validation or tell you a story and then I'll see what you say. One of the projects that I'm personally involved in is the open source project Captain.

Everybody drink. Let everybody drink. Everybody drink, that's right. There's a joke between us. Whenever I use the word Captain, you can have a sip of whatever. I knew it was coming, Andy. So Dave,

we're part of that open source project. And what is interesting, when we launched thatatrace so we build a set of tutorials and samples on connecting jenkins uh letting jenkins deploy into kubernetes and then setting up dynatrace monitoring all that stuff and eventually out of that evolved uh the open source project captain where we basically said it's really hard to integrate all these different tools because captain calling this tool i'm sorry j Jenkins calling this tool, then the other. And if you want to switch a tool, you have to touch all of your different pipelines and figure out how to change things. And so we evolved the whole thing into an orchestrator, basically, that is orchestrating

processes around continuous delivery and also operations that can then orchestrate, let's say, delivery with automated testing and quality gates and promotion by sending events and then trigger maybe Jenkins or Spinnaker or whatever. But it was basically an orchestrator. Now, what we did in the very beginning, we also picked the latest and greatest technology and we ripped it out after a year.

And we've kind of reinvented the whole thing multiple times because in the beginning we made the mistake, let's use the greatest that we see on Twitter and this must be the right thing. But for us, it was not the right thing. So now we are at a stage after about two years where we have the architecture right. And also what the other thing that we learned,

we always were so proud about that it is an event-driven architecture that we have. However, an event-driven architecture is not the benefit for the end user. If I tell you, if you're looking for a new continuous delivery system or a new orchestrator for delivery processes, if I sell you event-driven, that it might excite you because you're techie. But really, what's the benefit? And so what we've been doing is, I think the real value

that we have figured out over many iterations is that we're separating the process definition from the actual tooling, from the actors, and we're using eventing to connect those. Now, my question to you now, after all this talk and mentioning the word captain so that Brian can drink more and more, is you have been very active, obviously, in continuous delivery. You just mentioned not always look into the future. I mean, it's good to look into the future, but don't always take the latest and greatest. Maybe you can go back to some old technology. Do you though think, even after all the talk about

organizational cultural changes, that we have some tool deficiencies or that there are certain things we also have to do on the tooling side to enable organizations to get better, to automate more? Or are there already some tools that you have been seeing out there that just make it easier for people to deploy, to get feedback, to do their experiments. Is there anything that you see? I think I'm going to answer that question in a way that maybe is not what you intended when you asked it. So forgive me, but I think, I think that, so, so the the first i'm a software developer i use tools software

tools all day every day and they're important to me but i don't think that that's what makes me a good or a bad software developer i don't think the tools matter if you were If you were hiring carpenters, you wouldn't ask them, what kind of saw do you use? You'd ask them, what kind of chairs do you build? Or what kind of windows do you make? You're interested in the product, not the tools. And I think that one of the failings of our industry is that we are overly obsessed with tools it's interesting you know it's it's it's it's it's you know tools matter but they matter to the degree to which they can help us achieve some kind of outcome some some some kind of result um so the stuff that you're talking about so so so i am a big fan of you know event sourcing as an architectural pattern that's the way that I've been mostly developing software for a very long time when I build software.

Those are the kinds of systems that I tend to build. And one of the reasons for that is you kind of set up the question in my head and then answered it in your preamble. One of the values of that is when you do event sourcing well you tend to separate is that accidental complexity from essential complexity you're able to kind of focus on the policy which is the conversation that happens with your events separate from the implementation detail of how you fulfill that policy and that means then you can plug in different implementations to achieve different policies all these kind of good. So I'm a huge fan of that. And I think that there are some specific things.

I was – the place that I alluded to before where we built financial exchange was called LMAX, and we did some quite innovative things in terms of software architecture. I was part of an author of something called the Reactive Manifesto that describes some aspects of that, these event-based systems. But one of the things that I think is kind of really interesting about that is that we got to the stage where for most day-to-day development, you didn't have to worry about any

of the accidental complexity of the software. You didn't have to think about storage because that just happened in the background. You didn't have to think about resilience because that just happened in the background. You didn't have to think about resilience because that just happened in the background. You only had to worry about the performance of your bit because the performance of everything else happened in the background. So that was a pretty nice way to work.

That was pretty good. And I think there's more that we can do to kind of raise that abstraction with platforms and tools and all that kind of stuff. So I think there's absolutely more stuff to do, more stuff to do with tools. But I think where the real value gets to is the kind of, it seems to me, the kind of reasoning that I was just going through talking about event sourcing. Why is event sourcing useful? It's useful for some profound reasons. I don't care about it in terms of, you know, is it RabbitMQ

or is it Kafka or whatever. That's just implementation detail. It doesn't matter. The thing that matters is the degree to which I can achieve modularity, cohesion, separation of concerns, those sorts of more deeper fundamental principles in the in the design of my software through the use of the and the application of these tools and so that's the stuff that matters to me much more profoundly and one of my fond hopes is that you know as we mature as a discipline we can start having conversations more like that than about whether i don't react is better than angular or c-sharp

is better than job who cares i don't i really really don't care about those things you can give me a turing complete language then then I can write you a software. Cool. Thank you so much. Sorry, Brian, I didn't want to interrupt you. Oh, no, no, no. I was going to go off on something, but I'll wait till you finish.

There's a little clickbaity idea I had I just wanted to bring up. Not much to get feedback on so much, but some commentary I wanted to add. But go on, Andy. Well, I think maybe this is something that we also take offline because otherwise we would probably have been talking for too much longer. But I really would like to get your personal and professional opinion on the stuff that we are building because we've been doing this for two years now and

we want to always make sure and validate with the market if we are actually, you know, doing something and actually solving problems. That's kind of my point. And so maybe I'll keep the captain conversation for something offline later on, but it would be awesome to get your insights on insights on it uh yeah ideas all right brian up to you now so i was just going to go more into the philosophical world of this a little bit not not to go too deep here but you know you know taking a cue a little bit from you dave's wearing a skynet shirt. We were talking about Weyland-Yutani a little bit earlier, right? And Andy was asking about why people make these decisions because it's the safe decision, right? And I think the biggest problem that we see with not just continuous delivery, but any of these kind of advanced things is the restriction is the human.

The human element is the problem. Our primate bios is programmed to avoid risk and to protect what we have. So what I think we see a lot of happening is once companies start becoming successful with, say, continuous delivery and rolling things out, as more money starts flowing in, as the business gets bigger, as investors pay more attention, all the bureaucracy and everything else builds up around this. This is just a human condition, whether it's technology or anything else. Everything builds in to protect that and then dampen or hinder the progress that was being made to get it there in the first place. And I almost feel like there's always, throughout history, there's always been the cutting-edge people. You know, Copernicus, you talk elon musk or even though i have

plenty to say about that guy but you know these people who are visionaries who execute these things they're the anomaly they're the ones who had a bug in their code right that broke through and the rest of us just try to follow and make these safe decisions and so i almost feel like, you know, the dream of getting to these wonderful places won't happen while we're in charge of running them. But then if we aren't in charge of running them, we end up with Skynet, we end up with Weyland-Yutani, you know. And I don't know if there's, you know, any solution to that, but it's kind of the pickle we've been in throughout the dawn of human intelligence. And we're seeing it again. There's so many ways we can apply this. If we could just break our molding and everything else, there's so many wonderful things we can do. And I just don't know if that's ever going to happen in our lifetime, but maybe slowly, surely. I don't know if you think about that at all much, Dave, because it seemed like you were smiling a lot as I was saying this.

I think about that stuff a lot. I'm smiling because I love these kinds of ideas. I think some of these ideas are quite deep and kind of philosophical about the way software works, but the way that the world works a little bit as well. I am a technocrat i i am a a child of a you know science scientific education and you know that that's my belief that's my that's that's where i come from i i think that if you look at the history of you know human existence we've been around as a distinct species for somewhere between two and two hundred

and three hundred thousand years and we made no progress for, you know, all of that time, you know, on any non-logarithmic scale. It's a flat line until about 300 or 400 years ago. And then it started the exponential tick up, and then through the 20th century and the 21st century, we've had explosive growth. At the moment, human progress is said to be something, we double all of our knowledge every 14 months.

That's how fast we are learning at the moment. That's unprecedented, outstanding. And I think that's a direct consequence of the adoption of science and specifically engineering. If you think about everything that surrounds you, it's the product of that kind of thinking somewhere along the way. Scientific rationalism applied to solving problems. It works better.

And the reason it works better is exactly for the reasons that you've talked about, because we are human beings that are prone to all of these mistakes. So being a fan of physics and science one of my heroes is richard feinman the nobel prize winner and he he very famously said that you know science is our best protection against fooling ourselves and we are the easiest people to fool yeah so so science is about working in ways that stop us fooling ourselves. Instead of wishing that I'm going to use this because I think it's cool or whatever, decide to use it because it's going to make you more effective

because it allows you to build software with fewer bugs or produce software faster. That would be a better way of making those sorts of decisions. So I think we can start applying this. The other thing, I think there's been a bit of a blip partly due to 20 for early 21st century politics which i'm not going to go into you know of late but on the whole there's the work of stephen pinker i think it's fashionable to say that how things are getting worse and worse and worse actually that's not the case on all sorts of measures things are getting better and better

and better and again i would put that largely down to you know technical scientific rationalism there have been you know political blips where we've started not to distrust science and distrust engineering but i think that's the hope this is this is a little bit profound beyond beyond where the scope of what we're talking about but But I think that's the hope for humanity. That's how we get to survive, is by applying that kind of thinking. And if we don't, we won't. You know, the AI overlords will take over.

Yeah, I think it's all relevant, though, because when you start looking at this continuous delivery stuff, at least when I did, my eyes opened up opened up to like beyond software and automaking how all this could be applied everywhere so i think it definitely has a humongous scope i think there's a whole lot for us to learn to apply this and just in in so many different even in my daily work i'm like okay how can i apply this you know i'll take a step back like what am i actually trying to do not what do i do but what am i trying to get done and maybe I should change how I'm trying to get there. It's from the smallest to the grandest. It's a wonderful thought process. I'm really, really

happy it's come up during my lifetime and that I'm in an industry where we're aware of it. And hopefully it'll catch fire and spread further. I genuinely believe that what it is that we're talking about is the application of scientific rationalism to solving problems in software. And science works. It's humanity's best problem-solving technique. So if I'm correct, continuous delivery is going to work.

Very nice closing words as well, I would say. Hey, Dave. Hi. Thank you so much for being on the show. And I think, I mean, this episode will air after your talk at Scaled Continuous Delivery. So I encourage everyone to obviously watch that talk. We're also going to provide the links to your podcast, to your website. People should definitely, you know, take a look at your and subscribe to your channel.

I also took a couple of notes, actually, when I watched your different episodes on YouTube, and I can just really encourage everyone to look at the Cyberpunk, even though you already have half a million visitors already, but viewers, but I really liked, because I guess I didn't think so much about it you are the plan uh the planning triangle with uh scope time and resources that you've explained very well really i think it's everyone in the industry um should be reminded about this and then also you know use this as an argument to push back if somebody makes stupid not stupid but maybe from their side good decisions but just to explain them why certain things won't work.

As you said, you cannot make a baby in one month just by using men and women. This just doesn't work. And so with that, I don't know, do you have any, what's your, besides obviously getting to a million or two million viewers, what else is your hope for 2021? So I'm working quite hard on my next book at the moment, and I'm hoping to release that this year. And my ambition is if I can move the dial just a tiny little bit

on people's capability to do better build better software faster I will feel like that's a job well done so my mission is to try and figure out how I can help people to do that to think differently about software development and just improve the way that they do things a little bit that's what I'm trying to do

awesome well thank you very very much Steve we really really appreciate you being on I thought this was fascinating in so many levels but really really thankful for you being on and thankful for you being the 125th episode and I want to also thank

all of our listeners for being with us for 125 episodes if there's any who've been there since day one and I guess that's about it. You can follow us on Pure underscore DT on Twitter, or you can send an old-fashioned email, pureperformance at dynatrace.com. And Dave, we're going to have some links for where people can follow you.

Anything else besides your talk that's you and your book that's going to be coming out that you needed to plug, or did you get everything everything else in there i've got some nice training courses if anybody wants some training awesome all right we'll have all those links up on on the site as well because it's funny we always talk about putting links up but yet it's a podcast but where it's listed you know there's there's no audio qr code right that'd be that's something for somebody to develop right um i guess it's the modem sound right anyway um that's it for for to develop, right? I guess it's the modem sound, right? Anyway, that's it for me. Thanks, everyone, for listening. Until next time.

Thank you. Thank you.