PurePerformance - 021 How Thoughtworks helped Otto.de transform into a real DevOps Culture

Episode Date: November 21, 2016

Finn Lorbeer (@finnlorbeer) is a quality enthusiast working for Thoughtworks Germany. I met Finn earlier this year at the German Testing Days where he presented the transformation story at Otto.de. He helped transform one of their 14 “line of business” teams by changing the way QA was seen by the organization. Instead of a WALL between Dev and Ops the teams started to work as a real DevOps team. Further architectural and organizational changes ultimately allowed them to increase deployment speed from 2-3 per week to up to 200 per week for the best performing teams.*****Related Links:******Process Automation and Continuous Delivery at OTTO.dehttps://dev.otto.de/2015/11/24/process-automation-and-continuous-delivery-at-otto-de/Are we only Test Manager?http://www.lor.beer/are-we-only-test-manager/Sind wir wirklich nur Testmanagerinnen?https://dev.otto.de/2016/06/08/sind-wir-wirklich-nur-testmanagerinnen/Das Leben ist hasselhoffhttp://giphy.com/search/david-hasselhoff

It's time for Pure Performance. Get your stopwatches ready. It's time of Pure Performance. I am Brian Wilson and with me as always is my co-host Andy Grabner. Hello Andy. Hey Brian, hi. Thanks for another show. Well, recording, right? We'll see how the show is really turning out, but I think it's going to be good because not only are we distributed again across three time zones,

leveraging the performance of the networks that connect us all. And in my case, it is actually the AT&T LTE connection from my iPhone because the hotel I'm in right now, and I'm not going to mention

who it is, where I am, is not really reliable. So hopefully this will work. You're at a conference currently, aren't you? I'm at a conference. Yeah, I'm at StarWest. I'm not at a StarWest hotel. I was too cheap for that, and I booked a hotel close by that was a little more affordable, and now I pay the price for it because the Wi-Fi just doesn't work.

That's what you get for that. Yeah. And I see Adam Auerbach is there. He was our guest. If people haven't listened yet, there was a really good digital transformation story, Capital One with Adam. And you talked about stalking him and possibly getting him a beer or, or a glass of wine or something. Have you been able to successfully stalk him? Not yet. I came in midnight last night and now it's seven o'clock in the morning. So I haven't

yet had a chance, but I will do this later today. So, uh, you might want to wake him up with a beer to show up at his door. It is 5 PM somewhere in the world, which reminds me it should be 5 p.m. in the time zone where our guest speaker is in, right? Yeah. Well, I don't know the time zone as well, but it should be close to there. Yeah, I believe it is. Hey, Finn, are you there? Hey, I'm there. Nice to have been invited.

It's 5.20 p.m. in Germany right now, exactly. Well, and it's German. You're supposed to have beer at this time aren't you supposed to have beer at five o'clock to be very honest i wanted to have a beer because you know this is just the right thing to relax but i had too many things to organize right before this call so i just didn't have the 15 minutes to go and grab some oh my god yeah it's october too right i mean even though octoberfest isn't octoberfest it just it was just done right it it stops with the um the um the third of

october i believe now at least the first weekend in october so i think it's over in munich exactly all the joy or the mess however you want to put it it's over since the last weekend yes all right um is it i'll let andy pronounce your last name for everybody because i will i'm sure i will mess it up well because you would say lower beer probably or something like that more beer i was thinking more beer but yeah more beer yeah finn lower beer i hope i hope i got this correctly I hope I didn't lose my German accent. But Finn, we two met last year at a German testing conference in Frankfurt, if I remember correctly. Right, but it was this year. It's only five months ago or something.

It was this year. It's amazing. Yeah, it was this year. It was early. Yeah, it was this year. Yeah, it was early this year. We met in Frankfurt. We had a session back-to-back in the DevOps track. I believe they actually called it the DevOps track. And you talked about the phenomenal transformation that you actually helped Otto go through.

I mean, maybe not the whole transformation, but parts of the transformation. Otto, by the way, for people that are not familiar with who Otto is, Otto is a very large retail company in Germany, Otto.de, their website. And Finn, you were presenting before me, and I think it was a perfect, I'm not sure who organized the presentations or the schedule, but it felt like a perfect fit first to you with your transformation story.

And then I came in with my metrics driven continuous delivery story. So I thought that was great. And I really enjoyed your talk, first of all, because you are a great presenter and you have a great story to tell. And you're working also for a company that is known for helping a lot of people going through the transformation. And without further ado, I actually would like to let you introduce yourself a little bit more and talk company that is known for helping a lot of people going through the transformation and without further ado actually would like to let you introduce yourself a little more and talk a little

bit about more about yourself okay yeah so you know my name i'm finn i'm currently working for thought works um and yes as part of my work i had the pleasure to join otto for some time in their digital transformation um otto is a company that exists and since the 50s never had a real store where they sold things from always worked with catalogs they sent around and by now they discovered that you can make some money in the internet and are really trying at the moment really really successful to go into the Internet and with all their retail. And of course, everybody knows Amazon, so that's a big player in Germany as well. And Otto has to succeed in this transformation in order to stay competitive.

And this is where we come in and help since years and hopefully for some longer time. It's really amazing to see how this pretty old company is still very flexible going forward. I think it's really cool that it's most of the times you hear of the brick and mortar storefront to digital transformation this one is a little bit different right because it's catalog to um you know digital which there's a lot of parallels between the two right because there's no store actually but i just thought that was um that's the kind of a cool variation on that transformation so exactly to some extent you could say you're just exchanging the medium on which you're presenting a product in their case. Basically, they already have obviously figured out distribution in the back.

So the distribution, like distributing the products and then handling, dealing with returns and all that stuff, that has already been totally done over decades. But really leveraging the new, as you said, medium from the web, the browser, the mobile device is obviously a cool thing to do. Before we dive into that, I just wanted to ask, you know, Finn, I think we've only done it once so far. We put Pat Meenan through it, so we have to make sure we do it to some others.

We wanted to ask, you know, over the course of everyone's career, there's always at least one of those moments where you kind of think, oh, my gosh, I can't believe I did that. And how did how did I get out of that? And in the end, it usually becomes something you learn a great deal from. I was curious if you had a similar sort of experience where, you know, maybe you or did something specifically or were part of something where, you know, you're just like, wow, that was really crazy that we did that. But in the end, you know, something good happened. Yeah, there was one thinking in the context of what we're going to talk about, I thought about this, this transformation, and how we were to discuss yet another thing of the old processes. And I had a look at the extensive documentation that we were going to review in a meeting.

And I was, yeah, it was insane. It was so insanely long that I just pasted some David Hasselhoff GIFs that were most colorful somewhere in the documentation, in the official documentation of the company, just to make a point that probably no one is reading it. And then we went into this meeting and apparently the topic seemed to be more interesting than i than i anticipated and all the major executives of the department joined as well and then the mail opened where all the links were to those places and i was a bit afraid i really didn't know how this would end but i learned that in, people don't read the documentation.

So no one there noticed. We didn't open the links. We discussed it for two hours without looking at it, which just proved the point in the beginning. So maybe I learned that you can still get away with it, but it was a close one, I would say, yes. I think I know the check you're talking about is one of them where it zooms in on the picture of himself on his briefs and it keeps zooming in infinitely. Oh, yes. That was one of them. And then there's another one where he's just sitting in the car screaming.

Awesome. Yes. And David Hasselhoff. I mean, David Hasselhoff was a big shot, I think, in Europe, especially in Germany and Austria. So I'm not sure. I mean, he was also popular, obviously, in the U.S., but I think for whatever reason, Knight Rider and Baywatch. Something happened in Germany, I guess his pop music career, too. Didn't that really take off over there?

Yes, he was singing, I've been looking for freedom in the time that the wall came down here. So that was the perfect fit. And he was actually standing on the Berlin Wall wall when it was teared down singing this song and they still invite him for uh the the years when we remember the days um to to come back and sing so this is somehow i think how we are a bit more attached then and he still thinks he was the one that brought the wall down or is it is he finally realizing that it was actually somebody else and he was just happy to lucky to be there no no no it was it was i think his major major part well cheers to david has off thank you thank you for tearing down that wall david

yeah bring down that wall or i keep singing All right. So, Andy, I know you have a thought pattern for this. I do. For the topic. So let's hear it. always hear when we talk about all of these cool examples like Amazon and Facebook and the ones are how many deployments people actually do, these companies do into production. And now it's always about, you know, these are unicorn companies and they started from scratch and everything. It's just so much easier. But now when you did your presentation at the German testing days, you also told us the story, the multi-year transformation that Otto went through. And especially, I think, focusing on the last one and a half years when you have been part of that transformation project, how you actually went from monthly deployments to, I believe, like 100 deployments per week. And I thought this is phenomenal that you can do something with a

traditional software that you have additional pieces, you need to break it apart. So I will be very interested in getting some background information on how you actually achieved that you were able to deploy up to 100 times a week, which is phenomenal, which is multiple times per day, what it meant from a, maybe from an architecture perspective, from a team structure perspective, from a culture perspective, because there's a lot of things, obviously, that have to go hand in hand in order to get there.

And I would just be very interested in your thoughts and what you typically do in these projects and how to achieve this transformation. Yeah, so to give a bit of background around those numbers, we briefly talked about them already and I said it's 100 deploys per week. Then I figured that's 100 deploys per team, per best performing team

and I just pulled up the actual numbers of last summer which were up to almost 300 deploys. But let's focus on the best performing team, because this is actually where the story takes place. There are 14 teams at Otto, and I was part of one team. And Otto started four years ago to build an entirely new web page because they were at quarterly releases, and the typical problem was there. They had a too big application. They couldn't develop anymore.

They couldn't deploy it, and the deploys were very painful and horrible. So they divided the shop into smaller verticals. They started with four or five verticals single systems on a service-oriented architecture base and started developing and this worked really well they found more and more verticals like business domains that they would spin off and ended up with 14 maybe by now so we are talking about 14 teams and then in one of the older teams where i was in we figured out that after four or five years even in this vertical we had a big monolith that we could deploy every one or two or maybe sometimes only three days and that makes them one to two

deploys per week by by the beginning of 2014 maybe and we knew we had to improve there. And then we just applied the same principle of splitting the stuff. And at the same time, all the microservices concepts came up and we just went with it. And went with it very successfully. We can go into details later on. And increased the deploys while looking at the risk of each deploy and handling all the difficulties people may have with it

that are not too familiar with the technology behind it and that are a bit afraid of pushing blindly to production. And meeting all those concerns and how we could address them and how we could talk about them and where we improved was then the bigger part of work than actually implementing a microservice. And then we went to 200, 300 deploys for the entire department. We made maybe 100 deploys for this team per week. Cool.

Hey, Finn, can you remind me when i talk about department of business unit i think it was one out of 14 at least as far as i remember uh i think it would be interesting for audience from a business perspective how did the company ought to itself in the very beginning try to break down that big company monolith into individual business units the original idea was to find the business unit the biggest business unit is the original idea was to find the business unit. The biggest business unit is the basket where you click on the button, where you have the conversion that you would aim for. So this is one of these vertical, one of these business units. Then you have a navigation throughout the shop that is consistent throughout the shop. This is another vertical, if you want.

You have a user and user management management you have a product and somehow product management which is not so much user facing but to have all the data is is a lot of work and our vertical my team was for the personalization to provide a personalized experience if you buy this why don't you buy that and things like this. Wow. So that means they took the whole website and basically said, what are the individual pieces or services to make up the whole website? And let's break it into its individual teams. So 14 came out, you named a couple of them. And I think this was done over the last couple of years. And then when you came on board on the personalization team, you realize that even though we are a small team, we can only deploy two or three times a week.

And that's simply not enough. Exactly. So when I came in, we already started looking into those microservices. We had one big monolith and started thinking about where to create the first microservice. And, of course, for just deploying once a week or twice a week, you have a lot of changes in your monoliths that you would deploy in any team. So there was a really big and lengthy process around any deployment. The deployment took 60 to 120 minutes, one to two hours, and had a lot of manual steps. You needed to have two people click specific buttons in a Jenkins interface in order to make a deployment role so that you would know who is responsible for this deployment.

It was said how much you have to look on the website, how much design testing you have to do. You would have to document all the test cases and so on and so on. So a lot of process around it. And of course, the initial thought of those people who designed those processes was to apply them for the microservice as well, because they thought from a user perspective only and said, the shop doesn't change, the service doesn't change. All our processes to protect the users should apply still. So all of this process machinery

applies for microservices as well and this was the bigger problem there that i could tackle then being a test manager in the team but therefore having access and contact to the other testers at auto and then we started talking about general QA concepts instead of the pure testing, and this is how we slowly, slowly improved things there. I wanted to ask, when you rolled into Otto to help with this deployment, did they have their own tech, you know, IT team that was managing what they, you know, the monolithic application? And if so, did they, you know, what kind of challenges were not necessarily challenges, but how did the team perform in, as Andy always says, leveling up and maybe, you and maybe embracing the new style?

Were there any kind of challenges that you faced there or were they already kind of trying to follow, even though they were monolithic, somewhat agile type deployments? Did you face anything on that side? Yes. So the team consisting mainly of developers and some business designers they had a tester to fight against them in the team somehow that was the setup you know because developers only break things and only do bad stuff that was the perception so they need someone to control them and to check them that was the setup the team tried to go as agile as possible and the testers were always looking on the process to

do it right to document everything the team i went into had already two that were let's put it in words more on the side of the team going for this agile however they had no backup for that in their own testing department. They were always asked to go for these processes, to go for the lengthy stuff, to somehow wrap the waterfall model around the scrum teams still, and weren't sure how to proceed there. And this is where we then made some advantages to engage in these conversations, to discuss a lot and a long time what are deploys that is less risky in comparison to a deploy every week, how to mitigate risk that still exists, and how to apply different test methods and automated

testing all over instead of a lot of manual clicking through a service that doesn't even touch the services you want to deploy. So when ThoughtWorks came in, you have multi-years of experience as a company with all the different customers that you helped over the years. Well, I'm just reading the DevOps Handbook, which was co-authored by one of your former colleagues, Chess Humble. I read his biography, and he basically worked for ThoughtWorks in the beginning. So when you walk into these companies, and I think you already answered the question that I have, but is the major part in the beginning to actually sit down with them and actually try to figure out what is wrong and broken from a cultural perspective? Like what

you said, the ops guys saw that the testing team is kind of like a shield against the evil developers, right? And is that the major work that you have to do from getting people together and kind of telling them, hey, guys, you're a team. You're not enemies. Is that the major part in the beginning where you spend a lot of your time? That was my personal journey in this endeavor, I would say maybe. In general, we as ThoughtWorks, when we came in, that was years ago when Otto asked for help to develop a new software and not release it at once as a big bang,

but in slow steps with alpha, beta, beta 2 release or something like that, and then to deploy continuously. So that was the initial conversations we had. That was purely, more or less purely technical. We come in with more people. We don't just advocate for something and show some presentations, how to do things better and then leave again. We come with developers and teams to contribute

and to build with them on the product and share by daily activity our culture. So this is how we bring in our culture. And here we had a problem because up until then, Otto would only get developers from ThoughtWorks. And the developers had difficulties to reach out to this QA department. And the biggest challenge was just to get one QA staff so that we could show different ways to the dev team in the beginning and brought them a cool new mindset, they then have to fight the battle against established QA and ops team.

Did the management at all understand that in order to really become successful, you have to change the whole culture of every team? And therefore, you just need to bring external help like additional guys from ThoughtWorks to also change the mindset of the next team like what you did with the QA team or did the management come up with that idea or did you propose it to them upon seeing that it was the next major problem? The original proposal came from two of the ThoughtWorks developers in the team that said this is our major issue at the moment we don't get somewhere they didn't have the microservices yet. They had only the monolith, and somehow it all worked. So there was no big need for management to do something yet. That was the original idea, too, of the ThoughtWorks developers.

They asked for a QA and it was luckily me to go in. So the management later picked up when we got more and more services and more and more of the teams and they when they picked up the trace of this cultural change in the first one or two teams then we had a big problem because some teams really wanted to go forward and quicker and quicker and some teams were still hesitant about it they were they were used to the old processes they gave them safety don't forget that a process that works for four years gives you some And some teams were still hesitant about it. They were used to the old processes. They gave them safety. Don't forget that a process that works for four years gives you some safety in deployments,

which is the most uncomfortable situation people are usually in in such a setup. And this gave some friction and some problems. And we then started discussions and then management saw that there is some need to to discuss first of all and to see where the different problems and needs come from and so over time they got the hang of it as well and then we had workshops then we discussed it but then people were already aware of the topic. So if you are there and do a very little bit every day, it's a slow progress that you can then enhance and enhance more and more. It's not so much the single meetings you have on a single day where you would go out and say, today I achieved something again. I think two more people understood how we have to do it in the next year it's more raising the general awareness

every day and then making the people more comfortable with the ideas and when they are a bit more open to the thing then discuss how you would actually tackle things and and proceed and and actually do a lot of deployments and do all the testing and be comfortable that your css doesn't break because yeah that's that's not an obvious one now um can you tell us a little more about the cultural things that you changed i assume i mean what you did is you tried to make the qa team more of a part of the development team because that's kind of the spirit of agile extending it into the other areas so that if you really become a quote-unquote DevOps feature team,

you basically just want to have all the roles within your teams knowing that it's not only functionality, that the functionality is correct, but also well tested so they can automatically run it through the pipeline and then deploy it. So what is some of the advice we can give to the audience to say, here are some of the basic things we always do first to get these things closer together? Yeah, so in general, we didn't go in and say, okay, well, I did in my team go in and say, I'm now part of the team and I will work with you. I won't work against you. And there were some ThoughtWorks people already who just smiled and knew how we would proceed from there and some were yeah not so sure what this would mean but then you just go and you work with the people and of course as usual if you work with people they are much more happy about it for all the other teams the teams were solely independent and they

had different cultures as well and i didn't want to inflict anything on them so the thing was that we just slowly changed our team we work with developers if i work with developers if i know what developers do while they work because i talk to them daily hourly maybe if i have the time if i pair with them the entire day and if we discuss in a test-driven setup that is one of the assumptions and when we discuss how they write the tests and if we discuss in a test driven setup that is one of the assumptions when we discuss how they write the tests and if i'm sure the test pyramid for example if i know there's a sufficient amount of unit tests if i know the integration tests are at the right place and if i know that we just get one or two end-to-end tests on top and if i know that a development pair will take care of all of this before they finish the story, then I know a big amount about the quality inside this piece of code,

and then I don't have to test everything and every edge case in the end, and I have more time to do other things and to spread the culture and to spread exactly this, what I achieved in the team. And then when you start working in this environment and you deploy more other teams, they want to do the same. And you can tell them and you can help them as well a bit. And in the end, it's maybe seven or eight of the teams are getting more and more in this direction. And then you have a workshop.

And we had a workshop and I just put, we we collected topics what to talk about in this workshop and i just put up one sticky note saying are we only test manager that was the question then in the end after we we worked with the teams and then everybody said oh no this is not our job description we do so much more and then they came up again with it. They said, oh, we do pair a lot. And they said, oh, and we know about the testing pyramid and we coach our teams. And so in the end, we wrote a really nice blog post with a lot of people.

And so they somehow then in the end came up with all of those ideas on their own again. And this is, I think, much better than just going there, showing it to them and telling them how it should be to help them learn. And if you're in a team and you want to get started and you don't know where, this is the typical setup. There are so many things to improve.

You never know what. You don't have big trust. Often as a tester, the team looks at you with some suspicion i always say that maybe look for the easiest challenge you can find not for the biggest impact but for the easiest and smallest thing to change and if you change this and improve there then you will get trust and then you will trust yourself as well and then you can change the next bigger thing. And then you have two allies and can change the third thing. And then you can go on and do more and more. And by time, coming back to the idea of leveling up. When you have that challenge, no matter what organization you're in, especially if you're on the test side of the team, is always a scary one, right? Because if the developers start writing in some testing into their deployments, you can quickly look around and say, hey, where am I going to be in a month? Am I going to be needed here anymore? But I love the idea that you mentioned on,

you have to go through a major, major change, right? And you have to change your processes and change your way of thinking. And instead of trying to tackle the whole thing at once, take, bite off that easy piece, right? That shows that you're going to, number one, it's going to show that you're ready to make progress and that you're participating in, you know, the new style, but it's also getting you to slowly learn and level up so that when you do, as you were talking

about, sit with the developers and check out what they're testing so that you can modify, you know, okay, they're covering components X, Y, and Z in what they're doing. So now I can just concentrate on the last few components that they can't really bake into their test. It automatically sets up that conversation, that workflow, and that cooperation that by nature will help everybody level up in that way. So it sounds like it doesn't become so much of a chore or so much of a stressful point. It just, if you take that mindset of, let me just chip off one little bit at a time, you're going to learn through that process and you're going to automatically level up and do it. Exactly. And you have to keep in mind that it's what just sounded so easy. Take one thing, do the next and the next, and then you're already done is a process of half a year in a small team

and a small software and was over the course of one and a half years in this context that we just talked about but yes you be eager to learn and then you'll you'll get there it reminds me of a movie i don't know if you any of you saw it. It's a Bill Murray movie called What About Bob? Are you familiar with that, any of you two? No. That's a good one. There's a whole book called Baby Steps. This guy has a lot of problems, and he goes to a psychiatrist, and the guy's like, here, here's a book I wrote, Baby Steps.

You put one foot in front of the other, and you take baby steps. Thank you. It's a very good movie, though. You should check it out in the in this project at otto all people were in the end all the time talking about baby steps for everything that we did they were always referencing and now finally i have to talk to you to get this reference it's awesome thank you there you go that's fine i have a follow-up question on this one so finn when you came in was the biggest challenge in the beginning?

The developers were actually accepting the fact that we need to do test-driven development. Did you then play a mentoring role for the developers when you pair with them to help them create initial tests and show them the value of having all these tests? Not only unit tests, of course, but you said integration tests and some end-to-end tests. Because, I mean, I have a tester tester background and I see the value of tests, but I really see the value of a developer having tests. Because within minutes, I'm as a developer, I get immediate feedback of the code change that I just made if I have to write tests. If I execute these tests tests i get immediate feedback and so was it one of your things in the beginning where you had to actually sit down with them and show them this is how you write tests this is how you do integration end-to-end tests and this is how we automate the whole thing so when you check in something into jenkins everything gets executed and you

immediately see the feedback is this something you taught them as well no not at all i'm lucky to be in a company where this is one of our core values now talking about thought works so when developing new software we have i wouldn't say dogmas but we have two dogmas which is pair programming and test-driven development to make sure that we have quick feedback and a minimum code quality and a good knowledge sharing. There are downsides to it as well, but we decided to buy in for the benefits. So this was discussed in the three and a half,

four years before I joined. Those were fights of other people before me that advocated for this a lot and that was hugely accepted throughout the organization of other people before me that advocated for this a lot. And that was hugely accepted throughout the organization by time then as well. But that was already in place when I came. So the bigger challenge was to work with my QA team there that I became part of. So changing the mindset, right? I mean, that makes a lot of sense.

You mentioned that what you basically did, you applied the same kind of monolith to microservice breakup from Otto from several years ago when they broke up the big thing into individual business units. You then also broke up your business unit, I think the personalized feature business unit, into smaller microservices and and anything where were you as as the testing within that organization you did you have a word to say when they created new architecture especially when it comes to to testability did you have a word in that so this is now talking in the context of a team by that time the team so i was there for half a year when we discussed what we tackle first and how we take out the first microservices. There was good trust in the team. It was a really good team workings. And we saw ourselves as a real, truly cross-functional team.

So every team member had to say and how we cut those services. And I can remember that we had one discussion lasting over two days with eight people and we deliberately took the time to make sure that we really get it right because this is something you wouldn't change too fast but there was nothing special about me or my role because in this conversation within the team it was all the team members with equal say in this so that's cool because that means that everyone could chime in and say, hey, this is what we want from an architectural perspective. And every team could say, well, from a development perspective, we have these constraints.

From a testing perspective, we need to have good interfaces or contracts that we can use to write our tests again. And I'm sure that the ops part of the team could say well uh if you break up the monolith into microservices how we how do we deploy it uh can we automate the whole thing so you really spent two days in figuring out how to break it apart and how to make sure that everyone was comfortable enough with the move exactly as comfortable as you can get so at the beginning we started small with four people then more and more people got renting in then we went into the conference room then things got hot as they

usually do then you get a crate of beer and you start all over with the discussion and then you go into the second day you will i guess rarely meet or find a solution that satisfies everyone involved to 100%. So you have to make compromises. And in the end, we decided to go for one of the solutions we had. And we had a lot of discussions later. And the team is up to today, one year later, still not sure if it was the right thing. So although we took some time and we were happy to take it, don't overdo it either and go for two weeks because you will, I guess, never find the best solution in the world.

Yeah, I think that the biggest thing is that we move ahead and not come to a conclusion. The great thing about being flexible and agile is you try something and if it doesn't work, you can still change things. But the worst thing you can do is not doing anything because you're not sure what the type of impact it may have. Yeah, exactly. And just remember to revisit things afterwards.

If you take such a decision, go back, maybe make even a date for it, a fixed date to review what you decided and to see if this was the best option to go forward or if you need to adopt again. So to summarize the whole thing, if I hear this correctly, Otto went through a large transformation over the last couple of years. You came in and were faced with the challenge that even though you had 14, quote unquote, independent operating teams, that the biggest challenge was that within these individual teams, you had QA, developers, and operations kind of working on their own and seeing each other more

as the enemy, using QA as the buffer in between ops and dev. And I think the biggest transformation, what I like, is that people actually realize that they all work better together. So that means that you brought people together to make sure that they have proper test automation. You brought teams together and all of them could say that the architecture should be like moving forward with the next iteration or version of the software so that everybody really had an equal voice and everybody felt they were part of the whole team. And I'm sure from a technical perspective, I'm sure there's a lot of stuff in your blogs as well, like what kind of tools you use and which patterns you use for your microservices, which would be too much for this podcast now, but to conclude, the most important thing is that you want to make sure that people understand they're one team. And yes, they have different roles to play, but in the end it's one team i believe we need to communicate we need to

talk we need to build trust and with that you achieve amazing things like uh what you did and you went from two deployments um a week to i think you said the high performing team had a hundred deployments per week. Is that right? Exactly. I couldn't summarize it better. That is one of Andy's great strengths is his summary skills. I'm going to have to start, Andy, I'm going to have to start getting some kind of like

summary music for you when it's the summary time. Yeah. All right. Well, we're going to wrap up this episode unless there was any other final thoughts. Was there anything else you wanted to add? I think I shared a lot. If you want to read on, I can just advertise for the Otto dev blog at def.otto.de if you want to read on. There's a lot of English stuff in there, just a few German articles.

And for some of the German articles, we provide a translation somewhere else later on. Okay, and I'll get those links from you specifically, and we'll put them up when we publish this podcast so that people can have a direct link to them from the page. So we're going to continue. Sorry to cut you off, Brian, but my last word, which also shows, again, the importance of the cultural change because you're willing to share your lessons learned with the public. And you're not afraid of sharing what you did, where you failed, where you struggled, and what you did right. So I think this is pretty cool. Great. And I'll make sure I hide a Hasselhoff link within those links as well.

See if anyone clicks uh clicks through them so so finn we're going to um come back with uh to you in our next episode right when we're going to discuss uh the life cycle of a feature and its feedback loop um and in the meantime uh if anybody wants to follow finn on twitter uh he's at finnlorbeer f-i-n-n-l-o-r-B-E-E-R. I am at Emperor Wilson, and Andy is at Grabner Andy. So if you want to follow us, because people like to follow, we're all sheep, right? Please do so, and we will be back in a moment with Finn. Thank you very much, Finn. It was a great story you had there.

Thanks for inviting me. Thank you very much, Finn. It was a great story you had there. Thanks for inviting me. Thank you. Bye.