PurePerformance - 062 How Microsoft became a cool company again – a DevOps Transformation Story with Donovan Brown

Episode Date: May 21, 2018

15 Minutes of your day! That’s all it takes to make the first step towards applying DevOps best practices. To hear more about this and other suggestions on how to jump start your DevOps transformation tune into this episode where we chat with Donovan Brown ( http://donovanbrown.com/ ), Principal DevOps Manager at Microsoft.Did you know that over the last 7 years the VSTS team has increased deployment velocity from once every 3 years to once every 3 weeks? Coordinating 50 different feature team that all commit to master daily? If you always thought that transformation like this are only possible in smaller development organizations then be proven wrong by Donovan, a member of the League of Extraordinary Cloud DevOps Activists. If you want instant advice simply tweet using #LoECDA and summon “the league”!

It's time for Pure Performance! Get your stopwatches ready, it's time for Pure Performance with Andy Grabner and Brian Wilson. Hello everybody and welcome to another episode of Beer Performance. My name is Brian Wilson and as always Andy Grabner is here with me. Andy, how are you doing today? I'm still freezing here. As I just said, you know, last weekend, actually last week we saw each other in Denver. We sure did. We sure did and it was really cold Friday, but then beautiful on the weekend.

Then I get back to Boston and it's early April and it's still freezing here. People don't understand Denver's got great weather. Hey, Andy, quick thing though, in relation to today's show, totally going to let you introduce the guest as always, but I had a wonderful moment as we were prepping and talking to our guest. There's always a lot of conversation or debate about how to say the name of the Microsoft cloud system, right? Whatever you want to say without saying the name. And it brings me back to back when Linux came out, some people were calling it Linux, some

people calling it Linux, some people calling it Linux, whatever. And I remember when I first installed Linux, there was an audio file by Linus Torvalds who, in the audio file, said, My name is Linus Torvalds. The proper pronunciation of Linux is Linux. And now, to me at least, to end the debate about this, we were talking to a Microsoft person who works with Azure and pronounced it Azure. Not Azure, not Azure. I don't know. Maybe when he comes on after you introduce him, he can correct me if I'm wrong. But to me, I was excited because I think there needs to be a definitive answer to this question.

Yeah, I agree with you. And I think what we can do now, we can actually, I click on the virtual replay button and I will let the guest of honor today, the man in the black shirt, as some people call him as well, kind of play the audio, introducing himself and giving us the proper proper pronunciation of the Microsoft Cloud Service. Play. Play. That's your intro. Interesting. I've never been intro'd like that before. But my name is Donovan Brown.

I'm a Principal DevOps Manager here at Microsoft. Some people might also know me as the manager of the League of Extraordinary Cloud DevOps Advocates. I know it's a mouthful, so most people just call us the League, but we're a collection of cloud advocates who span dev and ops, Windows to Linux, app service to containers,

and even Jenkins to VSTS to make sure that our customers can implement their solutions inside of Azure. So that's the way that I say Azure. I've heard it said every other way as well. I never correct anyone. As long as you're using it, you can call it whatever you like. That's a good answer. And sorry for giving you this strange way of introduction, but I thought, you know,

we always just come up with these things. I had no clue what Brian is going to say, so I just picked it up. And now what I want to pick you up on, though, is you said you are part of the league of extraordinary cloud DevOps advocates. Wow. So you're part of a superhero group now. It feels like it sometimes, right? Because we actually have a bat signal that you can use to get ahold of us, which it seems silly at first, but we actually have a hashtag, which is L-O-E-C-D-A, which is basically the first letters of that long ginormous name. And if you use that hashtag in any tweet, what happens is inside of our Microsoft Teams room, it lights up and it lets us know that someone just tweeted with that hashtag. And you have all five of us run to that tweet and read it and figure out how can we help that customer be

successful. And if we don't know the answers, we work at Microsoft. So we literally will reach inside of Microsoft and we will find someone who can answer that and add them to that thread for you. So you literally now have a bat signal that can get you inside of Microsoft just by using the league to get you there. So we shouldn't abuse that hashtag and start saying, you know, which dress looks better. Yeah, I have had that worry. But luckily so far, we haven't had anyone do anything silly with it. They've always been like legitimate questions. And it's like, wow, this is awesome.

And then you hear people just that they're so excited. They're like, wow, I can't believe the league just like answered that question for me. Like, that's what we're here to do. That's awesome. And we're trying to use social media to reach our customers like we never have before. Cool. How often do you get a question like, how can I update my Windows 98 system? More often than we'd like. We got one just the other day. This guy was like, yeah, I want to be

able to use deployment groups in TFS 2015. And my answer was upgrade the 2018 and you can do just that, right? I mean, there's only so much we can do. We can't go back and add features to old products. But we tried to give him all the ammunition that he needed to be able to justify the upgrade and go forward and be successful and told him to reach out to us if we could help any further. I'm running Microsoft Bob. I always got to plug Bob whenever you can. Oh, man. Wow.

It takes me way back. Hey, Donovan. So the reason why we reached out to you is because you actually are speaking or by the time this airs, you have spoken and you've delivered one of the most amazing breakout sessions in the history of conferences, obviously, at DevOne in Linz, Austria, my hometown, actually. Thanks for taking the invitation and speaking out there in Austria, enlightening a lot of the developers over there. Now, I believe the topic that got a lot of people excited in your audience

was around the DevOps journey for Microsoft itself. Correct? Yeah. So what happens is I'm actually giving two sessions. So they invited me to give the keynote at DevOne, and that will be the story of how Microsoft has gone from a three-year release cycle to a three-week release cycle. And to move an organization like Microsoft to a clearly waterfall three-year cadence, to a hyper-agile three-week cadence for a 24 by 7 process takes a lot of doing. It takes the changing of your process, your products, and the people's mindset to be able to achieve that. And what I do in the keynote is I share our learnings and our transformation. And what I do in the keynote is I share our learnings and our transformation. And what I find is that people gravitate towards this talk because they're all

on that exact same transformation today. And it's nice to hear that even Microsoft has struggled in some of these things and they don't feel so bad anymore that they might be stuck somewhere or having to try something for a new way of trying something to get past a road block because we had to do the same things. And it's really neat to see people just take this huge sigh of relief to realize that, don't worry, we didn't get it right the first time either. There's things you're just going to have to try and it's going to look different for you than it does for us here at Microsoft. But stay on that path because you too can move from every three years to every three weeks if that's what you need.

Yeah. And I think this is great that you guys are telling this story. I mean we've told our transformation story over the last two years I would say. And sometimes when we tell the story and we have an audience of very large enterprises, larger than Dynatrace obviously. Then they say, well, this sounds good. But we are not Dynatrace. We are much larger. But now you here you know you stay on stage and i think there's not there's not that many organizations that are larger than microsoft uh when it comes to to you know software companies and and it's great that you are talking about and the fact that you

know ours was six months to two weeks right right? You went three years to three weeks, which is, talk about a hurdle. Holy cow, man. It's unbelievable. Even sometimes when I tell the story, it's just like, wow. I mean, it's still amazing. It took us seven years to do it and counting, right? We're not done. Three weeks is where we are today.

And now with the advent of Windows containers coming out, we now have the ability to potentially even move faster than we do today. When you're building a DevOps pipeline, you need to focus on what hurts most first. When you achieve that hurdle, when you overcome that obstacle, what you're going to realize is that there's a new bottleneck now that now you can shine a light on because what used to take you a long time,

that's automated, that's an afterthought. Look at this over here now, this is what's taking us the longest amount of time to get code from the fingertips of our developers into the hands of our users. How can we make that even better? And being Microsoft, if there's not a product that already exists to help us with that,

we'll go write it, which is amazing, right? And then just put it as part of our tool chain, a part of our offering that we give to our customers, because we dog food Visual Studio Team Services, which we all call VSTS, which means VSTS, the service that we ship every three weeks, is shipped with VSTS every three weeks. So everything that we're doing, we're learning from it, we're using it, and we're using the same product that we're developing to deploy itself every three weeks. Andy, does that sound familiar?

It sounds familiar. But Donovan, I want to give you a better term than eating your own dog food. We call it drink around champagne. Oh, I heard that just the other day. Yeah. No, it's fantastic, and I completely agree with you.

So what we do, we monitor Dynatrace with Dynatrace. So when we push a new build through the pipeline, and when we run Dynatrace in the cloud for our customers, then we obviously monitor Dynatrace with Dynatrace. And I think that's a big, big testimonial to that you actually trust your own software and you hope that your customers become successful with it as well.

And I think the other great thing about your transformation story, and in fact, with all transformation stories out there with software companies that are selling quote unquote DevOps, if you are selling to your customers, you have to deliver a new feature update every other week.

But then you as a software vendor, it takes you six months or three years. Then obviously you are selling something that you don't do yourself. So obviously you now, you are doing it and you're promoting and obviously you've built the right products that allow that type of development model. That's awesome.

So you said the focus of your story is VSTS. Can you give us a little more background?, you mentioned that you start with the tough things early. I heard other companies say, we want to first have a couple of quick wins so that people get excited. So let's not pick the toughest nuts. So what about you? You said, you know, let's pick the toughest things first. Yeah. The reason why is because, and I think, I think we're the people you spoke to, I think, are coming at it from a completely different perspective. If I'm going into an organization that predominantly is not a software company, say, for example, automotive industry or

another industry that's not focusing on software, you going in and trying to preach DevOps, trying to preach agile development, they're going to resist that because that's not what they do today. It's not what they've been trained on doing. And what ends up happening is if you can't prove with results that they're going to get really good return, they're probably going to reject it. They're going to say, nope, we got deadlines. Let's just stay focused. We don't have time for anything new. So in a scenario like that, I would take a small pilot project and I'd like to get some quick wins because those results are going to speak volumes beyond any PowerPoint or theory that I might tell you about how successful we will be. So that's a little different than you are in the trenches.

You are now trying to take on this transformation and you have something that's slowing you down. If you have something, for example, running a manual test that take you seven hours to do, and then you go and get a quick win by setting up, I don't know, something on the right side of it, like a really cool monitoring situation, you haven't sped up your pipeline at all. Your pipeline is still seven hours long because of all this manual deployment. You should have focused on what was really slowing you down, which is the fact that we need to automate as much of that manual testing as we can so that we can actually push value to our customers a little bit more quickly. So I think based on the statement that you made, I think your customers are in the first bucket as we've never done this. Let's go prove that this will work versus what I'm saying is, now that you're in the trenches and you want to get more bang for your buck, you got to go tackle those big rocks first, not the little ones. It makes sense?

Makes a lot of sense, yeah. Got it. Yeah, and I see that too a lot of times. You know, I was actually even talking to my friend who works someplace and they're trying to put through some of this DevOps kind of thing, and they don't know where or how to start because although everyone within the organization

is talking about implementing this, the product owners are still saying, we don't care we still have to get new features out the door right they just care about shipping shipping it shipping it over and over again yeah we all know part of that story in general if you're gonna go big as you're saying you're gonna have to put the brakes on everything and if you're gonna go big and if you were a non-software company but i would not want you to

put the brakes on everything that's why i'm i don't even want you but that's the way they feel but that's the way they feel that's why i'm saying if they take that small project and woodshop it you know great they go ahead and they they prove out they work out their internal kinks they work out kinks with the personnel just even trying to get over the fact of giving up some control or i'm going to give up some of my job protection by collaborating and teaching other people what I do so that I can in turn learn what they do so that we can be all be better. But this is a cultural shift as well. If you're doing that on that smaller scale in that situation you're talking about, I think that definitely does set you up for, um, more success, uh, a better road for success so that you can eventually get to that,

that bigger project. Got it. Got it. No, I would agree with that. In scenarios where you have the scenario where they're like, nope, just keep doing it the way that we've been doing it. I think the first mistake in that scenario is they went and asked someone if they had permission to do their job, right? Your job is to ship software, right? And that's what you're being asked to do by your managers. And you know, as a professional in this industry, the best way for you to ship software is to ship software, right? And that's what you're being asked to do by your managers. And you know, as a professional in this industry, the best way for you to ship software is to automate as much as you possibly can. So why, if that is your job, are you going and asking permission to do your job? You should have just go and implement continuous integration, continuous delivery, automated testing. That's your job. But so many people feel the need to go ask their manager, is it okay if I go and implement continuous integration? And the answer is always no, because the manager just sees the deadline, that milestone, which they now feel is going to be in jeopardy if they give you permission to go implement something that they don't see any value in. Well, they're not going to see the value in it because it's you there at night. It's you there

on the weekend trying to make these ridiculous deadlines, not them. So for them to say no means there's no consequence for them saying no. But they believe there's going to be a negative consequence if they say yes. So my advice to the developers who are in the trenches is stop asking permission to do your job. Go do what empowers you to do your job better, which is implement a lot of these DevOps best practices. Now, the way that you can do it without asking permission is not try to do it all at once. Because the statement that you made, you have to put on the brakes for a while. I only hear that in scenarios where they think they have to stop what they're doing and build an entire DevOps pipeline from beginning to end.

So they say for a period of a sprint or two, we're just going to stop adding features. We're going to stop fixing bugs, and we're going to go build this end to end pipeline. And it's going to be perfect for a shot, right? It'll never be right. You're absolutely right. It'll never be right because you're going to struggle and you're going to find out things you didn't know, the things you thought you knew didn't work out the way your assumptions were all wrong. And then all of a sudden, you're never going to get permission to go back in and take a break to try to figure that stuff out again, because now you've lost an entire sprint of zero productivity and have nothing to show for it.

However, let's replay that. You're sprinting every three weeks, and one of your developers takes 15 minutes out of their day and goes and tries to set up a continuous integration build for your system. 15 minutes, that's like a portion of your lunch. No one is going to fault you for making that investment in your future. You don't even have to go ask permission for 15 minutes to go set up a continuous integration build for your team. And no one on your team has to stop but you, which is another thing I try to explain to people. You can have a team of 13 developers all committing into a repository every single day, one developer says, you know what? I think that we should have continuous integration. So what I'm going to do is I'm going to take 15 minutes out of my day and I'm going to go learn Jenkins or VSTS or TeamCity or whatever it is that you want to use for your CI system. And I'm going to set it up. No one else on your team will even know that you set it up. They're just going to keep committing code like normal. And what you're going to start getting are signals

on the quality of the code that's being checked in because that CI system will start to fail if someone breaks the build. And you, the developer who created it, will instantly get value from that investment because we've all been on that team that does not have CI. We come in in the morning, everyone clones the repo or pulls the latest changes or gets the latest, and everyone's broken because someone last minute checked in a code that broke everything. Now, had you had a CI system, when you came in in the morning, you would have seen that the build is red and no one should do a get latest because if you do, you will be broken like everyone else. Setting up that CI system gave you instant value in 15 minutes or less. And you did not have to go ask permission

to do that. Now that you have that, what's going to be interesting is that water cooler talk. When everyone's sitting around complaining about the person who broke the build and you're working and they're going to be like, didn't you do a pool this morning? Like, no, of course not. The build was broken. How did you know that? Well, see, I have the CI system running and it showed me that the build is broken. Holy crap. Can I monitor that too? I wish I would have known about that. And again, you didn't have to convince anyone that a CI was a good idea. You didn't have to go create charts or run statistics or go do any research. You just implemented it and you let the results speak for themselves. And now once everyone is on board for CI, you can start trying to take that package that is being built and deploy it to a dev environment. And just like you did with CI, you slowly start to build this pipeline. And as you start to build it, certain

areas are going to realize, wow, man, if I could automate that because that takes us forever. Or, man, that's really, really risky. Every time we do this, I get an anxiety because I know that's where it's going to screw up. Let's figure out a way to automate that. And you slowly build that pipeline. You never have to put on the brakes and you never have to go ask for permission. Can I tell you why I love everything you just said? Sure. ask for permission. Did I tell you why I love everything you just said? Because Andy, this is

bringing you in, especially so many stories, even our own story has all been around the fact that upper management has bought in and given everybody the tools and the protections and everything they need in order to go ahead and do this. And we've seen a ton of times when people have had success, when they've had buy-in from upper management to give them full support behind it. And another thing we've seen that it seems to be more difficult for people who don't have that support. But to me, at least what you're describing is a way to get around not having that support from upper management, not even asking, right? Because if you ask, you might get the no, just going ahead and doing it in a way that's not going to intrude on you still putting in quotes shipping it because taking 15, 20 minutes out of your day is not going to impact your deadline and you could still make improvements that will suddenly slowly adopt themselves into the culture.

So I thank you. I love that. And I love the fact that there seems to be there's obviously you've seen it and there's hope, but yeah, you were about to. That's just it. I mean, you, you articulated it perfectly is that don't go ask for permission because when they say no and you do it anyway, even if it's successful, you're being insubordinate. You know what I mean? But if you don't ask, and what's so weird – I said this publicly on Twitter the other day. So this company is using TFS 2015. I was like, well, upgrade to 2018. Oh, well, there's all these reasons on why they don't – management doesn't want us to take the time. I'm like, hold on. How often does your – he called them the business. He said, how often does the business connect to TFS?

I said, I bet you you could upgrade your TFS right now, and the business would not even realize that you did it. So why are you going and asking people who don't even use the system, nor would they even know if you upgraded the system, if you can upgrade the system or not? To me, it makes no sense. I said, if you need to convince them to upgrade, which is absurd to me, because in my mind, I'm thinking you should have to convince them on why you're not upgrading, right? Because it's the logical thing to do. It makes sense to be on the, on the great latest and greatest features of the product that are enabling you to do exactly what they're asking you to do yet here you are with your hands tied when in my mind it is your job to continuously ship and you know i hired you because you know how to do your job better than i do so go do your job and if that's implementing these devops best practices by all means you

should be doing that and not wasting my time asking me for permission to do so. I want to – I mean I want to just quickly add one thought to – well, actually two thoughts. First of all, to the business with the updates, you should probably ask them. You should probably ask them, hey, when you look at your phone, at your private phone, and you see there is a new version of the app that you're using every day, do you update or not? Of course you update. Interesting. you're using every day do you update or not of course you update right interesting and so and the reason why you update is exactly for the same reason that you just said because you have the latest and greatest and the upgrade is seamless because vendors like in this case microsoft make sure that an update from vsts is going to give you new features and it hopefully is frictionless

so that's why i do it right and now the thing what I love, coming back to what you said earlier about taking the 15 minutes and set up a pipeline and visualize, and I think that's a key thing, visualize the quality status of your pipeline.

So Donovan, you will see it when you are in Linz or by the time this airs, you will have seen it. One of the things that we came up with our engineering team

is we call it the Dynatrace pipeline ufo the ufo is like a flying saucer you'll hear ufo there ufo yeah okay good to know yeah so um so what we have actually done what our engineering team has done is they were just as frustrated with broken builds and they said we want to visualize the status of the current build pipeline and the state in production in a way that nobody can ignore it when they walk out to the office so they build this little 3d printed device it's hooked up with wi-fi and basically what we do whenever a jenkins build runs through and the jenkins build fails we make the ufo you know line light up in red or in shades

of red depending on when the pipeline broke or where it broke and then uh it's strategically placed next to the coffee machine so every time you're committing uh code changes and the build runs and you walk over to the coffee machine you will see whether you broke the build or not and i think that's uh yeah and that's that's something that was if you asked if you will see whether you broke the build or not. And I think that's, yeah, and that's, that's something that was, if you asked, if you will talk to our team members over there, and it's definitely been a major part of our DevOps transformation, a visual indication, if somebody broke the build, or if somebody broke production, obviously, that's even worse, but it's just the feedback that's very important. It is. Again, it's monitor and learn over and over and over again.

And you're monitoring your build and you're learning from the failures and hopefully being able to mitigate how often that build is going to be broken in the future. So I love your advice to developers. Take 15 minutes out of your day. Nobody will punish you for that, but you will learn a lot. And more importantly, not only will you benefit from it if you automate something that where you spend and waste a lot of time every day, but eventually your teammates will also benefit and your business will benefit from it. This is a great suggestion. Any other suggestions? What are the other small things people can do to actually start down the DevOps path? It's a mindset that's the hardest part.

So I defined DevOps for Microsoft. And if you go to visualstudio.com, whack DevOps, or aka.ms, whack DevOps, you're going to see a definition that is, DevOps is the union of people, process, and products to enable continuous delivery of value

to our end users. And to me, I think that one sentence sums up the essence of what DevOps is about. Contin, continuously delivering value, right? That is the goal, no matter if it's mobile or otherwise containerized or otherwise, no matter what language you're programming in, if you're writing software, it should be to continuously deliver value to your end users. And to do that takes three things, in my opinion, people, process, and products. The products and the process,

that's the easy part. And I always joke on stage that you're going to hear me say things like CI and CD and infrastructure as code, configuration as code, and you're all going to be nodding yes. And I'm going to say, are you doing any of this? And you're all going to start nodding no. And the reason why is because the people don't get it. And that's the hard part is figuring out how do you get your people to understand the importance of implementing some of these DevOps best practices. And again, when you're trying to get your people to understand it, as a Microsoft PM, I took it as my responsibility to make our products just so easy to use, so simple and so intuitive and so obvious that they're going to make your life easier, that convincing the people to use them is easier as well. And then once they start to see that process in motion, it's going to make life a lot easier. So I would say as another step is to evaluate the tool chains that are out there, evaluate the products, and really make sure that

you pick a really nice, good, easy to use product for what it is you're trying to do, and allow that product to help you convince the people. Because if it's a cool product, as developers, we're all curious, right? We always want to play with the cool new toy, but not if it's hard to use, not if it's putting up more roadblocks or makes my job feel harder. So one thing that I would really encourage you to do is make sure you think about what is it that we're trying to build? What tool chain is going to be the best to do that? And don't get caught up in the best of breed, which a lot of people, I call them vanity pipelines, which means they built a DevOps pipeline out of vanity alone. They wanted to have the best CI system, the best monitoring system. And on paper, it seems to make sense, right?

Why would I not want the best of everything? Reason why is because sometimes the best of everything comes from different vendors. So the best CI system might not be owned by the best CD system, which might not be owned by the best source control vendor. And when you try to stitch all this stuff together, you start paying an integration tax. It becomes very expensive. And someone's job in your organization to go stitch all these different tools together. And when vendor A decides to upgrade a portion of their product line, does that now break vendor B? And now, instead of you adding value to your product, someone's now maintaining your tool chain,

which becomes very expensive. So instead of trying to build best of breed, what you need to do is build what we call best fit. What is the best fit for our organization? Is it a tool chain that has continuous integration, continuous delivery, source control, package management, all built together that automatically talk to each other so that that pipeline is not something that we focus on. And as a developer, all I do is commit code. And then the rest of that stuff magically happens. Or do I truly need to replace a piece? And you want to make sure that when you're building that pipeline, you leave yourself the flexibility to pick the tool that's the best fit, even if it isn't from the vendor.

But I think you're doing yourself a favor when you reduce your vendor count. It might never be one, right? But the fewer that you have, the less tax that you pay and the more time I think you're able to spend on adding value and not maintaining your tool chain. So one other thing, like I said, you were asking what else could someone do is make sure that you're wise and you really put some thought into what is our goal with our pipeline? What are the best fit tools for us and how can we reduce our vendor count? Yeah, that's very well said. Obviously, from a vendor itself, I mean, this is what we also try to achieve with our product. We see a lot of our users out there doing tool consolidation

because over the years they added a lot of different tools and having something available from one single vendor that covers a broad range of use cases well integrated makes it much easier because then as you said you don't you need less time and less effort in maintaining everything keeping everything up to date and making sure everything keeps stitching together. Yeah, and what's interesting is that historically, you had no choice but to build it yourself because there was no one vendor that had a solution that worked for everything. When TFS first came out, 2005, that time frame,

it only did.NET on Windows very well, right? So we didn't have an answer for the Java devs and the Node devs and now the Go developers and PHP and Python and all those. We didn't have an answer for you. You could get it to work, but you had to hire a consultant and it was brutally painful afterwards, right? And so for you to have that nice pipeline, you had to build it. And it usually meant integrating multiple vendors. But we're at a new age in this world where not only Microsoft, but there's other vendors out there who are trying to say, nope, we're at a new age in this world where not only Microsoft, but there's other vendors out there who are trying to say, nope, we're going to give you that turnkey solution,

but not force you to be locked into one particular vendor. Because that was the fear. If we put all of our eggs in the Microsoft basket, and then someone comes up with a better CD system, now we're stuck using Microsoft's, and we have to wait for them to update it. But if you look at something like VSTS, or even TFS for that matter, we have service endpoints, we have REST APIs, we have webhooks in and out to where by default, out of the box, all of our stuff happily talks to each other. But if you are hell bent on using Jenkins, because you already have an investment in Jenkins, you already have thousands of builds and projects in Jenkins and you don't want to move them, we're not going to force you to. But what's really nice is that Jenkins then ties into the rest of our ecosystem.

You can use our source control. You can use our deployment pipeline. You can use our package management. You can use our Azure container registry for your Docker images. So that way, you still only have two vendors versus a dozen vendors, which is really nice as well. Hey, coming back to your DevOps transformation at Microsoft, I believe in the beginning you said you went from three years to now three weeks deployments, meaning every three weeks a new build gets deployed of VSTS. Can you tell me what happens in these three weeks? So I assume you mentioned the ED on dog for the drink around champagne. What happens from code commit until that change is out there three weeks later? Got it.

So what happens is that we have about 50 feature teams that build VSTS. All 50 feature teams are on the same three-week cadence. So we're all agile shops, all agile teams, and we call it aligned autonomy. The alignment is every three weeks. The autonomy is you can do whatever you want within that three weeks, but you better be ready to ship three weeks from the beginning of that sprint. So some teams run them as three one-week sprints internally. Some of them use test-week boundary. So what will happen is at the beginning of the sprint, all 50 program managers who basically play the role as the product owner on each of those scrum teams will write up an email saying, these are the tasks that we have accepted in this particular sprint, and we intend to deliver three weeks from today. And we basically get flooded with 50 emails saying, this is what we

intend to do. Three weeks from that date, we get another email from that program manager that has a three to five minute video in it, which would essentially be the sprint review you would have sat in. But we can't sit in 50 sprint reviews. Not only do we not have enough time, they're all over the world. We have teams in Raleigh, North Carolina, in Redmond, Washington. We have them in India and Hyderabad as well. So there's no way you're sitting in 50 sprint reviews. So what we do is you get 50 emails, which you can watch in your leisure, that all have three to five minute videos in them saying, here is us demoing the actual features that we got done as promised in the last three weeks. Any explanations on features

that might not have made it and everything that is ready to actually be shipped. And then what happens is we go through this process of producing the release notes, which is automatic, semi-automatic, I should say. The release notes are actually scraped right from the work items and then put into a markdown file so that we can ship the release notes for everything ready to ship as we're deploying the features out into production. VSTS uses what we call safe deployment practices. So it doesn't go to one server and then everyone in the world gets the newest version of VSTS uses what we call safe deployment practices. So it doesn't go to one server and then everyone in the world gets the newest version of VSTS.

It goes through different rings, and the rings are made up of scale units. I don't have enough time on this call to explain that, but I have a show on Channel 9 called DevOps Interviews where I sit down with the engineer who runs our pipeline, and we go into crazy detail of what a ring is, what scale units are, how they're made up, and some of our biggest customers. But in a synopsis here, we have, I believe, five rings or six rings now. Ring zero, which is the very first instance of VSTS that gets the new code, is the actual instance that the VSTS team works out of. So we're the first people to get our own software. We have to live with it for 48 hours and we do our daily job on it. So let's say, for example, the deployment started Monday morning, stamp zero gets it. Monday and Tuesday, we're doing our job on the latest and hottest bits of our product. And we monitor our telemetry. We're looking for any new issues. We're looking

for performance issues. And if and only if it survives 48 hours of actually being used by the VSTS team to build VSTS, will we then deploy it to the next environment, which is ring one, which is our early adopters, our MVPs, people who have opted in to getting early bits for whatever reason. They just want to be on the bleeding edge or they just want to help Microsoft build a great product. It sits there for another 24 hours where we repeat the monitoring of our telemetry, monitoring for issues. And we rinse and repeat this till it actually gets deployed all the way out into the final ring.

That's about a 10-day process, assuming everything goes perfectly. So if we have no performance issues, no critical showstoppers, from the time we start our first deployment bits moving forward, 10 days later, you will have that code in every stamp of VSTS. So the three weeks, it's completely up to the team. Every three weeks, we come and say, okay, guys, where are we?

Everyone ready to ship? And then we start that shipment every three weeks at clockwork. So you say you have 11 days of of uh nothing until the next until that cycle starts again because because right because if it's three weeks oh no no no well i don't mean no activity but you have your it takes you 10 days to finally get the full widespread push and then in another 11 days you're starting that process again uh yes okay yeah yeah yeah so that's a good point. So the 10 days overlaps the beginning of the next sprint. So we sprint every three weeks and that never stops. And we ship every three

weeks with one exception. There's always one deployment that happens over Christmas and we never, ever, ever ship on Christmas. So that deployment gets skipped. And then the first deployment after the new year actually has two sprints worth of features in it. Other than that, every three weeks, like clockwork, we're starting the next sprint. And we have a SWAT team. It's like two individuals from each of the teams is basically babysitting the deployment for that 10 days. So if there's any hot fixes that need to go out, critical showstoppers that we find, we basically cut a branch that is that release branch

and any hot fixes are made there. And we deploy actually twice a day for hot fixes, bug fixes, performance fixes, things like that. But every three weeks are when we ship new features. And during development in the three weeks and you have 50 feature teams, are they doing trunk-based development? That means they all work on the same trunk?

Oh, good question. Absolutely. We all merge into master every single day, which is just bonkers. We do over 600 pull requests a day on average. So we run – and every one of those pull requests runs, I believe, at this point. I'll bring it up when I come there. But I have to check before I get on stage, because the first time I looked at a pull request, it was 40,000 unit tests run in about five minutes. The last time I looked, we were closer to 70,000 unit tests run in under seven minutes.

So every single pull request, that's 600 times a day, we run nearly 70,000 unit tests against the code to make sure it's of the highest quality before it even makes it into the master branch. So that pull request is, I'm going to run a build. I'm going to run 70,000 unit test. I'm going to run cred scans. I'm going to run linters against the code. I'm going to run static code analysis against the code. And if it passes all of that, we'll finally let your peers see it and give you their feedback on the code. And if they all bless it, it'll get merged into master. A CI system will kick off and run a completely different battery of test against that code before it ever makes its way into a deployment. It's intense.

Yeah. And the reason why I was asking, you know, the trunk-based development, I actually just wrote a blog from Dave Farley. He posted something last week, actually, March 30th is the date. Continuous integration and feature branching where he basically talks about don't feature branch. It's all trunk-based development. That's the only way because that's what continuous integration really is all about, making sure you continuously integrate your code changes. And if you have feature branches open too long, then obviously you're hiding code from your trunk and basically hiding it from your continuous integration and that's obviously not good and that's why but

it's very great to know that and again reinforcing what I said earlier it is not only possible in a two people startup it is possible in an organization like Microsoft we have 50 different feature teams working in parallel in three-week sprints and delivering new features on a continuous basis. And then your staged rollout in your individual rings, you know, that's great. We internally, we do something similar

where we have first internally, then different early access customers or like our power users. And it seems that's a great best practice. Yeah, it gives you exposure control, right? So if something bad were to happen, none of our customers even have to know that something bad happened. We're the only ones who pay the price.

What's also interesting about you being the first people to use it, the quality goes up quite a bit, right? Because you're the first person to pay the consequences for a mistake and it's going to jeopardize your ability to do your job, which I found very interesting on, man, if we're the first, this has to work, right? We could stop 50 feature teams, about 10 people per team from doing their job, right? And that could be catastrophic for us, let alone our customers who are, some of them have teams even bigger than that using VSTS. We just simply can't run the risk of them not being able to do their job.

So you have by now given this presentation at DevOne. So I'm sure if people want to, you know, watch your full keynote presentation, they can go to devone.at and watch the recordings because I believe they will get put up at least the ones from last year up there. So I'm sure they will put it up again. Now, Donovan, you also said that you have a second presentation and maybe just a quick highlight on what the other breakout is that you're doing. Right. So they scheduled a chalk talk for me as well. And what I'm going to do is take a application that I've been demoing for a while. And what I do is I basically dust off this demo and I add to it new technologies, new features,

new capabilities that we have in VSTS and in Azure. And the application is a simple MVC application. That's, I call it people tracker. It's a simple CRUD application with a single table in it that stores people. And what I do is show you how I can actually change the database schema, the app services, the mobile app, and the Docker container that runs the front end, all using VSTS. And it's a really fun demo because when time permits, when the demo was a little bit smaller, right now the build takes about 11 minutes, so it's a little hard to do on stage. But I used to do it on stage while people were using the app. So people are out on their phones, they're using my website, and they can see it only has first name and last name. And while they're using it without dropping a single

bit or an error, I would push a change that added middle name. Someone would refresh their page and then boom, they see middle name is now there. The database has been changed right from underneath them. It's just this really powerful demo showing you how you can move at cloud's cadence today using the tools that we provide at Microsoft. Since the original demo, I've added Docker containers, mobile applications, monitoring the whole nine yards. And what's really cool is that the app is built with – it has an Android and iOS front end. It has a – so that obviously requires a Mac to build. We have a Docker.NET core image running on Linux, which takes a Linux machine to build. And then I have a.NET web API that's built on Windows and a database project.

In a single build definition inside of VSTS, I'm using hosted agents from VSTS, which means we have hosted Linux machines, hosted Windows machines, and hosted Macs for you now. So you don't have to install anything. You just go to VSTS, create an account for free, check in all this code, and I'm building the iOS and Android on a Mac, the Linux container on a Linux machine, and the app service and database project on a Windows machine, all for free, all in one build definition. And then the output of all that, the APK, IPA, Docker image, zip file, and the database project

all get handed over to release management, which you then see deploy into Azure, the ARM templates, the database, using Hockey App and App Center to deploy the mobile application, run Selenium test. And it's just unbelievable to see everything come together

for a real world multi-tier application. And then people start to, the lights start to come on in people's heads, like, oh my gosh. So we were doing DevOps for the quote unquote front end, but we were doing our database manually. We didn't know that we could integrate the entire solution into a single pipeline. And that's what I try to instill in people before they leave is you don't need 12 different pipelines. You don't have to do some of it automated and some of it manually. You can literally deploy your entire system in an automated fashion. And that's what I show on stage. And is this something that people can, when they're at the conference, they get access to,

or is this something that if there is there a link, people can go and try this out themselves from anywhere? I had a, I need to update it. I had a GitHub repo and that's probably what I should do. I should put it all back in GitHub right now. Right now it's all running inside of Git, inside of VSTS, but I might take that as a challenge to make sure the latest version of PeopleTracker in my GitHub repo, because there is one there, it's a little older, is actually ready to be built by anyone. The app itself is just at peopletracker.us. So they will be able to access the app while they're on while I'm on stage. They want to play with it, which is kind of fun because they usually put their buddy's names in there or their company's names in there.

And then I bring it up. Right. And then if they're recording it, they're now they're in history forever because there's this recording that shows all their names during my demo, which people seem to have a lot of fun with. So that will definitely be happening one when I'm there as well. But it's a fun little app that I'm basically dusting it off right now, making sure that the code still builds, upgrading all the APIs and making sure my mobile app still builds because mobile development's hard. Even with Xamarin, it's just freaking hard. And you got to make sure that all the SDKs on both your build machine and your dev machine match and things like that. So that's the scenario I'm going through right now. Cool. And then I think we both, or the three of us,

we all agree that, or agreed, that we're going to do another recording in a couple of days from now. It will air, obviously, a couple of days from this one, probably. But we wanted to talk about exactly this particular scenario, about this particular app. And then I think we gave you also the challenge of including the concept of the unbreakable pipeline, which is stuff that we have been building over the last couple of weeks.

And we've actually been on the road and helping customers build the unbreakable pipeline. So like baking, monitoring into your end-to-end delivery pipeline. And I'm very happy to then chat with you and see what your findings are and how we can enlighten the whole world on how to build better continuous delivery. And, yeah, that's pretty cool. Yeah, I'm excited. Yeah. All right. Brian, do you have any other questions?

Do we need anything else? No, I mean, there's a lot of thoughts I had going through. They've all kind of been discussed at length, though. So no other. I mean, yeah, one last question, actually, that did come up. So you mentioned you have 40, no, 50 different teams. Feature teams, yes. you have uh 40 different no 50 was it 50 different teams feature um because what i'm hearing is this amazing you know huge amount of stuff being pushed out what is the total size of or if either if you have an average of what the sizes of those teams or how many in general people work on this project it's usually 10 to 12 people per feature teams times 50 feature teams okay so it is you're talking somewhere in five to 700 people well it's very sizable still in the end. I, and I'm just asking that more putting in context of,

you know, if someone's listening and thinking, oh, well, they have a lot of people, but even still, you can get this done with, you can, you know, this is a huge project you're working on for a huge enterprise organization. And if you're not working at a huge enterprise organization, you obviously don't have the kind of personnel that Microsoft has the ability to hire. But you don't need it. You know, you're working on small you're working on feature releases that are much smaller in scope and you can still get this done. People are getting it done left and right all over the place with small teams. Right. I am the sole developer on a couple of pet projects of mine, and I do all of this stuff in every single one of my deployments. Matter of fact, I pushed out a feature the other day and one of my customers said, oh,

Donovan, did you change something? Because it's not working for us all of a sudden. And what happened was it worked great for the customer that I had implemented it for, but it wasn't working for the entire larger group. My testing simply let something escape to production. And because I have a CICD pipeline set up, I was like, hold on. And I basically redeployed the previous build out into production, knowing that it would bring everyone back to a known good state. And I wouldn't have been able to do that. I did that from a plane. I was flying home from Turkey. I got internet access. And I said, just put the old version back in there real quick for me. That way I can relax on the plane. When I landed, I came in,

got some more realistic data from what was happening in production, saw the error, was able to fix it. I just committed it to the repo and left. And I knew that my pipeline was going to deploy that new feature, that bug fix, all the way out into production. There was nothing for me to do. And I always tell people, if you have to do more than commit code to a repo to deploy it, you still got work to do. Because that's all a developer should ever have to do is commit code to the repo and that pipeline should just hand carry it all the way into production for you. I was going to say the two amazing things about that story is number one, you did it from a plane and number two, more importantly, you actually had enough of a

connection to be able to do it from the plane. Yeah, I've actually had quite a bit of luck with lately. A lot of people have been like, man, the internet's so horrible. I mean, I fly United a lot, and I've literally streamed YouTube and videos from Pluralsight and Linda Learning while I'm on the plane. So it's getting better. I was just saying,

deploying into the cloud while flying above the clouds. Pretty cool. It's an amazing world. It's a fun time to be a developer. So Andy, do you want to uh summon the summer raider then let's do it i want to summon the summer raider yeah so what i learned today is that you're obviously part of a of a special group of people um the special league of devops advocates and you have your little what's the hashtag again it's l-o-e-c-d-a so just remember league of extraordinary cloud devops advocates it's the hashtag again? It's L-O-E-C-D-A. So just remember, League of Extraordinary Cloud DevOps Advocates.

It's the first letter of every one of that. So L-O-E-C-D-A. If you use that hashtag, my entire team will come read your tweet. All right. Perfect. So let's do that. But obviously, let's make sure with power comes responsibility. Let's not abuse it because you want to make sure Donovan and friends have other stuff to do as well.

So then basically what I learned, I liked your, hey, it just takes 15 minutes of your day to get started. Start automating things that should be automated and should no longer be manual. Visualize the state of your pipeline. This avoids having your colleagues check out your trunk and then basically end up with a broken build anyway. And you can avoid this by just making this more visible. I think what you also said, I like this best fit versus the sum of best breed, not the best tool or the best tools in the world

may not be as good as the best fit for you. So let's have a look at the best fit. And then just knowing that DevOps and continuous integration and continuous delivery is something that can not only be done by small shops, but even a company like Microsoft, and in this case, the VSTS team, they have 50 feature teams working on three-week sprints and all aligned and trunk-based development that obviously helps, right? And being able to push out changes in the cadence that you have with such a

globally distributed team. And also I think a lesson learned, you said it took you seven years and you're still in that transformation. I think the transformation is never really done. So don't expect it happens overnight. It will take a while. Don't be discouraged, but start with the small wins. And don't think it has to

be a full stop break of not doing anything for a while, but you can actually make small incremental improvements until you reach your final destination. All sounds great. Yep. Agreed. Perfect. Thank you, Andy. My last thoughts, if I may, is that, number one, again, I love the story about this 15 minute thing. I'm going to reiterate that again, just so you know, I guess I wouldn't reiterate it again, uh, but, uh, reiterate it because, you know, there are so many stories that we

hear about even our own story with this whole upper management buy-in. And I think even in my head, there's a lot of thought that you can't do this without that buy-in, but you just gave a perfect example of how to go through it and not get that buy-in and just do it. You know, get her done, as that guy says, whatever. The other thing that I love, and one day I'm going to get used to this, I promise, but I love being completely flabbergasted about the new Microsoft. You know, for those of us who've been around technology for a while, of being completely flabbergasted about the new Microsoft. You know, there's that, for those of us who've been around technology for a while,

there's the old, what's called the monolithic idea of Microsoft, as you mentioned. Every three years you had a deployment, Windows operating systems, what year did Windows 98 come out in or Windows 2000? It was always the joke of how long it takes things to come out in. Although they were great products, it was just a very slow-moving, everyone was slow-moving back then. Sure. But everybody, when Java hit, everything else was cool, but Microsoft was not cool. And now you have, what's the.NET one?

I'm blanking on the name. .NET Core, you have Microsoftrosoft running linux you're gonna have microsoft content all this really awesome stuff and like microsoft is cool again and i just can't it's every time we have these discussions with microsoft people i'm like wow that's amazing and like i promise one day i'm gonna be like well duh of course you know but right now it's still awesome hearing about these all the time because i i i always love an underdog and micro and not that Microsoft was an underdog It's such a huge company, but in terms of the cool factor It was always like well Microsoft and and I think we were talking to one of her own

colleagues a while back Andy The guy who does all the dotnet core stuff and we're saying yeah you can actually go to a conference now and wear like a Microsoft or dotnet shirt and It's and it's okay. He won't be ostracNET shirt. And it's OK. You won't be ostracized. Yeah. So it's awesome. I love hearing about it.

I really appreciate you coming on. It was a really fun conversation. So thank you. Any last thoughts or words from you, Donovan? No, I mean, I think the new, I always joke that this is not your daddy's Microsoft. And for us, the older crowd, it is like, wow, I can't believe they're the largest contributors open source in the world. But I'm noticing that people who are coming out of college, that's the only Microsoft they've ever known. And to me, that's just like, why are you not amazed by

what we're doing? Because this is the only Microsoft we've ever known. And that's a good thing, right? That a lot of people with that stigma or that almost built in distrust for Microsoft are becoming less and less predominant. And even they are having to rethink like, wow, man, this is a new Microsoft. This is a Microsoft that allows you to carry a Mac. Matter of fact, I have two Macs, an iPhone and a Windows machine, right? So it's like I have more Apple stuff because I work at Microsoft. But do you have a Zoom player? Oh, no, no, never. I wasn't here then.

I wasn't here for that. But what's funny is that the first time I ever installed Linux was after I joined Microsoft. So that just gives you an idea of the different type of Microsoft that I live in that I joined and then I started installing and using Linux, which is really weird to me. Yeah. But I love it. Well, excellent. Thank you so much for taking the time to to record with us today uh as andy mentioned we're going to go ahead and schedule another recording to tackle the next topic so um if any of our for all of our listeners who enjoy today's show we'll be

having donovan back very very soon um i certainly enjoyed it and uh thanks a lot good luck at your good luck at your talk at devops at the DevOps conference, and enjoy Austria. Thank you so much. Oh, and what's the term he should learn how to use, Andy? OIDA. OIDA, exactly. Go look up OIDA. Yeah, Andy's got a great video for that.

Yeah, it's the only word you need if you want to get around in Vienna. All right, you have to send that to me, too. Yeah, O-I-D-A. Oida. It's a great YouTube video. And it's hilarious. Awesome. All right.

Thanks for having me. Thank you. Bye-bye.