PurePerformance - Unbreakable Continuous Delivery with Jason Westerhouse

Episode Date: January 29, 2019

In this episode Jason Westerhouse shares details on how KeyBank is integrating Dynatrace across their DevOps toolchain and driving automated quality gates, continuous feedback and faster incident response time by embracing GitFlow – using Jenkins, containers and release automation to automate CD.

Coming to you from Dynatrace Perform in Las Vegas, it's Pure Performance! Hello everybody and welcome to Pure Performance and PerfBytes coming to you from Dynatrace Perform 2019 in Las Vegas. I'm Brian Wilson and my co-host is the one and only Mark Tomlinson from PerfBytes. Hi Mark, how are you today? Hey Brian, how are you? Pretty exciting going on. Things going on here, right? It's nice, it is. It's nice here. We're enjoying our time. I like it. Mark, we have a guest we want to bring on. I'll let him introduce himself more properly,

but we have Jason Westerhaus. Jason, why don't you introduce yourself? You're from KeyBank. Introduce us, tell us a little about yourself and what you do. Sure, yeah, yeah. First of all, how are you guys doing? Great. So, yeah, I'm Jason Westerhaus.

I'm from KeyBank. My title is Tech Lead of Monitoring or Application Performance Monitoring at Key. Typically, that entails kind of the Dynatrace suite as far as Atman, DC. I was brought in to kind of help out with the Atmont Dynatrace transition. So all last year was kind of kicking it up, Dynatrace getting some test apps in there and kind of our main applications inside of Dynatrace. And this next year, in kind of what I've talked about, is transitioning Dynatrace into more of a, less of a monitoring after the fact of in-production monitoring, where of course that's really good, but now we're wanting to bring it into that DevOps space

and eventually into no ops if we can. So getting the developers involved and getting that process figured out to where Dynatrace can really help out the application teams. You know, I wish I knew that you did the Atman to Dynatrace can really help out the application teams. I wish I knew that you did the Atman to Dynatrace conversion because just on Monday, I taught the class Dynatrace for Atman users. I could have had you come in or got to talk to you about what we should include in that class.

It's a new one. So you just did the presentation with the rest of your team um was that guy and mick right from key bank yeah yeah so we actually had two two separate um uh speaking points here one the first one was was mick kind of giving an overview of what the what the strategy of key bank uh is and was and and um then yeah myself and guy kind of walked through how we actually did that. And this was the DevOps to Know Ops at KeyBank talk, just for in case anyone was at it or was looking at it on the schedule and wishing they could be there.

You know, I was looking through some of your presentation, and obviously it's kind of outlining how you did that. But I want to start by asking you, there's one slide in there in particular that really got me scratching my head is like how do you go about doing this i think probably a lot of people um who are thinking about this have the same thought you have the the don't boil the ocean slide and how do you how does one go from devops to noops without boiling the ocean because it sounds like you you've done it or you're in process doing it but it seems like the only way to do it, sort of.

But tell us what your approach was on that. Yeah, so I think the main concept of that was try to get to that no ops or DevOps space where we're trying to utilize as much as we can with Dynatrace without disrupting what's going on currently, right? We don't want to have the developers or application team absolutely change their process where, you know, we might add days or weeks or even typically hours onto their process just to

add, you know, more monitoring and more space around what we can do with to help them out. So I think that's kind of the main concept of it. We just, we just don't want to interrupt the things that are going on and kind of add to it rather than kind of disrupt the, um, their processes. So how do you, it almost sounds, and I might be, again, I'm, uh, I couldn't attend your presentation, so I'm just kind of going from the deck here. But it almost sounds like you went from Atman to Dynatrace, and with Dynatrace going to NoOps, this to me almost sounds like a cloud migration where you're going from maybe monolithic self-hosted applications

to some sort of microservice-y, container-y kind of thing in the cloud where going all at once is difficult. Would you say this is a similar approach to switching, changing your monitoring from something like Atman to Dynatrace to NoOps? Were you taking that sort of piece-by-piece approach where you say, we'll take a small piece of this and start moving it over?

Or was it larger? Because as you just said, you don't want to interrupt the normal flow of things or you want to have the least amount of impact. Especially I can imagine most people would say, we don't want monitoring to have an impact on our schedule. So was it an approach like that or how would you describe that? Yeah. Yeah. So that's typically, so when I started, it was Dynatrace is just coming in or just walking in the door, right? Everybody was on Atman. And typically, you know, the WA space and kind of the web logic, stuff like that.

And so the move was this, our three main applications at Key were moving over to OpenShift. So, you know, kind of the microservice type area there. So we do have them kind of hosted in-house, but they are talking about moving to the cloud. So that's also where, you know, this comes into where we really need to get these, you know, our strategies and our processes figured out with monitoring before we move to there. Again, it's that little three applications. I say little, but it's, you know, our biggest applications. This next year, we're actually gonna be working on moving all of the rest from Atman. And those still are on Woz. Some of those are going to be moving over to OpenShift as well. But it's,

like you said, it's kind of doing the most monitoring that we can and the most help that we can do with this development in DevOps or no-op space. But it is kind of going to be a slow process where we actually get our main OpenShift applications and then adding in the other applications that kind of help out with you know that those applications that we already have over there in OpenShift. Also it looks like you're I'm looking at one of the other slides and you have your pipeline right you're using combination of very common pipeline tools. And it seems some of the blueprinting or some of the stuff you all figured out was where to fit in Dynatrace into this pipeline. I see one of the observations from the blueprinting was the, you know, two,

Mark and I both have an extensive test and background in load testing. So too late testing, you know, 8 a.m. calls to discover 8 AM calls to discuss the problems that were done. When you walked into this, you said about a year ago, was a lot of the monitoring baked into the pipeline or was that something you figured out recently as part of this blueprinting session? And what was the, if there have been improvements in that, what have you seen in that? Yeah, yeah, definitely. I think, so when I first started, like I said, we had Atmana and DCRUM. So monitoring was there and it had been there

for a number of years. So there was a good kind of concept of, you know, at the 8 a.m. calls that you mentioned, hey, what did Dianne Trace see? And Dianne Trace was kind of that blanket term, right? And we do have a monitoring team. And before I started, you know, it was run by great individuals that were able to assist with that, right? So it wasn't 8 a.m. scramble, try to figure out what monitoring was showing. Typically, the monitoring team kind of already had an idea what was going on, and it was more of just a recap. Hey, something went down at 8 a.m. or 8 p.m. last night. This is what monitoring saw.

Whatever, right? That greatly improved with Dynatrace and the actual Dynatrace product now. Now we have more application teams and application performance people looking at dynatrace when the issue comes up or before the issue comes up. There's a number of times where we have just kind of a little jabber chat going where someone posts a screenshot of dynatrace. And that really helps. And especially when we start doing it in the performance space where we're doing the testing, right? They have a Dynatrace dashboard up where, you know, before they were trying to look at Atman, which was really helpful,

but Dynatrace really bubbles those problems up and really helps out. And that's what we've been seeing as we're kind of going through this process. And we're just at the beginning, too. So this is only going to get better for us in the DevOps space. Jason, one of the other things you mentioned on the observations from the blueprinting, it sounds like a step back from sort of continuous deployment back to waterfall in dev or in engineering. How did that work out? And was that expected? Yeah, I think it was kind of expected. I say that kind of loosely, but it's one of those

step back and relook at it kind of things. We're not forcing ourselves to go back, but it's kind of that where we need to kind of look and rethink how we're actually doing this and then approach it differently. So it's not, I mean, I mentioned earlier that, you know, we don't want to disrupt things, but we just want to make them better. Right. And sometimes that is taking a step back and rethinking how we're actually going to do this. Did it open your eyes to kind of the maturity of the engineering skill? Like they start getting more feedback and more truth, I guess, learning more truth about the real impacts of their code, of their decisions. And did that sort of kind of take them back a little bit?

Yeah, I think so. I mean, there was always, you know, and that's why we have, you know, those testing, those performance testing teams. So we're able to see those. And like I said, with Atman or Dynatrace before we even started this DevOps stuff. But it does help, right? It kind of, it's that real quick feedback that we weren't missing, but it just kind of, like I said before, bubbles up to the top where you actually be able to see that. And you get those, oh, I just did

deployment. You know, this was the direct impact to the application. And they know exactly what, you know, what was going on. I might have to do some, you know, kind of investigations. But you know kind of my direct impact on the application. So it's definitely helpful. Right. Yeah, definitely. And I think, Brian, to the boil the ocean comment, we see that as well with some of the really early implementations of Atman, the first times people would actually see the interconnections across a system, you get overwhelmed all the dependencies that come out of that.

So I do like Jason's idea of the island of sanity. It's kind of a peaceful term, the island of sanity. I'll have to bring my, I'll have to have my ukulele playing at that. That would be good. And then Chris Pine washes up on the shore and and then wonder woman you know then there's a battle scene that was pretty good all right making a wonder woman reference why not that's fine why not why not um jason you were talking about um right there's the integrations and the tool test testing and all and and you said you're starting to take steps towards uh no ops and if

i if i recall correctly you said you you you're not 100 there at no ops yet but it's the journey you're on now correct yeah yep yeah we want to we want to focus on devops now and no ops is definitely that that uh goal that we're striving towards um you know that will take take a little bit of time because it's it's a process and we're still on the monolithic factor for most of our applications. But moving toward the DevOps with microservices and hoping to get to no ops, yeah. And I guess besides the platforms that you're using on your application side, right? Obviously, if you're monolithic, no ops is a lot more difficult. So once you start transitioning to containers

and serverless and all that kind of stuff, you can tackle that more. I would assume a large part of that then has to do, and this is my shameless plug here, is a large part of it, especially on your side, has to do with the fact that you can grab all this data from the API to feed those no ops scripts, right? To say, Hey, here's, here's, here's what's running.

Here's what's happening. We've got all the data that we're monitoring. And from that we can take action. And I don't mean to put words in your mouth, but is that kind of the way you're seeing it or what was, what kind of gave you all the vision to say, let's, let's move to dev no ops with this? Yeah. I think that was actually, you know, the vision to say, let's move to no ops with this. Yeah, I think that was actually the strategy behind that.

We want to be able to have that quick feedback and be able to use those metrics that we're already getting. So the API allows us to do that, whether it's posting these deployment events to Dynatrace saying, hey, Dynatrace, we're actually doing a deployment right now. And then Dynatrace knows if something goes wrong, they're able to tie it to that deployment event. Two, hey, we had a deployment yesterday. Let's see, pull it from the API into any tools, any monitoring tools that anybody else wants to look at or whatever, we were able to pull those metrics in and like Jenkins, look directly in our build, right? Instead of going back and forth, right, they're able to pull these metrics right in,

you can look into Jenkins and how those builds are doing. And that really helps. Just utilizing, like you said, the API that Dynatrace provides to either push or pull really helps out. You know, Mark, it's funny. It reminds me of going back several years. You know, there's all this talk about APIs now, and they're nothing new, right? We go back to the old tools we used to use. They all said it's APIs, and either no one really quite had the vision on how to use them well, or maybe they were clunky.

I don't know what, but now it's like time for APIs to shine. And the thing that I like to point out to people is that what these really mean, what the APIs really mean, I'm not even just talking about ours, but you know, this API integration of all these modern tools is you don't even necessarily really have to go into the tool anymore. If you think about this concept of no ops, you know, you don't have to now pull up Dynatrace and look at a dashboard and find some some data this is all just something being automated in the back end with the data that

tools like dynatrace or other ones collect and share and they can interact with each other as if they were living and fortunately they're not living yet um but you're not opening you're not interacting it's pulling you're becoming one more step removed from having to do that work, which is just really awesome. I would go. I think things have changed, though, to Jason's experience in particular. When you look at kind of the information you shared about GitOps, the way you looked at GitOps in your across the pipeline, Jason, you know, the APIs technically, I think we figured them out a long time ago. People have been getting system A to talk to system B and trade data and do all sorts of stuff. I think when you look at the self-healing ideas that we're hearing about this week, that we've been hearing about from Dynatrace for a while now, and looking at the way that Jason's sort of bringing no ops into that, it's applying those APIs for

information across the process and automating the default decisions that one might make based on information taken from APIs and applied to APIs of the automated process itself. And I think that's the magic of putting these APIs across all the tooling versus Brian, I think your point was, you know, APIs in applications have been around a long time, right? EDI transactions even, but now we're, we're actually automating sort of the decision-making process over time, um, to, to achieve exactly what Jason's, uh, uh, getting at here, which is pretty cool. Yeah, exactly. So kind of second that as well. Cool.

Jason, we have to ask you an obligatory question now. Well, two obligatory questions. Number one, is this your first time in Vegas? No, nope. It's my second perform and my second time in Vegas. Oh, awesome. Yeah. I forget, were you speaking last time or were you just an attendee?

No, just an attendee. Yep. Okay. Cool. All right. And last year, were you up or down when you left the town? Up or down? time or no you were just just an attendee no just an attendee yeah it was kind of all right and last year did were you up or down when you left the town up or down oh yes money wise i was i was very close to even which is oh that's good but i was but i was down but very close to even yeah very close to even that's that well you might call that success yes i do yes yeah because if Success. I do, yes. Yeah, because you didn't lose a lot. Now the other question is, now that it's 2019, right, and we just had the new year, do you have any performance-related New Year's resolutions?

Yeah, I mentioned it previously. We want to get off Atman and fully on it. Yay! Yay! Nothing off Atman and fully on it. Nothing against Atman. We really like Atman, but Dynatrace is that next level and we definitely want to get there. Yep. And we have a ton to do so. I started using Atman in I think 2009 or 10 and I think it's a wonderful tool, but yeah, it's, it's time to move on. It's yeah. Yeah. Anyway. Yeah. I'm, I'm sure our, my, my, my seniors would love to hear that. That's your goal to get off that month.

So congratulations. Hopefully we can help you do that. Thanks a lot for, for taking some time to speak with us. Are you sticking around after the conference to spend some time out here or are you heading back home? Yeah, not going home but yeah I'm taking off Thursday morning so going off to Dallas to visit a friend and really enjoy

some warmer weather because I'm from Michigan not too warm out there right now. Alright well enjoy thank you very much for being on with us. Cool. From Dynatrace Perform 2019 in Las Vegas. Thank you.

Thank you. Cool. Thanks, Jason.