PurePerformance - How to become a Performance Engineer to please our “Instant Society” with Mike Kobush

Episode Date: December 6, 2021

The rise of smart phones clearly created a new demand for “instant gratification” when it comes to interacting with online services through web sites or apps. To ensure services are available at any point in time without any interruption or delay it requires performance engineers to automate performance and scalability engineering into the development processes.In this episode we invited Mike Kobush, Performance Engineer at NAIC, to hear how he found his way in quality engineering and later on got hooked on performance. He walks us through current performance challenges as his organization is moving to the cloud. Mike also discusses why he is embracing automation as it makes him more desirable as an employee. He shares his goals and motivations such as: Learning something new every day!Tune in and get inspired by Mike. And remember: it always pays off to spend the extra time with somebody – even if it’s a Sunday afternoon in sunny San Diego!Show Links:Mike on Linkedinhttps://www.linkedin.com/in/michael-kobush-31-performance/

It's time for Pure Performance! Get your stopwatches ready, it another episode of Pure Performance. I'm cracking up at my stupid intro that I do every single time. And as always, I have with me my co-host Andy Grabner. But I didn't say who I am, so you probably have no idea who I am. I'm Brian Wilson. And Andy, how are you doing today? It's the same thing every time. I know, I know. and Andy how are you doing today? It's the same thing every time. Gosh, anyhow.

I know, but I think it's the first time that I'm slightly tipsy and intoxicated that I'm doing this podcast but I already did a webinar before this so I think I'm good. Should you be announcing that you did that? Well, I should announce it. I'm straight.

You're Austrian, it's okay. It was a good day today was a good day to drink water and you in Austria, in Austria, beer flows from the Alps, right? Exactly. No, but I'm happy to be here.

And obviously I am sober enough to do all of this because I'm, I'm really excited to have this conversation because it's a conversation on a topic that we love, which is performance, performance engineering, automating performance. And today I'm allowed to use the word captain without getting like you saying, you said captain again. And you know what, this would be a great time to turn to do the captain drinking game with you saying, cause you've already got a headstart. So you'll be like on the floor by the end if we do that so far for our guests who are about to introduce that used to be andy was dropped the the captain word all the time like towards the end and

it would get predictable and i'd be like up here here it comes here it comes and i thought it might be a good drinking game but anyway i'm referring to our guests without having introduced them yet but usually andy let's see if you can um out of this here. That's a tough one, though. I think the guest today actually helped us to get Captain to where Captain is today. And not only Captain, but also the commercial offering we as Dynatrace provide to our customers as part of our Dynatrace Cloud Automation. Now, today is not about Captain. It's not about cloud automation even though we'll talk a lot about it but i really want to hear from mike who is our guest today how he started with performance why he is as excited about performance engineering as we are

and how he thinks or why he thinks that more people should participate in this great club of performance enthusiasts. So, Mike, now the word is over to you. Hello. Hey, that was a great intro. Thanks. So, yeah, my name is Mike Kobusch. I'm a performance engineer at the National Association of Insurance Commissioners currently. So, yeah, thanks for that great intro.

I appreciate it. Mike, I know we have done a couple of things together. We've done some YouTube videos. We have done, we have done, well, we're doing the podcast today. We're going to do a stage appearance together in Vegas in February. We are, and you better not make me dance the salsa i hear you make people salsa dance well the thing is right i'll just feed you a couple of beers or whatever you prefer and then uh a couple of beers i'll do i'll do just about anything you're right

forbidden dance yes so i'll try not to say captain too much today so we don't make any drink anymore we'll make notes one two three so yeah for those that don't know you can you give us a little background uh about yourself yeah so i'll tell you how i got started in performance engineering so i'm not like a traditional performance engineer so i have a degree in biology and a minor in chemistry so it background is probably not my strong suit, right? It wasn't my strong suit. I was a communication major. Yeah. So, you know, I figured if I could get out of college with a degree in biology and a minor in chemistry, I could pretty much do anything. And I mean, that's kind of my mindset, you know,

every day. So yeah, I got a job out of college, um, working in a mutual fund shop, just kind of doing reconciliation stuff. And then I moved into a QA role where I was doing manual testing. Um, and it took us three months to test a release. And I was thinking, man, this is so dumb. We could do things more efficiently. And I got into automation and I started automating things at this company. And I took a three month timeframe of testing down to three weeks, which was awesome. Wow. So I did automation for a couple of years and, you know, I just kind of started getting bored and I had a, um, um, a supervisor come up and say, Hey, would you like to get into performance engineering? I was like, yeah, that's great. I mean, I'd love to learn something new that would help my resume, you know.

So I got hooked up with a mentor from a consulting company that we were using, and he was in Hyderabad. His name is Vijay, one of the smartest people I know. And we would just sit and chat every morning about performance and, you know, kind of the basic concepts of performance. And from there, you know, we just started talking every day. I brought him over to the States for three or four months, and we just kind of sat with each other. And I just kind of sucked about, you know, looking at these servers and trying to make things faster and better for everybody. Just because that's kind of my nature, right? I mean, I fell out of college into this QA role where I like breaking stuff and fixing things.

And performance just seems like, I don't know how to say it correctly. Being a performance engineer is just, I think it's more complex. There's more moving parts, I think, than there is in other QA areas. And I kind of like that. So yeah, this DJ, he was amazing. And then, so I have another story. So there's somebody else that really got me into performance engineering and that's our very own Andy Grabner. He probably doesn't know that. So I first met Andy at a conference, STP con way back, I think in maybe 2011 or 2012. Right. So hopefully we have time for a quick story here. Yeah.

So I get on this plane. I'm in Kansas city. I get on the plane is probably 18 degrees land in Denver. It's snowing in Denver. Then I landed San Diego and it's just blue sky and 75 degrees. Right. And this is a Sunday afternoon and there's this seminar this andy guy's giving from like one to five i was like four hour seminar it's 75 degrees the hotel is right on the water all i all i want to do is just walk around right just enjoy being out of the winter months. So I walk into this musty, hot conference room, and here's Andy. And I sit down and I think, man, the very first break, I'm going to jet out of here and just walk

around. So Andy starts, he starts talking and about three minutes in, I have five pages of notes just written down. And I'm thinking, this guy is telling me everything that I've wanted to know, right? Man, he's answering so many questions in this first three minutes. And I think at the end of the four hours, I had probably 30 pages of notes, and I was just hooked from there. So, yeah, Andy and Vijay are kind of the two people that kind of got me going into performance engineering and so i appreciate it yeah and here i am today on a podcast with andy yeah did andy make any sound of music references during that presentation because that used to be a staple i asked him i said have you ever watched the sound of music and he's like i have never seen it

and i said you're from austria and you've never seen the sound of music and he's like i have never seen it and i said you're from austria and you've never seen the sound of music you didn't know austrians had sarcasm did you i don't think he really did did you have no back then back then and no really uh no no i have not it was you know if it was 10 years ago i saw it in boston at the boston opera house I saw it, I think, maybe five years ago. We watched it play. The correct answer, Andy, is I didn't see it. I lived it. I lived it.

He did live it, yeah. So yeah, I did see a reference on one of his performance clinics maybe a year or so after, and I was like, yeah, he watched it. So that was pretty. So I'm going to take credit for playing a role in him singing the sound of music. And not to harp on this stuff too much, but Andy, I loved when you used to do the difference between Austrians and Australians. That was also another favorite of mine. Yeah, yeah. So yeah, I was hooked from that day forward, man.

I mean, I had so many notes. And then he was talking about Dynatrace. And we were using some tools, CA Wiley, I think, which was really clunky. And I was thinking, if I could just get Dynatrace, man, my knowledge would just expand exponentially. So I moved to this new company, got Dynatrace in. It didn't take very long. And from there, man, we're off to the races. And it's been such an amazing experience.

So, I mean, first of all, thank you so much for kind of walking down a memory lane because I remember that conference as well. And for those people that may think, why would I ever spend a Sunday afternoon in an awesome location like San Diego, not going to the beach, but going to the conference room? Now you have a good reason. That's right. To stay in the conference room.

To meet Andy. No, not to meet me, but to meet people that, you know, you may have the chance to impact people in a positive way, you may have a chance to be impacted by people. And then years later, get on a podcast and then have the chance to inspire people to follow your lead. I think it changed my entire career path, actually. I mean, just not to be like mushy or crazy, right? But I mean, I really think it did. I mean, just the way you would, you spoke about performance engineering and your passion for it. And I was thinking, man, if I could just, if I knew a quarter of what he did, I would be, I would be amazing. So yeah. And then you were always so nice. You're always so nice. Like everywhere you go, you always make time for everybody and

talk to everybody and, and just, you know, just spill everything you know about performance engineering. So yeah, it's really great. Thank you. So, but now let's have the discussion, move it back to you. So in the last couple of years, right, especially now that you're doing, you're spending a lot of time in automating. And I remember a conversation I had with Nestor. I think we all know Nestor.

And Nestor, he made this nice quote. And whether it's his own or whether he kind of borrowed this from somewhere. But he basically said, I always try to automate myself out of my current role into the next. And this seems exactly what you've been doing. Because essentially, you could have stayed in manual testing. And maybe you thought, job security, that's great. I can always do the stuff and i i'm just here um but you said you you're not satisfied with status quo you want to

a improve things because it's boring to do the same thing manually all the time and you also want to challenge yourself and learn something new yeah and and i wonder i assume that most listeners right they also like to be challenged they also like to strive for something new for something for automation but in case there are some that are still resistant um is there anything you can tell them to maybe take away their fear that automation can potentially kill their job because i think this is a big fear of people i don't think i think automation is only going to enhance your job it makes you more desirable it shows that you have a drive to make change it shows that you have the will to to do something

new right so i think automation can only help enhance somebody's job versus, you know, we don't want to like, how do I want to say this? So we know now during COVID that the old norm wasn't working, right? So we kind of built a new norm around all these Zoom calls and WebExes and working from home, right? So, I mean, we don't want to stay in that old mindset of everything. We have to do everything manually because the more we can automate, and like I said before to you, Andy, the more time I have to go fishing, right? But it's not necessarily fishing. I mean, and it's the more time we get to spend with family, right? But I like to say it like this, the more we automate, the more we can start failing.

And I mean, what I mean by that is then we can start using new technologies, right? And we can start failing with those new technologies. And then over time, we're going to start gaining wins and more wins. And then we're going to break through to something new,

kind of like we have with, man, I can't, I don't want to say it, kept in in cloud automation right so i mean yeah you just have to you just have to move forward and automate as much as possible um yeah if i could automate myself man that would be awesome a lot of fish that you can catch yeah i could there's a lot of fish i could learn a lot more about fishing instead of performance engineering so yeah i mean i think you just have to make that leap and and i think the more vulnerable you are at times i think the more you can learn right so um and i'm in i put myself into vulnerable

situations all the time trying to move forward and some some things don't work and some things do work so yeah you just have to keep going you know it also goes to the the idea that if you don't start automating or even just continually learning and expanding on that knowledge base you'll be out of a job anyway. I think so. If you think about like QuickTest Pro, you remember like Windrunner, then there was QuickTest Pro on the functional automation side. And then Selenium was a new big thing. And yet a lot of people are like, oh, I'm not going to learn Selenium. I'm just going to stick with QuickTest Pro because that's what everyone uses.

And then almost overnight, but not quite overnight, QuickTest Pro just started disappearing. And it's like, if you're not learning and moving on, there might be a few places that still use these technologies or still want somebody to manually analyze the results and all the stuff but that's going to be fewer and far between so just even at the very least just to be able to keep working at a good place and not getting settling for some place that still does it the old way that i've been doing it that's that's enough right there obviously there's everything else you talked about, but just

on a baseline, it's. Yeah. Yeah. Don't, don't, don't ever want to settle. Right. So all these managers and supervisors, they always give you like a list of goals for the year. Right. I always hate those do, do better communication, do this better. Right. But I always set out, I have, I have one goal throughout the year. And my one goal is to learn something new every day. And if I do that, then I have accomplished something, right? If I've learned one thing new each day, then I'm better than I was yesterday. I don't always learn something new.

Some days are a fail, right? But probably 90% of time i'm i'm out there learning at least one thing in the not just not performance engineering or or career-wise but maybe it's you know i coach two soccer teams so maybe it's learning something new about soccer or cooking or something right but just as long as i learn one new thing a day i think i've accomplished something and that's kind of my mindset going into things right so i'm always trying to look for the um latest and greatest technologies and just even with dynatrace i mean there's dynatrace is so expansive man you can easily learn one new thing a day just digging through Donna trees. At least every two weeks.

But, you know, to wrap up on that point, I wanted to see, you know, the idea of, because I embrace that idea of like trying to learn something new every day. And it's not that you even have to try to, right? You just have to be open to it. Like if you have that mindset of I'm going to be open to observing and noticing something that I didn't know before or noticed before, that's learning something. And that just opens you up to so much more information and so many more thoughts. And think about how many times, you know, I'm just going to put words

in your mouth because you never said this, but think about how many times maybe you learned something about on the soccer field that you're like, hey, that applies to other things as well. Right. The way I got those kids to do that together, oh, I see what I did there. That could be used somewhere else, right? So a lot of times these small little things teach us something that can be applied elsewhere, which is the excellent benefit of that approach. Yeah, you're right. And on the soccer field, I tell my kids, I mean, don't just practice hard on the soccer field,

but do it in school. Do it if you're cleaning your room, if you're reading a book. Practice hard on the soccer field, but do it in school. Do it if you're cleaning your room, you know, doing if you're reading a book, you know, practice hard doing everything. I tell my kids, be the best you that you can be every day. And I probably tell them that almost every day when I drop them off at school, be the best you that you can be. And if you're the best you that you can be, then. Hey, you've you've passed, right? If you're given 50% a day, then you're failing just yourself. So be the best you that you can be.

Hey, Mike, I need to pick a brain now because coming Monday, Monday is 15th of November, which reveals now it is the day of the recording. It's November 11th, which actually is the beginning of Carnival here in Austria and in, I think, Central Europe. 11th, 11 at 11, 11. So at 11, 11 on the 11th of November, we start the Carnival. Now, there's not a whole lot to celebrate these days, unfortunately, with COVID rising again. But I want to ask you something. Monday, I have a meetup presentation in Vienna. And it's at the Java meetup group. And I'm supposed to talk about top performance problems, challenges, and distributed architectures. And I do have my couple of use cases that I always present.

But I wonder if you have something, I wonder if you have maybe an additional use case where you say, Hey, Andy, I remember you at SDPCon, you talked about this use case that triggered kind of my excitement. Here's something I recently found at NAIC or here's some use cases and here are the metrics I look at, because you know I like to talk about metrics and how to detect patterns. So maybe you can just tell me a story. Man, you are trying metrics I look at. Because you know I like to talk about metrics and how to detect patterns. Yeah.

So maybe you can just tell me a story. Man, you are trying to pick my brain. So I'm going to tell you something that you may be hinting at, but let's talk about Captain and Cloud Automation for a little bit. A use case there. So after we run performance scripts, you know, there's thousands and thousands of lines of data and pure paths and charts and graphs and, you know, analyzing data

takes us hours for a test run, right? If everything goes well, maybe 40 minutes, 30 minutes. If there's some issue, right, maybe two, three hours. And that takes a lot of time. And this is where we've kind of talked about automation. So we've kind of got into Keptin and cloud automation. And we've taken that data analysis time down from two, three hours to probably 30 seconds. So I think that's kind of a use case that I'm working on right now. Building dashboards, making sure I have all my KPIs that I want to report on. Stuff like memory and database usage and network IO.

So we're moving to the cloud from on-prem, another use case kind of. So we've noticed that in that transition, we're spending a lot more time in network, of course, just because there's um gosh the just the distance right i mean there's a lot of different distance between us and then our availability zones and where our database lives so we've been noticing a lot of network iowa time we're contributing that to a lot of database time right so we're opening up pipes making the pipeline bigger so we can transfer more data. We're kind of tweaking data sources and stuff like that.

And all of this is made possible by Donna Trace because we can see all of that. We can see the network IOTOM. We can see what our data source looks like and how it's being utilized. And we can see into our JVMs and our memory and all of that. So yeah, it's all made possible by Dynatrace. So I think that's a great use case. So what I was hinting at, I mean, thanks for the segue right away into the automation piece. But I was really looking for a use case.

And I think you perfectly explained the use case. People are migrating from on-premise to the cloud, which means, and I correct if I'm wrong, but what you're explaining is you're moving pieces into the cloud, which means you have communication going on between what is still on-premise and what's in the cloud. Obviously, you mentioned the distance. For me, this is latency. Latency kicks in, therefore has an impact on response time. But in the end, I assume also cost because you have to pay for ingress and egress.

And therefore, optimizing not only for performance, but also optimizing for costs, I think becomes a big deal. Yeah, I think you're right. So I'm not like on the cost side. So none of that really matters to me. You know, I mean, it matters to me, like, so we want to keep our costs down. We don't want to be, you know, we don't want to go crazy, but I mean, I'm not really looking at the cost. All I'm looking at is can I make this faster and better? And then the powers that be will tell me hey this is not cost effective or is cost effective right so so in your in your cloud migration scenario did you like take one example of your apps what do you what did you move to the cloud and what stayed on premise

which components did you move the database to the cloud or the database yeah we moved we moved the database to the cloud um so yeah so our first move was we moved our database to the cloud or the database yeah we moved we moved the database to the cloud um so yeah so our first move was we moved our database to the cloud left our um application on-prem and that was just that was kind of a nightmare things were so totally slow doing that um and and with with um dynatrace i mean you can just see your n plus one queries and how long they're taking right so now we've started to move all of our on-prem applications to the cloud as well um so now we have we're going to be um let's just say 90 cloud yeah so it's been um quite it's been quite the last seven or eight months, man. It's just a lot of testing, a lot of digging through data and looking at Dynatrace and seeing what has worked, what hasn't worked.

We're really tuning our applications now. We're in the homestretch, so we're really tuning our JVMs to run as fast as possible. And yeah, just looking at data source and memory utilization and network and latency. It's, yeah, it's been fun though. I mean, like it's been a learning experience for me for sure. So that's something that I'll take out of this transformation that we're doing. Have you seen any change in behavior have you seen any new metrics that you that you start to look at as you you moved kind of 90 to the cloud or is it still the same um it's it's basically the same in the end i think the one new metric that i've been

looking most at is tweaking the data source, you know, so just kind of figuring out how our connections and just kind of, you know, do we want, how many connections do we want active at all times? What is our max connections? I think that was a huge player in moving to the cloud. I don't know why that was. It wasn't as big of a deal on Prim, I don't think. But in the cloud... Are you talking about connection pool sizes from the app to the... I mean, I can probably

make just a guess, right? I mean, I'm not sure if I'm right here. But if you have a certain connection pool size, let's say 30, and an application takes a connection out of the pool, and then does its n plus 1 queries, and then once it's done, it pulls it back. Now, if you're moving to the cloud, just the database, and latency kicks in, it means the application holds on to one connection out of the pool for much longer than before.

Therefore, the connection itself becomes a scarce resource. Right. And this is why I think you probably had to increase connection pool sizes. But increasing connection pools mean more connection parallel, meaning more pressure in parallel on the database, even though it's kind of stretched out a little bit. And more memory too, right? More memory too then right memory too right i mean this yeah so i think that's yeah i think you're you're right on it i mean being the data data source the connection pool size it takes a little bit of toll on the memory um so yeah i mean i'm seeing a little bit more time we're spending a little bit more time in suspension um just in the young

gen right which is i mean yeah that's it's okay ish but i think that had an adverse effect on that um which may be hindering our response time so yeah i mean yeah you're right i mean one one change is going to affect something else somewhere else so you've always got to keep your eye out on what's going on. And that's one of the great things about, here we go again, kept in the cloud automation, because if I build a dashboard and I want to look at memory utilization and GC and database connections and network time, I can do all that in 30 seconds.

And then I can kick off another script, right? Make a change, kick off a script, analyze that that in 30 seconds and then I can kick off another script, right? Make a change, kick off a script, analyze that data in 30 seconds and then keep moving, right? So that's one of the great things about where we're moving with cloud automation. Gives you two hours more for phishing. Yep.

Or even if you think about, you know, that discovery you had to do with that database by automating some of your other processes that freed up your time to be able to do that research to be able to look into that more without suddenly being like okay well we're gonna we're gonna have to delay our analysis our three-hour analysis you're gonna have to be delayed by four hours because I have to do all this research it's like that analysis was automated so you can spend that time and not have it be a detriment to

anybody or bottleneck anything yeah you're right on target with that i mean we can do more um faster now so yeah i mean i think you hit the mark with that and data analysis times are going to come down um testing times are going to go up we're going to be able to do more tests run more tests catch more things right um and less of data analysis which is good which is a really good thing hey there any, coming back to my presentation I have on Monday, because I wonder, as I have you here, right? I'm using you, by the way. I'm using you, right? Don't use me.

Don't use me. You mentioned the M plus one query problem. Have you seen any other, or do you remember any use case, any success story where you say, hey, this is what you have found out in the data you look at, and this is what the developers then did to fix the problem? Was there anything that was kind of stuck to your memory? Yeah, and most of it, just because we're just an Oracle shop here, most of it is in plus one query problems.

When we first launched on a trace in our applications uh it took probably less than 10 minutes and we we saw um hundreds of n plus one queries and we're like oh my gosh this is where all of our time is being spent right and we the developers were able to quickly get on that um fix those. And we have an application right now that a couple of years ago, I told this team that, hey, we need to fix all our N plus one queries before we move to the cloud, because once we move, those times are just going to get exaggerated. So now we've pretty much made that move to the cloud.

And in the last hour, they're fixing all their N plus one query issues because those response times you just exaggerated now. So that's pretty much what our focus is for a lot of these apps in plus one query issues. Not having the databases tuned right, not the databases, the JVMs, just JVM tuning is a big thing um i think on the on-prem apps before i was here they would just use default settings right and nobody was wanting to make

a change um and then so i come in and start testing all this and say okay let's let's change the garbage collection to g1 and see what happens, right? Let's increase memory a little bit and see what happens. And we can make those incremental changes, sort of those through cloud automation, see our scores creep up and keep making little changes. Yeah. So this reminds me, I know we already had a discussion about this via email. Our friends from Akamas, they have this product

that is automatically making changes to all of the different JVM parameters. And it's basically doing goal-based, goal-based modifications. So you can say your goal is to bring GC down or your goal is to bring response time down. And then they are tweaking all the JVM settings in a couple of experiments and then they are tweaking all the JVM settings in a couple of

experiments and then come up with the ideal setting. So maybe time to look into this as well. Yeah, I think you're right. I think once I get cloud automation down and we get it where we want it, right, I think there's still more changes we need to make to it. I think there's room for improvement. Not that it's great, that it's not great,. I think there's room for improvement. Not that it's great, that it's not great, but I think there's some room for improvement. I mean, there's some things that I see that I'm like, oh man, if I just had this, or if I just had that, it would be so much better. But yeah, I think after the first of the year, once we moved to the cloud, I think that's something to really look into. And you talk about automating yourself off a job, man, that could do it, right?

I mean, what are there like 700 JVM settings or something? Andy, I forget the number that they've said. Like who knows what even half of them are like? Yeah, I know probably three. Yeah, I probably know three. And even with GC, there's probably all these other little tweaks that you could try in conjunction with the ones that you're doing. And it's just, I still haven't seen that in actual practice, obviously because I'm working on like a pre-sales engineering side,

but that whole, that whole iteration, it reminds me of some of the stuff we have in, you know, for audio. Like I have like a mastering plugin that goes through and you, you mix the song and at the end you say, master it right now. Mastering is usually done by like a professional but all his home recorders can't afford you know five hundred dollars per song or something like that so you put it through the scene has an ai algorithm that just goes and tweaks and tweaks and tries all these different things and then tries to map it to get it to sound

like what it what it should do then at the end it says how's this and it's you can say i want it loud it's the same exact kind of situation like hey, hey, I want less GC going. I don't want this. And I'm fascinated with how these tools can start to adjust, measure, adjust, measure, and tweak, and then say, here, here's what we found. It's pretty awesome. You have totally piqued my interest now. Now I'm going to have to dig further into this and start learning and seeing if this is a direction that I want to go. You know, when I first saw your email, I was like, oh, okay, this is kind of cool, right? And then, hey, I got to get back on my cloud transformation testing

and looking at all this stuff. And then I got to get back on cloud automation and make sure this is working right. I'll come back to that later. But now you've piqued my interest. And now I think that's something that, yeah, I want to dig into. Maybe some good homework for you or maybe a goal. Set a goal for Perform. Maybe at our session at Perform.

That's right. So everybody listening now, what's their website? Alchemist.com probably. I think it's at I-O. Oh, at I-O. Yeah, so everybody now get on that and check it out, and then you can teach me how to do it. But I mean, coming back to the problem patterns, I think it's great that you just confirmed

that the N plus one query problem is just still out there. Yeah, that's the biggest. It's causing massive pain, yeah. And I think that any company a performance engineer goes to is probably going to face that just because, and especially if you don't have good APM tools,

you're really not going to see it, right? So before Dynatrace, man, these developers are going to a war room and try to figure stuff out. Now, post-Dynatrace, there's never been a war room, right? We figure things out instantly, most of the time instantly. Some things are a little harder than others. But the N plus one query issue, I mean, that's easy to see, right?

Maybe not easy for developers to fix all the time, but easy to see. And yeah, I think so. So I think some of our issues have been around how we write queries as well. And kind of Dynatrace sniffs that out. And that's one of the things we've learned moving to the cloud is we have more efficient query processes. You're going to have more efficient response times. I mean, that's just reality. And I think probably everybody knows that.

That's kind of just remedial, right? But maybe not. Maybe everybody doesn't know that. And I think the more efficient your queries are, the better response time you're going to have just across the board. Well, you know it, but if you're not observing it, then you're not paying attention to it. You're like, oh, yeah. And then once you're looking at it, you're like, oh, wait.

Yeah, I think if you have like five or six second response times on a page, you're like, oh, okay. But then when you really dig into it and look at it, you're like, oh, man, I could make this one second, Right. And then that's what you go do. I mean, that's, that's as a performance engineer, that's what you strive to do is take something from six seconds to one second or two seconds. Right. Because nobody wants to wait six seconds for a page to load. I, I don't, I hate that. It just, right. That's just what you say there's one person out there all of this all of this is brought on by the smartphone right i've always said that the

smartphone this kind of just enhanced this performance engineering industry because now we want all of our stuff real time we want it instantly and that's what the smartphone has done to us as a society right we want everything right now and so now performance engineering has to keep up with that right you want your data right now you want your information right now if i'm gonna if i'm buying something and there's two in stock i don't want to see a little spinner wheel on there when somebody else is trying to you know get the same thing that i want that's that is maddening right so yeah i like that that's a i just took it down as a note so basically performance engineering and now has the challenge of what uh the smartphone triggered in us that we want to have instant gratification instant

feedback because we have software at our fingertips all the time. We often don't even, I think, realize that behind the scenes there is potentially calls going from one cloud to another to an on-premise data center somewhere, and the end user doesn't care how many components are involved. No, the end user has no idea, right? The end user has no idea what's going on behind the scenes. All they know is they have a front-end application. They click on submit or check out, and they want that right now, right?

So I was booking a flight to perform the other day, and the airline website was a little slow. There was one ticket left at that certain price, and I was like, oh, my gosh. Am I going to get this, or is somebody else trying to get it right? Why is this so slow? Come on, come on website. Right. But it's really not just the website. It's all the servers and working in harmony. You should even have mainframe involved in there. Oh my gosh, don't do that. Don't do that. Yeah.

So yeah, the smartphone has made us just like an instant society right um so it's interesting you bring it as a smartphone because for a long time people have talked about like the google effect right because google gives fast results but what you're you're by changing at the smartphone you're bringing it into the real world it's not just about how fast is my browser it's how fast is that product getting to my door because you even see it like if you look at you know the amazons and all they now have you know besides the prime shipping whatever now there's like oh next day prime and amazon direct where you can get it in two hours if you and you're just like oh yeah yeah this instant gratification is getting but it's all from the cell phone that's not the google effect

that's the cell phone effect where i can get everything i need right now and i like to perform fast which is the google on top of it. Especially this time of the year, you know, I have Christmas gifts being sent to the house, right? So I'm tracking, I'm tracking the truck and telling my kids, Hey, go outside and kick the soccer ball. Right. Oh yeah. Yeah. So you can show the truck on the map now too.

That's really weird because now I can call the truck and take everything from them. I had my cell phone in my pocket a couple of months ago and I pulled it out and it was just burning hot. And I was like, what is going on? And then I turned it on and says, oh, hey, you did a factory reset. So I lost everything. And therefore, like three hours, I think my entire life was lost.

And I was just walking around like, what in the world am I going to do? Right. And then about three hours after that, I kind of just stopped and thought, this thing is rules our life. Right. I mean, these smartphones are ruling our life and if they're not performing right, or if something goes wrong, we're lost. And that's, that's crazy. But I mean, that's that's crazy but i mean that's kind of how it is that's we've depended on this smartphone for instant results and yeah as the performance engineering industry we have to keep up with those instant results and and yeah and i like where we're going with it like with captain and cloud automation getting instant results from your data analysis. That's a great step forward.

And hopefully we can keep improving and doing better and making things faster, right? So releasing, so CICD pipeline, right? I think cloud automation is really going to play a big part in that. By being able to release the different stages without having to really do anything except kick off cloud automation evaluations and scripts and let it do the rest, right? So one of these days, yeah, I mean, I think we're just going to hit a button, sit back and tip a beer,

and then our code goes to prod, right? But somebody still has to click that button. So the more you automate, like we've talked about earlier, I think the more potential you're going to be desirable in the job market. So get out there and automate as much as possible. That's a nice, I think also closure, almost a closure statement to this. That is, I think that is great. That's what I was segwaying into that.

So we can kind of close this out. Exactly. Did I say that out loud? Yeah. Oh, perfect. We can edit that out later. No, no, no. That's perfect.

We don't edit anything out. This is all kind of live and unscripted and unedited. So Mike, I'm really looking forward to having you on stage in Vegas. Yeah, thanks. I think it's going to be great. Yeah, so you're going to get me out of my comfort zone, but I hope everybody comes out and kind of, you know, sees what we have to offer.

And if you see us walking around, especially me, come chat me up. You know, I'm always willing to talk about performance engineering or cloud automation. So there's certain people there that I'm kind of intimidated by, like that Mark, what's his last name? Tomlinson. Tomlinson? Yeah, I think he's a really smart individual, and if I go up and talk to him, I'm going to

sound so dumb compared to what he has to say. So I'm like, I'll just let him be, but come chat me up. Mark is fantastic. Mark, the way you mentioned you met Andy, and who was your the other... Vijay. Yeah, Vijay.

For both Andy and I, we had a similar experience with Mark early on in our performance careers. So he's always, he's always friendly and amicable. Like, yes, he knows a ton. He'll be talking about like,

yeah, improved performance, but you know, make sure you're down to the level of, do you have the same processor in the machine? Like thinking about the hardware is the hardware switches that, you know, cause he's been just doing this stuff forever and has seen everything under the

sun, but he's always like looking to help people and and help level people up so definitely don't avoid him if you have i'm always i'm always looking to learn um yeah i'm always just open to learning and trying to learn just one new thing a day i do have to say though when you guys go to vegas make sure you take some time and go to Omega Mart, which is the Meow Wolf installation out there. If you know what that is, any listeners going to Vegas, Meow Wolf is this fantastic art collective that started out in Santa Fe. And they did this huge installation where the whole theme of it was this family that discovered some interdimensional thing. But then there's a bunch of artists who designed all these different rooms and stuff.

You open the washing machine and you go through there into some other world and junk. They created two other ones. Actually, George R. R. Martin bought the building for them to do that one in Santa Fe. The Game of Thrones author, right? Then they did a second one called Omega Mart, which is based on

some sort of weird supermarket with all these weird products, but then you open up the freezer door and you go into the whole back interdimensional area, and now they have a new one out here in Denver, which is the interdimensional transport hub. So it's really, really awesome, really trippy, really artsy, but definitely when you're out there, make time to go. I get nothing from saying this. I'm just a huge proponent of them because it's just like this, you know, it's kind of like, you know, Disney World for people who like weird, freaky things,

you know. But it's just all artsy and, you know, fantastic stuff. But anyway, nothing to do with anything. But since you're mentioning Vegas and Perform, you know, to me, I'm like, I got to get out to Omega Mart sometime and I know I'm not going to perform this year, so I'll have to wait until next year.

But make the time to go. Unrelated. Sorry about that. That's my commercial for them. All right. Hey, with this, Mike, thank you so much for being on the show. Thanks for giving me some new ideas also for my presentation on Monday. Thanks for having me.

I will follow up with you soon anyway regarding performing our presentation because we need to work on the presentation itself but that's my action item. And the royalties that you're going to give for what you're using on Monday, right? Exactly. Yeah, send a carton of beer from Austria. Exactly. But folks

remember, it always pays off to spend the extra time on a Sunday afternoon afternoon yeah even in sunday san diego because you never know where you get out of it that's right that's our moral of our story here exactly it took us how long 50 minutes to um get to the moral but that's okay yeah that's that's the story the morals at the end it was the beginning everything like oh click don't ever skip out on somebody because they may just change your career.

Always be welcome to talk to other people in your line of business too because everyone's always, we're all in this alone. A lot of us performance people, at least in the older days, I'm not sure what it's like today, but a lot of us were alone doing it. There's not a lot of us out there. We were the person at the organization

who was responsible, so we kind of had to figure it out so the more you talk to other people the more tips and tricks and just things you learn you also just get that community right and that's that's key as well you're exactly right 100 you're exactly right yeah well thanks for having me this is perfect it's been awesome thank you so much andy any words, Andy, or should I wrap it up? No, just stay healthy out there. And that's it. Looking forward to meeting more of you in person once this is all over the stuff that is keeping us at home. Yeah. All right. Well, thank you everyone for listening. If you have any questions or comments, which nobody ever does,

you can tweet us at pure underscore DT or send an old-fashioned email to pureperformance at dynatrace.com. And I do want to mention that every time I hear the word cloud automation, I think we're saying Klaus Automation, which, Andy, I hope you get back to him because that's just amazing. I should actually get a T-shirt that says Klaus Automation. Get one of those and wear it to perform, Andy. Anyhow, I hope you all have a great time. Thank you for listening.

And thank you, everybody. And thank you, Mike, I hope you all have a great time. Thank you for listening. And thank you. Yeah. Thank you everybody. And thank you, uh, uh, Mike for, for joining us today. Appreciate it. Thank you. Bye-bye.