PurePerformance - From Performance Testing to Designing for Performance with Ernst Ambichl

Episode Date: April 22, 2024

Use Things you Understand! Learn the fundamentals to understand the layers of abstraction! And remember that we don't live in a world with unlimited resources!These are advice from our recent conversation with Ernst Ambichl, Chief Product Architect at Dynatrace, who has started his performance career in the late 80s building the first load testing tools for databases which later became one of the most successful performance engineering tools in the market.Tune in and learn about how Ernst has evolved from being a performance engineer to become an advocate for "Designing and Architecting for Performance". Ernst explains how important good upfront analysis of performance requirements and characteristics of the underlying infrastructure is, how to define baselines and constantly evaluate your changes against your goals.On a personal note: I want to say THANK YOU Ernst for being one of my personal mentors over the past 20+ years. You inspired me with your passion about performance and building resilient systems

It's time for Pure Performance! Get your stopwatches ready, it's time for Pure Performance with Andy Grabner and Brian Wilson. Hello Andy and welcome to another episode of Pure Performance. My name is Brian Wilson and as always I have Andy who I just already said. I kind of messed that up but we'll go with it. Andy, how are you doing today? I'm actually really good and it's a first. Hopefully the audio works well because I'm in a new setup as people might be able to tell by looking at our little thumbnail. I'm in a new apartment.

There's no longer my backdrop with my old kitchen. So, new era in my life. New era for old eras. You know, I had a little bit of a weird dream last night. It was 2010, and I was still working at WebMD, and I was doing some load testing with a different, with a tool, I won't mention his name. But I was just getting really frustrated because, you know, the troubles of old school, all in one load testing where you had to boil the ocean and test the entire system for every release and project management breathing down your neck.

I remember in my case, it was maybe they'd give us a new project, and I'd scope it out as two months of work, and they'd be like, oh, well, great, because we signed a contract with advertisers, and this whole thing from coding to testing is due in a month, and the developers need two months.

I'm like, oh, this is going to be fun. But then I joined Dynatrace, and I've been happy since. But yeah i was just having some some some old dreams of of performance testing and uh now we're here it's like going down memory lane huh yeah it sure is andy yeah it's almost like a perfect uh a perfect uh guest then actually for going down memory lane at least on my side i've been in performance engineering if you want to call it like that for since 2001 which was the time when i joined a company called segway software back then and one of the first people i interacted with is actually our guest today ernst anbichl servus ernst thank you so much for being here hi guys i'm happy to be here yeah um ernst i, I gotta tell you,

you have, and probably you don't even know it, but you've impacted my career in many, many ways because you taught me in the very early days when I started at Segway, I actually started as a tester on SIL Performer. I was testing SIL Performer with SIL Performer, which I thought was actually, I think, the best thing that happened to me because you get a completely different appreciation as an engineer about quality. And I remember then after I think it was six months in QA or in testing, I was then going into engineering and then later also helped our customers to actually use SIL Performer to run load tests. And you helped me a lot with insights on how to actually analyze, like what are the different

metrics? Why are certain queue lengths important? And back then we were testing mainly on Windows, I believe. That was the main platform. But Ernst, I would like to hear from you a little bit because going down memory lane, you have been doing performance testing and performance engineering. You started the whole thing even before I joined Segway. Can you give us a little bit of background when your journey started towards performance engineering? What triggered it? Sure. I mean, it started basically

on university. So there was an assistant professor at that point in time. It was the late 80s, so you may guess my age. And we had a project running with Siemens at that point in time where we were testing their local area network with the upcoming SQL databases that were around at that point in time. And for that, I developed basically the first testing tool that was able to simulate real user load, real user transactions again against a SQL database. And so with that idea, I founded my own company and made a tool out of it, which has then become Silk Performer. Started as a database load testing tool, but shortly evolved into HTTP and web load testing because I mean that in the, I would say the late 90s, the web took over everything, I mean from the client server area. So we evolved it basically into web load testing and all the protocols coming at that point

in time, like Cobra, if you can remember, IOP, things like that. So I always was basically very involved into performance and how to make systems be more performing, better scalable, into performance and how to make systems be more performing, better scalable, using the resources that you have better than you might expect. I mean, you can think about when we started

with load testing, I mean, sometimes what we call five virtual users broke down the application. So in these times, now we are talking about hundreds of thousands of users that we still need to handle. So pre-devolvement in that area and also the requirements regarding performance and scalability. You know, I just got to jump in and say that. Earlier, you said you came from Silk Performer. I didn't realize you created it, so I bow down to you. I had no idea that that was the case,

so that's absolutely amazing. I'm honored to be here with you even more now. Yes, the first small part. I mean, it was a teamwork in the end. Right, you know. Right, right, as always. And I can also relate to what you said earlier that in the early days, five virtual users were killing applications.

I even remember a scenario where I always said that the poor man's load testing tool, my index finger hitting the f5 key on my keyboard to refresh the browser sometimes even killed applications right and and then five virtual users now it's you know if you if you think back in the late 80s you started with with a sql performance then you mentioned all the different protocols the web took off um what what made you then actually jump over to the other side with the other side i mean because with with uh silk performer we were driving a lot of load we were testing applications we already had the ways to analyze things by capturing metrics what made you you then go over to the observability

in the like Dynatrace? Yeah, I think a good question. I think the world changed a little in terms of complexity. I mean, you know, we had pretty simple systems to test at the beginning. I mean, with the client server world, you tested the client against the server.

Three tiers application came, still pretty simple. But then with the web evolving and new technologies evolving, you ended up in a very complex environment that you need to test. And very often the challenge was not in testing it was in analyzing the problems in in finding root causes in understanding how the the systems interact with each other and this was pretty much more challenging from an architectural or from a developer perspective to build such a thing and i think it also also is very valuable. And if you don't do it, I mean, what to do with driving load and not understanding where the problems came from. systems were then used by thousands, tens of thousands, millions of people. The demand on just being able to find problems, to detect where the problems are, got more

interesting. Mhm. Mhm. Now Ernst, now you can hopefully confirm something, but if you are disagreeing or have some other thoughts on this, Brian and I have done this podcast for how long now? Seven or eight years? Something like that, quite a long time. I think 2015 might have been when we started with talking about top performance problems in Java applications,.NET applications, because that was obviously the application, the technology stakes that we analyzed. And I remember in the early days, we always talked about the same patterns. We always talked about what are the performance problem patterns that actually bring down applications.

And still to this day, I still refer to the N plus one query problem which is capturing too much data, too many round trips, inefficient ways of how connected systems talk with each other is one of the major reasons why systems don't scale is this something that you can

confirm that you see as well do you see no there's other things that you have seen in your life that actually are kind of the number one use case or root cause of problems? I certainly can confirm that this is one of the major problems with applications. Not thinking about how much data I need, when to query the data, how often to query the data. And I think this has not changed at all. I'm currently really within our own organization, I see the same patterns still being applied,

not with SQL, but with DQL and Grail. It's a continuous struggle or fight against, I would say, doing the things in the wrong way. And even with modern applications, think about, I mean, we're all speaking about reactive applications. Many things get overdone. You think you have a reactive application that may interact or may respond to you by just doing simple clicks or mouse overs. But if you don't acknowledge that if you do a query on a backend to serve such reactive behavior, you make your application even less reactive than with different patterns. I think you don't do your customers a favor. So, even nowadays, I think many of the same challenges are there because we try to do so many things on an application screen, updating things concurrently, doing things in a way we would not have done it in the days where you were more restricted or limited to the resources the database can provide. Currently, many people think the world is unlimited in terms of there's

unlimited performance. Everything scales. You need more machines. You need bigger machines. You can scale. But that's not really the truth. There's always an end on performance. There's always the need to optimize things. Also think about the costs that it brings with it, doing it in this unreflected way.

It costs a lot of resources in the end. A lot of machines running cycles they need not to do. And with all the involvement in computer technology, in speed of networks, in speed of I.O. that you can provide, it's still limited. You can't do endless things. And algorithms, the order of an algorithm stays the same. That means if you have an algorithm with an order of n, it will deliver if n is very, very, very high, bad numbers, or even if n square, you can think about it. So it's still important.

And there's no limitlessness in technology. Some might think about. I need to, you may notice that I'm always taking notes because Brian and I, we always say we learn so much with these podcasts because we always have guests like you that share their experience. And then there's no unlimitedness in technology. And I think that is great because we had this wild, wild west thinking when the cloud came up. Now we can do, at the instant, we can get as many resources as we want. We can scale up, we can get as many resources as we want.

We can scale up, we can scale down. So who cares? But we have to care, not only for costs. Also, there's a lot of talk, obviously, about our responsibility around carbon impact, because everything costs something, but it costs twice the dollars or the euros, but then also the carbon impact. Ernst, but if I think back about this so what you're basically saying we need to think about performance actually from the design phase of an application because you brought up a good point if you have an application with this with a reactive

screen with many different widgets and many different controls and charts and everything you probably need to think about what actually needs to be updated, at which pace is it really necessary, and what's kind of like the minimum viable feature set from a data freshness perspective. How fresh does the data need to be before you even start coding? Because otherwise, as you said, probably developers go off and just constantly refresh and that would then cause big problems.

Yes, exactly. I think the thinking you need to apply on the front ends, what can help you there is that you always not only test but really develop against real data or data sets that are large enough. This is a very old rule, but it's always broken.

We are basically developing our code against happy path data. That means data that you would expect in limited scale, but not against data that might come in, but is not what you expect. So, for instance, our advice here at Dynatrace always is

you test your applications before we go out with a live dataset against a live dataset. Against the live dataset, which is large enough. Because then you immediately see, even without load testing, you will see the results. If your queries are too slow, if you're doing too much data transfer, you will see it when the datasets are too, when they are large. You know, the one thing you said earlier, which struck me as, I guess, like a critical mass type of situation, is, you know, first we had the,

as we sometimes like to say, the one plus N query problem, but somehow it got named N plus one, onto the database. Then when we went to microservices, we started seeing that on the service calls back and forth. But then you mentioned Dequel, right? Many tools, not just Dynatrace,

but many, many tools now have a user interface where the users who are not developers can write queries. We know that with the N plus one problem still being prevalent, that developers, although they should be aware of it, just are not. It's not something that's ingrained with them at their development birth.

So that's still going on. Now we're going to take end users who are potentially not developers or don't know about these things and giving them the ability to write N plus one queries in whatever query language these tools have to offer. So it's going to just the regular consumer, and I don't mean that in a negative way, but just someone who might not even be thinking of performance. And it feels like

tools need to put in guardrails to do pre-analysis of those queries that the customers can write. Because just as you're saying, you can just have customers writing bad queries against the data they want to analyze and potentially take their tooling or whatever system down in that fashion. And if we know developers aren't on top of this, we can be even more sure that just an end user writing a query is not even going to be thinking about that at all, which is just a fascinating new development in performance risk.

Yeah, you can have someone refreshing. We talked a lot of times when we talk about mainframe and banking apps, you can keep refreshing your balance, which is going to hit the mainframe all the time. But most people don't do that. But we're just opening this up to regular users. Exactly. I think that's one of the main problems

we currently have within our own development, within Grail, what we offer there with DQL, with the abilities to use that inside automation. So many of the applications are now not user-driven anymore. They are driven through events, automation events, regular things from an audience, as you said, that is not really caring about the response times,

as long as it's good. I mean, one effect with these high-scalable systems we have is also that even queries that take a lot of resources can be answered below seconds. But they take about 30 or 100

machines within that second. So, the end user might be happy with their response times, but the system gets completely overloaded when that pattern follows on. So one of the things we need to care about is how we share resources in such systems,

how we protect resources, how we assure that processes that are more important than others are getting more priority in these systems. So there are many questions that arise through that openness of the systems that we now have that create completely new use cases we even have not thought about. I mean, and now thinking of opening up this data to so many new users and with large language models and generative AI, it's getting even easier to just phrase a question

in human language and then who knows what's happening. But as long as the user gets a response within an acceptable amount of time, I guess the user doesn't care what's behind the scenes and he will never get educated. Exactly, yeah. So it's weird. But the AI has to be trained

to write good queries now. Of course. No, but that's the crazy part. Now you've got to think about how do we train the AI to write a proper query? Which, you know, that's doable at least.

I think that's a slight advantage because you can train a machine to follow rule patterns that people might not always follow or think about but it still has to be done right if you're gonna have it being created it's got to be done efficiently not just for speed and and let's get it out the door it can't be a dump learning but just just you know learning on all the executed decrypt statements. You need to just learn them on really good ones.

But then, yes, I agree, it can be an advantage there, so that at least the novice users don't run into the novice problems that they would produce. On the other side, I mean, this is all about query

optimization and how to detect it and being more smart on finding out an execution plan that's appropriate. Ernst, I wanted to shift to a different thing because I remember when we two talked about the podcast and I said, hey, Ernst, let's talk about performance engineering and how you do performance testing. And then you said, well, you know,'s talk about performance engineering and how you do performance testing. And then you said, well, you know, I've been in the business for a couple of years longer than you and maybe you have not done the actual load testing

in the recent years, but you're focused on other things like enabling an ever-growing engineering organization to think about performance early on. Yes. And I'm pretty sure that many of our listeners that have been following us for the last nine years also have kind of matured in performance testing nine years ago,

load testing, maybe self-performing, maybe load running, maybe other tools, then becoming efficient with observability tools to analyze and now maybe are in a similar role like you, Ernst, where you're leading a large team or you're responsible that an engineering organization of many people is producing high quality, high performing code. Can you tell us a little bit about the challenges that you have on a day-to-day basis and how

you solve them? Maybe some tips, some things that you have seen, how you can enable developers, especially that come in new. What do we do internally and what can people learn from us? Okay. I mean, a very easy thing to learn on is using Dynatrace profiling capabilities to analyze existing applications.

So with, and for instance, speaking about Java, which is the main stack we are using, you usually run into memory problems before you run into CPU problems. You run into garbage collection problems. basically allocated, how much data is basically going into the old space, I think you can at least find problems pretty easily. And that's what we are doing. I mean, we are usually looking on GC times of the processes we have.

And if we detect there's a degradation on that, we're looking into more details, seeing what's going on there and we find these hotspots. Very often these hotspots are caused by new APIs that the developers want to use. By far it's not new, but the introduction of stream APIs within Java just broke performance in many reasons. Because the implementation was done in a way that memory allocation was just done excessively compared to using basically the old construct. So that's one way you can do. You can look into the code and give advice on coding practices, on finding these hotspots and trying to fix them

and try to avoid giving guidance on which APIs to use. So I think this is still not an afterthought, but it's pretty late in the process. When you want to,

when we started with the third chain, a Dynatrace, and more specifically about Grail, before we even started with any development, we needed to find what we really want to build. What is the core essence? What should be the core advantage and the core requirements we have for the new development? And having this in mind, I mean, you know, we are living in the age of data explosion. We are seeing more and more data coming in, terabytes of data, petabytes of data.

I mean, Dynatrace, in the cloud environments we are in, where we are hosted, we are getting tons of data. It's multiple petabytes per day that we need to analyze. So the efficiency is just the core ingredient. So what we started is really to look into the infrastructure that's available. So if you go for cloud, you need to see what's in there, which services are there. What are the capabilities of these services that are provided. You do your analysis and then you figure out that, for instance, using S3 compared to EBS

brings you a cost advantage of being only at 20% of the costs, but not only that. It also has higher reliability. So you even need not really to invest in sophisticated backup solutions in there. And so, you need to understand the infrastructure you work in and then build your architecture based for that infrastructure.

And therefore, it's really the core things are important. You know, you need to understand the core values of the new technical foundations that you have in hands. So it starts, performance engineering really starts in knowing things like network bandwidth, the speed of disks, CPU cycles, the speed of compression algorithms you can reach.

So doing your groundwork, you start with a, let's say, when we started, we started on an Excel sheet, just bringing all these numbers together and see what is the reachable performance using that kind of infrastructure. And that's your baseline. If your base basically what you

can produce in terms of throughput or response times on the capabilities of the infrastructure that you use, you have a great baseline. The thing now is you need to follow that baseline or monitor that baseline with the smallest increments you do when developing the system. Which means start with prototypes. Pure performance prototypes, if you can

verify that the reported performance is the one that you really will get out of it. So for instance, we have seen dramatic differences in using different APIs to access S3 and how to use these APIs. So you learn a lot. You can do so many wrong things in there that you only get out a fraction of the pure performance that S3 can deliver to you.

The next step is after prototyping your micro-benchmark your code. That means you are load testing your smallest increments and do that regularly in a regression fashion so that you are basically monitoring if there is something that you introduced that is uh that that is is uh slowing down you or making a uh in in this sense so so that's where I think we start with. And the importance is that the developer is knowing there's this baseline and you're comparing him to this baseline.

You know, I mean, engineers are usually pretty smart guys. And if you give them the right goals, they will follow you. And if you set that goals and you see, okay, let's see. I mean, the theoretical goals, they will follow you. And if you set that goals and you see, okay, let's see, I mean, the theoretical performance

you can achieve is this and your performance is just this, he will do everything to make it better. So I think there, I think I would always start with it.

So that was our approach, basically, for getting into third-chain. And we're still monitoring against the physical limits of the hardware we are using, how we compete with that, or how we stand there. Well, I've been taking so many notes. Thank you. Now, we have an ever-growing number of engineers that come into the organization. We are lucky that we are hiring people and because we are expanding our product portfolio.

Do you know when we have new hires that don't yet have maybe the performance mindsets that existing people have. How do we get this performance mindset in there? Do we force them into some training? Do we force them to look into data? Do we make it apparent to them what impact they have when they change code and switch to a new API without the proper testing?

It's quite a problem. I need to say there's an onboarding I'm doing, which is basically, I mean, brings our architectural principles into play, meaning that we are performance aware, that you need to understand what is performance, what is scal scalability how to reach it what what are the things to to do there on that high scale although it might be that this onboarding after a month is forgotten about it's like um if you don't monitor it might be that your test coverage is declining. It's the same in here. This is a continuous investment. I think what makes it... So I think approach you need to do

is that within a team that you have, there's always one that is knowledgeable about the rules, that stays behind these rules, that basically helps to onboard these members to think in that way. Because very often, engineers want to use the newest tool every year. I mean, there's a new API coming. I want to use this. I want to use that. There's a new pattern. I want to apply the newest tool every year. I mean, there's every time there's a new API coming.

I want to use this. I want to use that. There's a new pattern. I want to apply that pattern. And they forget about that every new tool, every new approach that you are learning takes a while until you are confident with it. And at the beginning, you're making a lot of mistakes.

And also to say not all the approaches that are new are the good ones. You know, if it's performance that you need to keep in mind, the simpler you write your code, the less you use that you don't understand is the better. So I think this is also one advice is use things that you understand

and not use abstract. If you use abstractions and really do not understand what's going on behind the scenes, abstraction can be very bad. I was going to say that abstraction but that sounds exactly like the hibernate thing we used to always look at right you don't

understand that and that was that was one of the classic patterns in our old you know easy travel system was just throw it in there um you know one thing you said too about first defining what your requirements are when, you know, performance and all that, then going and finding what technology you're going to run on. So if you're going to do some new project, are we going to run it on Kubernetes? Are we going to run it on serverless? Are we going to run it on what, right? You have to go through and find out which is going to help service you to get to those requirements. And then, as you mentioned, the big thing is test to make sure that it can deliver.

Two things that call to mind is, one, obviously, when we first started talking about Grail way back on the show, there was the idea, I forget who we were talking to, someone else within Dynatrace, that, okay, we got it working on Amazon, but then when they started to try to port that over to Azure, they found out that the standard disk performance was completely different between Azure and Amazon. So you can't just move. You got to do all that testing that you're saying. But I think the real conflict this brings back all the time,

like I made a joke about it in my opening about, oh, it's got to be released by this date. In the world of our customers, there's always the rush to get it out the door. And people will then, I want to pick, we've seen it a million times, I want to go to Kubernetes because that's what everyone's doing. I'm going to just start running it on Kubernetes and put it out. I'm not going to research to learn deeply everything involved in knowing how to use it, how to use it properly how

to understand those abstractions and then test to make sure i'm getting that right so now you do a lot of catch-up later on trying to make it work better trying to figure out why things aren't giving you what you need whereas if they were given the time to do that research and testing in the beginning. While it may delay the initial release, once it's out, it'll be much more stable, much more solid, and they'll spend a lot less time going back and fixing and trying it. So it's always this chicken and egg, you know, which came first, chicken and egg kind of thing, where, you know, partially keeps us in business, right? It's terrible to say, but because people just get rushed to do things and can't do these steps that I'm so glad to hear that we do from that side. It's the same problem overall that we've always had. Get it out as quick as possible and sacrifice all that pre-work, which is going to save you in the end. And I just don't know if there's ever going to be a solution to that unless we end up having AI making all these decisions

and doing all this research because it can do it so much faster, which obviously then puts people out of jobs. But what are you going to do about that? Fascinating stuff. It is fascinating, yeah. But as you said, it's always a chicken and egg problem. You need to get a new business idea out the door to actually make money versus spending a lot of time on the ground work

and then maybe never making it to the market because somebody else beats you to it. We need to make sure our experimental sub isn't going to implode under the pressure of the ocean before we launch it. But hey, we've got to make the money. Ernst, the thing that Brian actually brought up, because we have a lot of discussions with people that also talk about hybrid cloud, like really running on different infrastructures, whether it's different cloud vendors or a combination of on-premise and in the cloud.

Anything else that you learned or that we learned as we moved to different vendors? Is there anything learned? Yeah, just like things that you've learned maybe that would be interesting for our listeners because many of them I think are also in a similar situation. I think what we have learned is that it's much more effort to go to multi-cloud than stay on one, even if you use Kubernetes as an abstraction layer. I mean, there are so many things where you think the service are similar and they are not. I mean, Brian, you mentioned disk performance. There's a difference in disk performance between Azure and AWS.

Yes, and it's sometimes substantial. There is also a difference if you think you're using the same database. You're going after Postgres. You're using Aurora on AWS. There's no Aurora on Azure. So you're using Postgres there. You immediately will find out that operational-wise huge differences in how to adopt new versions

on the performance characteristics on even consistency issues that you run into, you would not have thought because you think you are working on the same software. So I would say plan for more time. It's much more work going to multi-cloud than just staying on a single cloud. And Kubernetes is not helping you as much

there as you might think about. Yeah. And I think that's the, I mean, obviously there's a good vision with Kubernetes as the abstraction layer, but every vendor, whether it's EKS, AKS, GKE, they all have their differences and also the different services, different performance characteristics. And if we ever get there, that you can just move the same workload over and it works exactly the same way, we'll see.

I mean, that's the promise that never will get fulfilled it's like you are moving from one sql based database to another and it works you know yeah and everybody's going to want to try to say we have a better performance thing so they're going to try to outdo each other no matter what and whether or not they hit the mark or what you have to change in the way you interact with it in order to get that performance boost. I just don't think that's the reality, unfortunately. Maybe one advice if you go multi-cloud also can be that you're not relying on the native services the cloud provider is providing and you invest into the abstraction that then

is comparable or similar in different clouds because then the behavior might not be that different. Although that's a pretty large investment, I mean, thinking about how to orchestrate, for instance, your own database, your own Kafka service compared to using the services that are provided by the cloud providers. You know, Andy, it makes me think of Akamas again, because just even if you talk about this disk thing, right,

there are several different options in each of these rollouts, right? You can go with the standard disk, you can go premium disk, you can pick different CPUs, right? So if you had a tool, you know, like we talked about with Akamasa in the past that can try all these different settings, get the feedback, and then tweak those systems, that can

help in optimizing the differences between the different vendors. And I know they were doing something with that, but I don't know if they're testing all that kind of stuff. But it's the idea of using that feedback loop and automation and trying different switches and options to find that. So if anyone from there is listening, there's a new idea.

Yeah, exactly. I mean, I think that our friends from Akamas are doing something similar. I mean, it's goal-based optimization where you set a goal and then they try to tweak all the settings on your runtimes, on your Kubernetes cluster, on your VMs underneath the hood to reach that goal and then find the optimal configuration. So if people are listening in and they've never heard about them, Akamas, there's a whole company that they're using, obviously, observability data also from Dynatrace

to then observe if the experiments actually deliver what they promise. Ernst, we started the podcast when you said it was in the late 80s when you were at the university. I'm not sure if you're still involved somehow with the academic world, if you are familiar with what's happening right now on universities, but do you know, is performance a topic these days? Is this taught at universities or is this an afterthought? I think it's, I mean, I need to be fair because I'm not that involved. But what I see from the students coming out, I think performance does not get the right attendance.

It still is that many basic things are lost. important for creating a performance system in favor of teaching new techniques, new programming languages. Because there are so many, you need to be trained in all of this. But the foundation is pretty simple, what a computer is doing and I think the foundation gets too less attention on universities. Because as I said, I mean, you need to understand the foundations to know what to change to make a system performing. Otherwise, it's a trial and error approach. Yeah. Well, then maybe we were really lucky when we grew up. Because I also remember when I went to HTL, so that's like a technical high school, we

started with assembler. And really appreciate it, learned the hard way how to move bits around and how to be efficient with your memory and everything. And that's also what you said in the beginning, have a minimalistic approach. How would the system behave if you have only half of the resources that you have right now? What would you change?

Exactly. There's no application for lean systems anymore. I think this is one of the things that I need to acknowledge. Although in the end, I mean, the things what we are doing, for instance, if we would not you kind of gave away your age a little bit because you were in university in the late 80s. That means you've been around the block for a couple of years. He was a baby genius, so he was in university at 10. Let's keep it like that. Like a little Doogie Howser or a Cooper. Um, and,

uh, anything that you, um, like looking into the future, is there anything that you see on the horizon where you say, Hey, you know, I wish the next generation that is coming out of university. Now out of university in five years,

this is some of the things that you maybe, that you maybe want to give them along the way. If there's anything else that we haven't already said. I would say, try to understand the principles. Go back to really the things where basic algorithms are teached, where data structures are teached. Try to understand that basics because that's the best foundation you can have. Without that,

everything, you know, you're learning abstractions. And as I said before, without knowing what these abstractions are due with the actual hardware it's running on, it's hard to get to really great software.

So folks, if you're listening, if you're just coming out of kindergarten and you're going into a career, if you're out of kindergarten and you listen to this podcast, you're probably a genius anyway, if you made it that long. But in all seriousness, if you are new in the career, if you're making a career change, in all seriousness, if you are new in the career, if you are making a career change, I know a lot of people are doing career changes and trying to go into software engineering. And I think that's a great advice

because it's very easy to click together an app, very easy to click together a new service. But if you don't understand the foundational pieces, then you build something that is not sustainable that is not performing and that eventually will break the bank and that's not a good thing so going back to the basics and learning the foundations

I also really like just to recap a little bit of the things that I learned today is the whole thing you said in the very beginning if you're designing a new app, if you're creating a new service, think about what is the real requirement to the end user. If you have a big screen, do you really need to update everything all the time?

Brian, I remember when the web performance optimization days, I remember we had, at the times when we were still celebrating when websites crashed because they had so much load and people were kind of happy about it. I don't remember the company anymore, but it was also during Super Bowl. Yeah, the hardware one or something?

The lumber or something? Yeah, something like that. There were companies, I think it was GoDaddy. GoDaddy was basically building a second version, a second version, a trimmed down slim version of their website. And basically switched over during the Super Bowl

because they said, hey, we want to make sure our site is up and running. And we decided these are the five features that people need to do. They need to be able to search the key products and buy something. And this needs to work. All the rest, all the clutter, we can live without it during the peak period. And I think there was also kind of this minimalistic approach. What's the minimalistic version of the app so that it gets the value to the end users? Yeah.

See if you can get your code to fit on a three and a half inch floppy disk. Well, I joke about that, but back, you know, at some times there was like, there were websites, especially back then, you know, we know they're heavier now and we can, we can handle them a little better now, but you would have an image that wouldn't even on a page that wouldn't even fit on an old floppy, you know, and it's just like, what are you doing? But I think also that example you gave that was a breakthrough right that was some genius came up with that and i don't know you know i guess

fortunately we haven't heard of site crashes in a while i don't know how much we're paying attention to them anymore i'm sure james pulley's still looking at them but uh uh i wonder if that's been a problem that's been solved if people have really embraced that let's let's get rid of the excess stuff on our site or however they're handling load uh during those peaks but i think that um again that's just a great thought to have in your head thinking about load and and as you're saying if they're coming out of university hopefully you're thinking about that performance aspect of your code and always looking at it and always having an eye on what it does. We even go back to

way back to Gorenka, right? You're going to define how your code's going to perform before it goes out. And if it doesn't, it's getting yanked. But how many people can afford to do that in their companies, right? Well, Ernst, thank you so much for doing this podcast, for being an inspiration. I mean, I can only talk about myself, but you really inspired and in a very cool and positive way

steered my life into a direction where I am right now, thanks to the stuff that you taught me back then and continuously teaching me. And yeah, I'm looking forward to many more conversations and in the Dynatrace office in Linz, obviously to many more chats next to the beer app because that's always a good place. Did you say the beer app? The tap. I thought you said the beer app.

Thanks. It was really a pleasure. I mean, I just spoke about things that I really like and I'm really, really, I think, very into that. And, you know, performance is a key driver of my whole life as something that I think is important for us all. With all the involvement we have, we still need to look into being efficient and being nice to the world, you know? I mean, this is also part of, I think,

thinking we need to apply less resources for the same results. Yeah. And I think it's amazing to see how far the performance field has come since those earlier days, right? It used to only be an afterthought. It used to be only a thing that people would look at when they had a problem. And we're not in a perfect world yet, but I think it's definitely much more front and center. Obviously, the observability world wouldn't be so big right now if people weren't thinking of performance. The fact that it went from performance to its own term of observability, you know, even, is quite a thing.

So it's been amazing to see how it's evolved. And hopefully with efforts from people like you Ernst it'll keep evolving and getting better. It's been a real pleasure to meet you and thanks for sharing the ideas with with our listeners. Any closing thoughts from anyone? I only say tagline will be from performance testing to designing and architecting for performance. I think that's the big shift. Don't test after defect, but design and architect

for performance. Awesome. Alright, well thank you everyone for listening as always and we hope to have many more years of this going forward and many more great guests like Ernst as we move on so thanks everyone

for listening appreciate it thank you bye