PurePerformance - Dynatrace PERFORM 2018 Wednesday Afternoon Break

Episode Date: January 31, 2018

Popcorn time

Thank you. Diolch yn fawr. Check. Hello. Hey. Hey. Sorry about all that scratching yeah uh so i was trying to play that performance story and i think the audio is completely gone uh maybe we'll have to look at that later yes you're being meta though yeah that's meta exactly we're back in the afternoon break, Wednesday afternoon. Wednesday. What? No, it was Monday Monday was the song.

Well, the original was Wednesday Wednesday. It's not Wednesday. It's halfway through the week. There's also the, you can kiss me on a Monday, a Monday, a Monday. I heard the original demo. It was originally Wednesday Wednesday. Was it originally Wednesday? Halfway through the week. It's not.

Three more days to work. I don't think that's right. I'm feeling rather weak. That's enough. And then on Wednesday. I'm going to mute your mic. Brian. Brian.

We're in the third day. It is Wednesday. Can you tell we're in the third day by our vendor? How was the popcorn? I'm going to go get some popcorn. Let's go get some popcorn. We're going to get some popcorn? you tell we're in the third day by our vendor? How was the popcorn? I'm going to go get some popcorn. We're going to get some popcorn? Yeah. I had some popcorn already.

No, come get some more. Yeah, well, I don't want to eat anymore. Yes. Yummy popcorn. Hi. Thank you. Popcorn. Hello.

Where's this popcorn from? Right there. Right here. It's from right there. It's from Yolanda. Yolanda made it. Yolanda made it. You made the thank you.

Thank you. Smells fresh. Delicious popcorn. Thank you. Smells fresh. Delicious popcorn. Thank you. That's awesome. We're in the Wednesday afternoon break live broadcasting for Perform Dynatrace 2018 in Livery Stable, Las Vegas. Or I was going to say Lunghead, Las Vegas.

Lung Cancer, Las Vegas. That's actually probably appropriate. Actually, yes. Very appropriate. You know, it's funny because it brings you back, doesn't it? To when I'm so not used to being around smoking. Yeah. Yes, very appropriate. You know, it's funny because it brings you back, doesn't it? I'm so not used to being around smoking. Yeah, and one thing I remember.

In Europe, they probably still have a lot of smoking. No, a lot of places they don't, though. They don't? At least not in Italy, outdoors, but not inside. Not indoors, but indoors, yeah. I remember back in my younger days when I was playing in bands, and we'd go play shows, and you'd get home at like 3, 3.30 in the morning because you'd always be playing last,

or then you'd have to pack up and unload everything, and you'd get home, and you just reeked of smoke, and you're like, I'm really tired, I really want to go to sleep, but I really want to take a shower. Yeah. And then the smoking bans went into effect, and I'd be like, get home, and be like,

oh, I'm going to sleep. It was wonderful, quite wonderful. You're saving a lot of time. Yes. And water. And my eyes didn't have to hurt anymore. like, get home and be like, oh, I'm going to sleep. It was wonderful. Quite wonderful. You're saving a lot of time. Yes. And water. And my eyes didn't have to hurt anymore. Oh, yeah. It was good.

Can you imagine if you were allergic? To small clubs? Allergic to smoke in a small club? Anyhow. All right. So we're going to endeavor to capture just a few more performance stories. Yes. There's a chance we have a story from Wassum, which was about the inside the firewall, outside

the firewall ISP. Right. That was a great story. It was a great story. We may not have it because, I don't know, some technical difficulty. We'll have to see. One thing I wanted to ask you about him, because their big Dynatrace shop, at the time they were using Atmon. Right.

And they turned on UAM, which is very similar to your Puzzler. They turned on RUM. Right. But they turned on UAM, which is very similar to your puzzler. They turned on RUM. Right. But they turned on UAM. Wouldn't they have seen, did UAM tell them the difference between the, if UAM would happen wherever those local customers versus remote customers, why didn't UAM just instantly say, yes, of course, it's the ISP, have a nice day? Well, UAM doesn't know it's the ISP, right? You wouldn't see it in the transaction flow that there's a component?

You don't see routing? No, no, no. Isn't there infrastructure? I just don't know. I don't have UEM. I'm not a UEM. Now, in Dynatrace, you can add infrastructure. If there's an IP picked up or if there's something, you can insert it there,

but it's still not going to necessarily – I don't recall if it's going to be able to give time to it because it's not doing measurements, but you can in the service flows. I know at the very minimum, and I know any of my colleagues listening is like, Brian, don't you know the tool? It's not stuff I do on a regular basis, but I have to

go back and look. But I know you can insert infrastructure like that, your routers, all that, into the service flow view so that you know what's in between there. So you have a gateway or a firewall interface. And you know that there's a thing. They're like, oh, maybe it's that. I'll go check that because I know it's there now. And you could see that between two tiers, and you know that there's a thing. They're like, oh, maybe it's that. I'll go check that because I know it's there now.

And you could see that segment maybe between some external components that you just know that the IPs are out there. The other really cool thing you can do, though, let's say you have an F5. Everyone's got an F5, right? You have an F5. You put your F5 in there, and you have it on the diagram, which you can also do. We were talking about API this morning, right?

Right. You can also pull the metrics from your F5 into Dynatrace and assign it to that node. So now you're getting your F5 metrics into Dynatrace. Because it's all tagged. Yeah. This would tag it right onto that node. And you can just insert there.

You're going to have whatever metrics your F5 exposes, you can pull into Dynatrace and see what's going on right there too. I like that. So anyway, a little... But in his story, he had... In his story, he would have seen... Like an ISP issue of some kind, which I'm thinking could be like... But he said it was only the users in Saudi Arabia, wasn't it?

Saudi Arabia, yeah. It wasn't the... So it was the local ISP. It was the local ISP in Saudi Arabia, but then there were people all over the world. But they were fine, right? Because there weren't those synthetics running from an exterior node. That wasn't having an issue or some kind of proxy from an exterior node they had the internal users were fine so uem was telling them transactions are slow it's all these people in this location and if they actually had um bandwidth calculation turned on which is an atmon

it would say yeah your bandwidth is like a dial-up. It looks like you've got a 56k connection here, right? They were running their international banking customers on a 56k modem. Andrew was just here. He didn't want to get on the mic, though. Yeah, do you want to get on the mic? You've got to hang around if you want to get on the mic, brother.

I don't want the mic. You sure? You guys don't want me on the mic. Really? That is what you do on the mic. I sure you guys don't want me on the mic really oh you'll just start talking that's that is what you do on the mic i don't know if you know that thank you very much for moderating my session and you kept me dude right on time it was excellent thank you spot on i ran the he was my moderator for my session or we're live so you don't get to talk to us right now because people need to hear these things. So that was a good performance story. It was, and it's another one of those interesting ones

because I just wanted to point out, because when people get to hear it, one of the typical types of performance stories you don't hear about, because usually it's, oh, we found this crazy issue that was isolated here. This was, they found it by eliminating everything else, which helped them figure out it was something outside. This was, they found it by eliminating everything else, which helped them figure out

it was something outside of their control. And there was one thing left. Process of elimination. And that was their story. So Wassam's story was your ISP sucks, and UAM helped you prove that, yep, it's got to be out there somewhere. Exactly.

And of course, they could use the new plugin browser thing. But that's why I asked. This is a commercial banking site all there. It's not a SaaS solution. Well, they wouldn't even have to use the browser plug-in. They could just use RUM. Standard RUM. The same thing, right?

Right. Same bits of it. Very, very nice. And, in fact, the cool thing would be in Dynatrace now would be instead of them looking at the dashboards and seeing their slowness, problem patterns would automatically detect there's a degradation for your users in Saudi Arabia. Yeah. Actually, you'd ask Davis. Yeah.

Davis, what's up in Saudi Arabia? Well, I'm noticing some bottlenecks on the network in front of the component XYZ gateway. And sure enough, you want me to auto-dial the ISP support number for you. Right, because we also can pick up the ISPs. Sure. I think that's pretty cool.

And now you could put it rendered in augmented reality as the globe in front of you in your VR goggles. And in fact, what you would see in that area would be instead of all these high-speed connections, you would see an animation of a hand putting a phone in a cradle for a dial-up kind of a connection.

And play that modem pickup sound. Yeah. Yeah. Ding, ding, ding, ding, ding. Yeah. Yep. Wasn't it? You know what's sad?

There's going to be so many people in this world as we age who... Will never hear that again. Never know what that is. Although it'll probably be some kind of... Growing up, there was all these things from the 60s. Groovy Man is probably some far-removed bastardization of what was really going on in those time periods and all. But we have these nostalgic things that come back up into our pop culture. What?

I am sure the modem sound will come back into pop culture, and I'm sure there's probably some kind of really crummy podcast or two that uses it. At the beginning of the peer performance intro music. No, I think when we go dark net and we have to kind of go back off of the spy-driven, spy-governmental-dominated real Internet, which will, let's say 20 years from now, the current Internet will be like CompuServe. No one will be on it. It'll be so laden with crap and slow and awful.

They'll be like, forget it. Why would I ever go on the real Internet? I get on my neighborhood alternate, darknet, neighborhood net. So I'm getting into this Unify, Ubiquiti networking stuff. Yes, I've seen some of that. And they have a whole bunch of the devices. You can set up, like, just a local, in your neighborhood setup. And then you've got to follow the FCC rules, of course.

Right. Because you're broadcasting, you know, signal strengths and stuff. But Ubiquiti helps you manage that. And you can set up in your neighborhood with your local separate network. But you can't get onto the regular internet from there. Or if you do, then you're just feeding into it. But let's say you had a bartering system, a schedule of volunteering. You're depending on your neighbors keeping the device up.

And that's the weakness of those is that the neighbors, we need someone else to run a second pipeline and go back to the Internet of 1993. We're going to go back to Pine Mail. And 56K modems. And WebCrawler. Yeah. Well, we'll do it. Metacrawler, man.

Metacrawler was good. Yeah. Yeah. So, ah, the good old days. Yeah, here we were talking about the innovative Dynatrace things. It's the ISP thing in the 56K that brought us back. What is AIOps? I saw that on the screen this morning.

What's your take? Are you hearing anything more about it? Because Klaus was mentioning this thing, right? He was, and I'll just be straight up honest. I was only half tuned in on it this morning. So it's not no ops. There always is ops, but why not make it ai ops well i think you know part of it is and and i don't know if this is what they exactly what they were announcing today because we've announced it also in some blog posts earlier and i think i brought it up yesterday the idea of having your into automating remediation.

Right? Fixing yourself. Also, using it to automatically detect new dependencies. You know, normally, you spin up a new service, whatever, suddenly all the dependencies are met, but that's kind of standard. As soon as you put in a new thing, Dynatrace is going to learn all that.

That's going to become part of the AI. Right. So that's kind of, in a way, AIOps, because you're discovering all your dependencies of your networks, everything else. But to me, if you say AI ops, that gets me thinking, as you said before, no ops. Which gets me thinking, as we say,

certain conditions are met. The AI is going to find a detector problem. It's going to know the root cause. And there's going to be some new rules that you put into it and say, if root cause X, Y, Z. Blue-green switch.

And that's, I think, part of the spirit of the Ansible Tower. Throw some new things in. Do whatever. Sort of intelligent elasticity, not just sort of dumb elasticity. Or even, like you were saying, the switch. Yeah. You know, there was a, you know, I never remember who we talked to to but they have feature flags they were talking about not necessarily green appointments but yeah feature flag toggles to turn on and off so it could be all right let's uh turn on the toggle we're going

to go ahead and throw the switch and based on performance metrics they can you know you can maybe have it automatically turn off that switch if it's not working with that you haven't even make that decision yep like oh you know what i it's terrible. Testing in or operating in a compliance and regulated type environment, a lot of times you don't have the ability to sort of at will turn on, turn off. Yeah. Because it might be starting on date X, all of these transactions must capture for regulation purposes this new field. And so once you cross that line, you're really going, you can't turn it off.

Because now the price we pay when you turn something off like that is there's an incident and it's fully quantified by how many records were out of compliance. And then you have to design a backfill. And in some cases, it's a list of, let's say it's, in my case, 400 customers for that time period. You have to reach out to them with an email. We're sorry about X, Y, and Z. We would like to do some remediation on your account for compliance purposes, but it requires us to obtain your approval for X, Y, and Z.

It could be, I'm just describing a fictitious, very costly remediation to backfill those records. Right. So I have a heart for people in more compliance and regulated environments that don't have all this fun, free, I can just flip stuff on and off. People who aren't working on the Facebooks, right? Yeah. Where they have 10,000 servers. Yeah. We'll put it on three, we'll roll it out.

Right. But let me ask you, since you work in a much more compliance and regulated area, let's say there's a new regulation you have to comply with by next Friday. Right. Could you theoretically in most situations, or tell me what part of the situation, how many you could do this in, have it ready by today, turn it on where you don't need this information yet, work out issues turn on and off, and that gives you a week in production collecting the data you don't need to submit yet. This way you can go back and tweak and fix and have it ready by the time. I mean, sometimes it might be you can't see that data

by regulation until they say you can see the data, so I'd understand that. So you could have sort of, call it a beta period, call it a trial. It's deployed in a default off toggle. So I think the mechanisms are absolutely easy to use. It's just understanding with certain,

in certain situations, just because you can turn Fegel's toggle on, a feature toggle on, in certain situations you may not really be elegantly able automatically with AI, you would tell AI once this is on you can't turn it off and then that's a different condition. That's a different

state for the AI engine to figure out. Right. And this brings up a different future thought. Future thinking. Future thought. The future with Brian Wilson. So, AI,

whatever you want to call it all, based on regulatory verticals, whatever. So, maybe you have a more banking-tuned AI, which operates and has a lot of fundamental differences in it. Yes. Based on the fact that there's regulations and some of that stuff can be fed in even. Oh. Right? You have your general Facebook-y type of fun site, which has almost no regulation. But where you can build this, you can build a lot of these rules. You can feed.

Context-driven AI basics. And it would be, you would have teams working on AI that's more specific to banks. I think you could call that, let's say, an intentional bias. Yes. I'm going to take my machine learning algorithm. I'm going to tell it what things are more dangerous than others, and it would be different in different contexts. Right. So context-driven AI.

There would be commonality amongst the banking industry. Exactly. So they might think about it and get it. It might be a different branch of AI. Right, right. Now, you could also figure out how to deploy very strict rules engines within to the AI. This is something that you would do in other threshold-based things. We're relying too much, I think, though, on human beings to know all of the hundreds, if not thousands, of compliance and regulation and that.

And there's also, not to mention, there's many different ways to become compliant in terms of process, in terms of where the data lives and how you regulate that information as it flows and changes. So some people are like, this section of our application is absolutely quote unquote not compliant,

but that's because we've made the case that once it gets into our monolithic old back database, that's where our system of record is stored and that's where we enforce compliance. Right, right. Whereas other people say, well, we don't really have that monolithic database. It's kind of messaging spread out everywhere. Then if I were a regulator, if I were an inspector, I'd be like, you know what? You guys are really storing this data out in the system. You need to count those areas now as

where you're going to wave the flag and say, I am compliant because I did X. So the AI engine would have to adapt depending on how you made an argument that you were compliant because you implemented X way. And that's the subtlety of really,

really, really difficult stuff. I've got another future thought here, too, as you were talking. The future. The future. Future. Future. We've seen recently, I think it was in the last 12 months or less, there was a government conference where a lot of government IT was showing off the new way they're doing things. Most people think government is old, monolithic, everything's running on tape drives.

But there's a lot of innovation that a lot of government teams are picking up, right? They're starting to do containers, they're starting to do these things, right? So we can say, most people would think government's probably really backwards in their IT, but we know there's a lot of parts of government that are not backwards in their IT. So what if then as well, with these these regulations there were a regulation API that's you can then interface to say X, Y, and Z happened

ping the API comes back saying according to regulation blah blah blah the system being compliant this way comes back feeds and helps direct a remediation so you automate the remediation recommendation store of regulations and the checks.

So you would subscribe, my system subscribes to the following publicly shared regulations from the regulation body. Let's say it's the FDIC or the HIPAA compliance. There's somewhere that organizational enforcement body says, these are the rule sets. Let's say you're using vendor X. Here's rules for you. Vendor Y, rules for you. The consortium or a public consortium coming together, that's a good idea. Yeah, it could even be like if you had an issue, or let's say you had a problem arise in your system,

it could take the characteristics of those problems. And again, I don't know how this would exactly be possible, but feed that to the API. The API can say, based on the X, Y, and Z, you're out of compliance on this piece. You have to remediate over here. And this is, again, future. I like it. Hi, James.

Did you have a nice chat? Hi, guys. You guys drilled down a little more. Yes. Yeah. That was good. Yeah. Matt and I drilled down a little bit more, particularly in the challenge of the AngularJS thing.

It is. I suggested. I talked to Paul Grisafi, who hates coding against AngularJS. Yeah, I suggested that cake at the results presentation. And when you have an AngularJS app, you order the cake in advance. And you take the picture of the app and you put it on top of the cake in the frosting. Yes. And it's a three-layer cake so the top six and a half inches are angel food that represents the gooey and then you have a chocolate cookie crust yes which represents http yes and

then you have something deep dark rum soaked fruit included which represents sockets in this model yes and um and then so you bring everyone for the results you cut them a piece of cake and they're looking really oddly at this disproportionate cake and you're like oh well i'm testing from the cookie crust layer on down because the server doesn't care about what's in this fluffy angel food cake layer yeah it just knows i have a request and response pair. The simplest way I think I've heard you describe it is in that old adage, if it doesn't scale for one, it won't scale for many. Correct.

Right? Correct. So, which means, do I need to include all my single user, single session layers of the architecture if I really need to test the scalability of the shared layers of the multi-session layers of the architecture? No. So you're really, you're saying, look, you should definitely test the performance of the end web browser. Absolutely.

Oh, absolutely. I mean, that's the traditional, if you roll back a few years, GUI virtual user for five or ten users and then the majority of the time under load. And then you could get login underscore GUI minus login underscore HTTP. It's like, oh, okay, that's the weight of my client type of thing. Get on the mic there. Okay, I'm on the mic there.

But I think to your point, there are some web app architectures still running in a browser that do not just make a call to a web service, where you could just test the web service. And that's the line from single session to multi-session, and that's nice and clean. REST API, SOAP API, the tool. You don't have the weird GUI correlation stuff. Yep.

So there are still people, even with AngularJS, that have multiple asynchronous components all making concurrent web service calls at the same time. And that's what he was saying in the recording. It's like, clean up your GUIs, guys. Have a discrete presentation layer in the client and just call some web services. And you know what? Set a budget. Set a maximum budget on a representative piece of hardware that the GUI action shall take no longer than X amount of time. And then just measure it for one user? Do it during your functional test automation? Measure it. And if you never answer that question, you are always going to be unsatisfied when you get to the multi-user performance testing. Right.

Because you have no basis of understanding, well, did it work before? Yeah. Essentially, you were absent data that it said that it worked or didn't work. Very good. It may take 45 seconds for one user, and then you report it takes 40 for a whole lot of others. It's like, well, that's broken. Yes.

What's X Matters? I don't know. That looks cool. Slack, bots, Dynatrace. Is that what matters most to people? X Matters? I don't know. I don't know.

Should we go talk to them? Connect your tools with actionable communications. I like it. X Matters. There's another one. Ops Genie is here. Sauce Labs is here. They're giving out here. Are you out of hot sauce? I know. At Test Bash, we got

so much hot sauce. It's beyond me. They still have a bunch sitting over there, which is pretty awesome. So yeah, this is our afternoon session. We think maybe all of that talking with Wassam about the inside, the ISP, outside, we think it didn't actually record.

So we just sort of gave kind of a shortcut version of that. Yeah, see? Being meta. But I think people will be interested to hear it. Did you remember anything? What struck you about his story? Do you remember anything? It was really more negative testing than anything else.

That is, I'm eliminating all of the conditions that would normally be problematic. And he walked right around the stack. He said, it was our CPU. It wasn't that. It was our memory. It wasn't that. The disk, it wasn't that. And then I piped in and said, oh, well, it must be network.

Yeah. Well, it was to a degree. His data center ISP was not providing the full clean pipe and bandwidth. We think he was on a 56K modem. And very likely he didn't have any sort of monitoring tools on the throughput of that interface. They were independent of anything else. They had application monitoring tools, but the least cost routing on their, what they thought was going out and coming back into their data center actually was just bouncing back through a proxy

and coming back on the same third-party link because MPLS, that's the least cost route in order to get there. So I totally understand what happened there. But their monitoring agents out in Europe and elsewhere in the world were absolutely telling them something was wrong. I think they could benefit from a performance-based network monitoring solution. He was saying in Dynatrace you can put components, if you know them out on the route, you can put your F5 out on the route. You can put a gateway.

Even if you don't own the gateway, you can put it on the map. And you've got the API. Any metrics that it's exposing, you can pull in as well. And I thought in the story they were saying in Europe and elsewhere it was fine. It was only Saudi Arabia where the problem was. No, no, no. It was the ISP in Saudi Arabia that had an issue. So they were seeing the problem in Europe.

So they were seeing the problem with actual clients, including their synthetic agents in Europe. But inside of their data center in Saudi Arabia, they didn't see it. Their synthetic agents, which were based in their data center, but supposedly going out through a proxy, through a third party, were just simply routing back along the same line.

So it was never really going out and hitting their inbound ISP provider. Well, again, that's why you're... So even going to something as simple as synthetic agent testing, right? If you're running the original advanced synthetics, aka Gomez, aka Keynote, which are now... Okay, well, how do you decouple a network flow from the outflow in that case? A network flow from the outflow in that case. Because for a synthetic user, you actually have to test the app.

In this case, it was a network transit problem. How would that? They would have seen it. Well, they saw it with UEM already, but they would have also confirmed if their synthetic was outside, which is where these synthetics run. So on a purely control basis, they can say, yeah, it looks like it's the network.

But I call up my ISP and my ISP says, oh, no, it is your application. Your application is at fault. ISPs have funny accents. Yeah. So. No, but you would see. No, you would know. Exactly.

If you're running a synthetic test from the outside and collecting the application data, so if you're running a synthetic test with Dynatrace on the inside, you have the exact length of time it takes to execute on your application stack, which would be 300 milliseconds. You would see that it's taking 15 seconds on your browser, and you would also see time to first byte being 10 seconds. So you would know 300 milliseconds from my web server or whatever on down. You'd still know it's on your network segment.

You could. You would. Okay, you would by virtue of control elements. Right. Yes, you've nailed it to the network. What I'm suggesting is that there are many people who will refuse to accept that because you do not have a clean network sample. You do not have a trace route which shows the number of milliseconds involved

from hop to hop to hop that's showing that as an example. Well, that's where you put the idea. And, Mark, I see your hand is raised. The idea, though, is once you identify that it's your network, that's when you go in deeper. You should go and do those. That's when you go in deeper because you need to first know where do we concentrate our efforts. Mr. Tomlinson. Mark, you in the corner.

Lode Runner 702. Yes. I remember 702. Such a weird major release. 702. Yes. HTTP. You can have HTTP trace routing remember that yeah and it would do trace

routes and figure out the segments and then monitor those segments for you during the load test yeah and then for some reason it went away now probably an intellectual property could be an issue licensing I have no idea yeah but it went away. And to me, HTTPing. And you know what? As much as Lode Runner had a sublime... No, you don't need to wax philosophical. No, I just want to lament... We're going back to Joe Hoffman.

I want to lament Lode Runner for a moment. Because I was a Lode Runner. I used Lode Runner. You were a Lode Runner? I was a Lode Runner. You were a runner of the loads? I was collecting all the gold and digging the trenches for people to fall into. It's a BMW product.

It's a highly engineered product for highly engineered use. But it had so many different protocols and things that testing that you would probably hardly ever use. But every once in a while, you'd be like, oh, you need to use this one weird protocol. And they always had it. Yeah. And then you need, oh, I i gotta figure out how to do this weird function let me take a look at the function library because i had such a history oh they have a function for they have some weird you know and it was just like i mean again a lot of other things i could say to complain about it but man it just when the chips were down you could count on it yep and the world may move forward i don't think it's necessarily moved backwards.

Because if you're living in a C shop in a Windows world, you could still do some serious damage. I would go back to that. I would go back to Lodron or Heartbeat if I were in those environments. Because you can't do that kind of stuff with JMeter, can you? Can you? I don't know. You have to bring all these other open source pieces together. JMeter doesn't give you that kind of stuff.

And it also favors, since it's mostly used by developers, that you want that function, write it yourself. Yeah. Right? I mean, what are you paying for? And then, oh, you'll go into business, and it's the open source model, right? I'm going to pour a little out for load earner. Exactly. They give you the power of extensibility, but they don't do the extending for you. Yeah. And remember, when you write it and extend it, you are required to give it back to the open source community or Richard Stallman will come to your house.

That's right. And do something foul. No, we don't know that for certain. I think he would probably just, a lot of people on the forums would be very angry with you. He would say mean things at you. But to be honest, the foundation would write you a letter and you'd have to be warned. And then there would be legal proceedings probably at that point. As you don't contribute back.

Oh, I see. Yeah, because they'll pull your ability to use open source in your product through consortium. And you'll end up with a class action lawsuit. Really? Yeah. Interesting. There's more than one way to do it. I asked Linksys how that worked out for them.

Yeah. Well, we're not legal experts, and we just want to advise everyone on the podcast that you should not consider any of our words formal legal advice and consult with a proper attorney who's licensed in your area to give you proper advice. Attorney James Pulley. Not named James Pulley. Yeah. And we don't really know Richard Stoll.

Someone who has actually passed the bar. Exam. Exam. That's right. All right. So we're in afternoon Wednesday, and I have the strange feeling that this place is going to start to be torn down. Well, it's the closing sessions now. Is it the closing sessions now?

But I don't see the live stream. We're back in the main room. Yeah, I don't see the live stream. But they're back on main stage. So maybe we tune into the live stream. Maybe they're having some issues. But, yeah, it's back see the live stream. But they're back on main stage. So maybe we tune into the live stream. Maybe they're having some issues. But yeah, it's back on the main stage. Maybe these are still considered non-live stream because it's still customer stories.

Could be. On the main stage. Yeah. But this is going until 5, the next big session. And then 5 to 6.30 is a break. Is a break. And then we pick back up at the Brooklyn Bowl for a party. Do they come in here and break? I don't think so. I think we're going to at the Brooklyn Bowl for a party. Do they come in here and break?

I don't think so. I think we're going to start seeing all this stuff breaking down. Do we want to do our giveaway? Yes. Should we do our giveaway? Yes. Because we just have to. I can do a rundown of at least the stories I know.

And you guys can tell me if I missed any of the stories. Are you ready? Yes. All right. The first one was Aparo? Aparo? Aparo. Aparo.

And he talks about having a load test situation where he's monitoring a system that all of a sudden went from, let's say, 700 milliseconds of load time down to something, or like several seconds of load time, down to, like it looks like it got really fast. Like it went down to like 60 milliseconds. Like everything's been running great, and then all of a sudden all the metrics showed that the response time got really, really, really fast.

And it turned out to be something weird being monitored. Some sort of like attack. But it was only loading a one pixel by one pixel GIF that skewed all of the metrics. And it wasn't enough for their security to pick up on. Like DDoS. But it was enough to throw the mathematics off. Right, but they ended up finding that this heavy traffic was occurring because the response time dropped. Right.

So where is this traffic coming from, but things look really, really, really fast? Like, if you've ever done a load test where you're expecting the application under test to only give you one second response time. Now, all of a sudden, the test tool, which maybe has no think time or any pacing in it, all of a sudden you start getting 404s, and the transactions per second go through the roof because it's just spinning. And the transaction response time, if you're not checking for response, goes really, really low. It goes really, really fast. Yeah, yeah, yeah.

500s come back really fast. Really fast. Well, I've had one 500s come back really fast. Really fast. Well, I've had one where we were testing search way back. It was a three-tier search. I think it was in DECA in the middle or something else. I forget what the whole setup was years and years ago. You know, we're testing against the front end. There's a middle layer, and then there's the back end search database.

Right. Search database was returning no results to the middle layer. The middle layer was packaging that up nicely and saying, we're sorry, there are no results. And the results were coming back, you know, 20 milliseconds, 20 milliseconds, because that back end was down. And the middle layer was like, I got an error, so I'm going to say there are no results. So it was, again,

it was that nice way of handling it, which threw off... All right, so we've got Aparos giving the one-by-one GIF. Brian Rumsfeld gave us using Dynatrace to compare moving from... with load testing, right? So he did a load testing before to prep and move. Yep. So that was using Dynatrace, bringing Dynatrace in and to do a comparison.

But unfortunately, they didn't buy Dynatrace after that. It was very upsetting. Yeah, yeah. They acquired all the value and then refused to recognize the value. Yeah. So that's just kind of a sad story. Joe Hoffman told a story in WebLogic where WebLogic had implemented sort of a self-healing, changing the configuration of WebLogic, maybe auto-restarting it so that it could run in a more healthy way.

And the customer had shut them down. Right. Because it's, do not get your hands off my stuff. Get out of my. I don't want your agent running in my thing. They were prying our cold, dead hands off of their. Yeah. So they were vehemently opposed as a customer to this.

And you go down the road, what, two, three years later, he talks about coming back and talking with them. And sure enough, the new people he's talking to are like, wait, you can automatically adjust and restart for us? Of course we want that. Oh, my God. Where have you been all my life? Why would we want to spend all of our hours just doing that manual adjustment and restarting? I don't know. Exactly.

So his point being, times change, and sometimes you're just way ahead of your time. Yeah. Right? Harish? I don't remember Harish. He's the guy who had been a load tester for 20 years and then wanted to meet me and then told a story. I believe so. He was standing right here.

Yes, I do believe so. But I don't remember what his story was. It's probably on the air. Nope. Going to have to go back and listen to that. Okay. Do you remember it? No.

Now I'm feeling bad. We have to remember his. Or maybe that wasn't. No, that was'm feeling bad. We have to remember his. Or maybe that wasn't. No, that was Aparo. He was saying the one-by-one gif. That was the guy that was standing right there. Harsh is a different guy. But, again, I don't remember his story.

I didn't write a thing down there. Jeff Leonard shared a story about the slow middle tier app that ends up being in the middle tier. All on the queue side. Yeah, and making calls into the ESB, but it wasn't the ESB. It was all about the producer side of the queues, not the consumer side. Producer bottleneck in that. And then the one we just potentially missed from Wasm, which was your ISP sucks, turn on UEM,

and then call the ISP and tell them they suck. Which I think that's the shortest way I can abbreviate that story. Now, were there any other stories that you guys remember that I don't have on the list here? No, I think they covered... Is there anyone else you want to nominate? Is there anyone that stands out most for you? Hmm, James, what are you thinking? Should I turn on the Je jeopardy themes music there um i always like a good q story so i'm going to be biased towards uh

jeff leonard to jeff leonard in this case that's a and it was a weird one it was a weird one yeah it was you know leave it to jeff come up with a weird one but yeah it was a weird one. It was a weird one. Yeah. It was, you know, leave it to Jeff to come up with a weird one. But, yeah, it was a weird one. Agreed. All right. So I'm showing my cognitive biases in performance engineering. All right. In this case, which is a Whopper topic this year.

That's cognitive biases in performance engineering? Yeah. Brian, do you have a favorite out of the list there? We might have to arm wrestle for it. I myself am a little... One, two, three, four, I'd play a thumper. I'm a little particular to the non-traditional stories where it's... We were going to look for the problem and we found it by not finding it. So I'm leaning towards Wasam a little bit.

He did have a good story. It was one of those stories most performance issue stories are about. We found a problem with code. There was a queuing issue. There was this issue. We looked and looked. Everything on our side was absolutely fine.

Everything on our side was fine. It turned out. You still have to be smart enough to second guess and was fine. And it turned out... You still have to be smart enough to second guess and eliminate stuff. Yeah, and then having the data to go to that third party and say, it was you, stop saying it's not, and we can prove it. Put together a cogent argument. And then when they finally go, yeah, I guess it is me. That's when you're in a situation like that,

when you can tell that other vendor it is you and here's why, and they go, we can't argue with you, you're right a situation like that, and you can tell that other vendor, it is you and here's why. And they go, we can't argue with you. You're right. It's us. Kind of torn. That's kind of a. And again, I'm not super. There's a lot of good stories.

But that's just the one that struck me. I like Jeff's. And we can always come up with other giveaways that we want to maybe come in for a second. Because runner up. Popcorn? No, a pair of Perfite shoes. Yeah. That's always on the list if we want to do that. The supreme prize will be the Dynatrace UFO

and Amazon Echo. I actually kind of want to vote for Aparos 1x GIF, 1x1x GIF pixel. He had all the right tools again and all of the metrics are being gathered and everything, and he still had to do. It's very similar to Wasamzo. It's like you have all the right tools and all the right monitoring.

You still have to do extra work. And it was a puzzle. So we're down to three. Down to three. So now we need to settle on a first runner-up. Yeah, gold, silver, bronze. You know. settle on a first and a first place yeah the gold silver bronze you know and considering i am part of the vendor i think it would be better for you two to pick no no no it doesn't matter we don't we don't do this no no usually we have my wife do this

so and miss 2018 perf bites award is hmm uh now now Who do you think should get the gold? He's going to say Jeff. You know, we both talked about Wassam. I think Wassam's in the lead. I think so, because we all kind of have an affinity. And he got us talking. Got us talking. He got us talking more than...

You know what? Let's go with him. All right. Well, Sam's going to be our lead. He's going to be the winner, the top winner of this for the ISP sucks story. The only problem I have is that none of our listeners can actually hear him say it potentially. We have to go back and check. I have to go back and see if the MP3 got whacked.

But if they can't hear it, then it's going to be one of those We'll just invite him to an episode and he'll come up. We can just invite him to come and get on. It's not too late we can go find him somewhere. Or we'll find him online and put it on the Twitterverse. Now the toss up is

between Jeff and Op. Yes? She's going to decide for us? James' mom is on All right, now the toss-up is between Jeff and... Yes? Oh, she's going to decide for us? James' mom is on the phone there. Yes, yes, my mom. So now we've got to... First listener of PerfBytes is calling for feedback.

Sometimes the only listener, I think. So hi, James' mom. He'll be on the phone, I promise, in just a second. Jeff Leonard and Aparo. See, I did not hear Jeff's story, because I was trying to mix that other one. I cannot contribute to the vote. It's a good story. I cannot contribute to the vote.

I think I'm biased. I think Jeff, the middle tier is actually, they're all kind of similar puzzle type. The thing you think is the problem is not the problem and you have to go really break it all three of them are almost like puzzles yeah in that way they're not like some remember we we met off top off top was here one the two years ago three yeah he won a pair of shoes for and but it was story was like more like traditional load testing bottleneck database problem you know things you would find normal configuration issues, Apache threads, and something like that. These are all like everything that we were doing was right,

but each of those three were the thing you think it is is not what it is. I think the thing you think it is is not what you think it is. Right? Does that make me Andre the Giant in this conversation? Give me an argument. I'll stay impartial. I'm telling you, you're Tor Johnson. So why would you say Jeff should be second place?

And we'll do a pair of perp I choose. Okay, so normally it's the consumer that's the issue. Yes. That is, you're waiting, you're waiting, you're waiting, you're waiting, and you have a direct line to the response. Here we had a broker in the middle, the Enterprise Services bus. But why is that more unique to make him second place over a borrow that has the 1x1xGef skewing the existing results monitoring? Why is Jeff's story more compelling?

That's the thing, it isn't. Why is it? Make your case. Why do you think Jeff is? I could go for a geographical bias. He's from North Carolina. Well, that's very honest of you. But really, complexity-wise, was it more or less complex?

Was it unique? I did not hear a powerful story, so I'm at a challenge in that case. I think you're the only one who's heard both of them then. I've heard all three of them, actually. So I think this is going to have to rest squarely on your shoulders. You want me to just pull... You want to flip a point? Or we give each of them one shoe.

Whoa. How about second and third place both get perfect shoes? Whoa. How about we do that? One pair of low tops, one pair of high tops we could yes we could yeah just because one makes second place we get high tops the other one normal converse yeah yeah they're a little bit different yeah yeah and you can get more letters yeah on the on the heel all right well we're gonna don't. Yeah, yeah. All right. We're going to go with that. We're going to go with our congratulations. Second place will be Jeff Leonard.

Okay. Third place will be Aparo's 1x1xgif. 1x1xgif. I like that. All of them were good stories, though. Very good ones. Fantastic stories. And again, different than the other.

Yeah. And you know what's something nice about them? We weren't flooded with stories of the same old performance problems that have been going on and on and on. The first year we did this at Perform, there was a lot of repetitive. I was once testing a thing with a thing. People were making the exact same. Now we're getting into bizarre mistakes.

It was a dark and stormy night, and I was trapped in a data center. Hopefully people are making less of the same mistakes. I don't think it's true. I know based on when Andy would always do that share your pure path always over and over and over. Oh, M plus one again. M plus one on services. M plus one on this. Same problems over and over and over again.

You should listen to News of the Dam. Same patterns. But are we starting this? Are we breaking the pattern? Or did we just get good stories? People said, I'm going to tell them more. I think we just got some great stories. Great story. It also reflects the maturity of the problems and how people are attacking these problems differently.

Here's the new thing we're going to do after three years. We're going to invite all three of these guys to get on and do an interview with us. Do a PerfBytes interview and actually retell the complete story. Since it's on us, maybe we didn't capture some stories. Let's turn it into their opportunity to actually share their story. I love that idea. All right. So not only do they get to be a guest on PerfBytes and maybe simulcast to Peer Performance, they could come on Peer Performance and maybe tell an actual story.

Maybe as well. The Dynatrace portion. But we'll give them the gifts. And that's our gifts. Gifts will be the gifts. This year, not a drone, unfortunately. Yeah. We didn't give a drone.

That would be bad. All right. We're going to wrap it up. Maybe do some live stuff tonight from the party. We'll have to see. I want to try that. I'll show you the live setup. Spreaker on the phone?

Spreaker on the phone with the mobile mic. Oh. Yeah. You can do it with an H2N into your phone on the Spreaker app, or we'll try that a little later. I think that'll be fun. I've got USB-C on my phone. We'll see. Ha-ha.

Ha-ha. But how was your performance experience then, James? This is our third year. It was good. This one different? I've got to tell you. So there's a natural maturation cycle to these performance-related conferences. We hit, what, 2,000, 2,500 attendees? I think it was around 2,000 they were saying.

It's pretty major. This time. You're going to have now a conference in Europe as well, probably of similar size. I'll say over 3,000. It's moving from less pure engineering to there's some marketing side of it. Coming from the Mercury universe, it's like, okay, it's still wild and crazy performance engineering, and I love it. It's so fun. But, boy, it's got a little marketing.

But don't give them too much leeway. Don't let them take over. I'm watching warily. Yep. It's like, okay, if the engineers lose control. You'll lose the community. And the marketers just take over. Yeah, yeah.

But as long as they keep doing cool and innovative stuff, and that's what we've seen every single year. The cool and innovative keeps getting bigger and more mature. And so I'm excited. Like I said, I have my AI rule sets, and I'm waiting for the day when I can come and import my XML for my AI rule sets, and it will run my rule sets in the enterprise. The pulley perf rules. Yes. Import.

Yes. Very nice. Yes. Brian, second perform ever. I was attending one of the first ones. One of the early, early first ones. Yes. But here in the podcast.

Yes. Second perform. Have a good experience. Was this good? Yeah. The only complaint I have to say is the hallway walk. That's it? Maybe some of the breakfast foods.

But overall, I mean, listen. Lunch was good. Listen to Hot Days. I do have one complaint. Yeah. Dr. Pepper. There's no Dr. Pepper on the service breaks. But that's positive feedback.

Not necessarily a complaint. It's a recommendation for future optimization. As far as impressions this year, I think the hot days were jam-packed. It was great to have you as a puzzler. It was great to be a puzzler. We had a really good morning session on introduction

to Dynatrace. Last year, I helped out with introduction to Atman. And this year, making the switch, introduction to Dynatrace. And it was great to see so many people in there. And a lot of the people there were Atman users,

which they're starting to, within their company, make that transition, which all of us SCs, all of us old Atman sales engineers had to make that transition too. So as we're teaching this and they're asking questions,

I'm seeing exactly what was going on in my mind, going on in their mind, and knowing, oh, we can help you. I've been through exactly those steps. You're going to get there. You're going to love it. And I just even remember just being blown away during the first break on hot day going into the main hall

because there was also partner day going on that day, and the hall was pretty much you almost i would say a fire hazard not because it was full all the way down but the whole breadth of the hallway was filled up and you had to kind of squeeze in between people to go get a coffee i was like wow this is and this is the exact opposite because you have so much space it takes like you know 40 minutes just to walk to the next session. Well, but that was all here today. That was the hot days.

Oh, yeah, sorry. Yeah. In the hot days, yeah, sorry. But that's when I knew, okay, wow, this is getting much bigger. This is growing and growing and just felt a lot more, just felt a lot of excitement in there. Everything I think was on par with the Cosmo. The breakfast was a little lackluster. Not good non-carb, low-carb, no paleo options on the diet.

But that's always just fine feedback. I agree with the Dr. Pepper. The soda didn't come out at the same time in the morning as the coffee. They put it out afterwards. It's weird. The sequencing of the soda. It's weird that they didn't put out soda in the morning, right, for breakfast? Yeah, exactly.

Very weird. I will say, all of the announcements. Thank you. Thank you for your support, gentlemen. And we debated for weeks after last year's perform, like, what do you mean AI? We're really drilling, Andy. Like, where is the AI? Yeah, is it a full rules-based thing?

Is it ambiguous? Is it pattern recognition? Where does Davis fit in? All of the forward-looking, cool announcements were more pragmatic, except for the VR goggle. That's just, thank you, Joe Hoffman. You're way ahead of your time. That's moonshot stuff. It's way, but great to show the world what you can do.

That's what's going on in there. I want to see if it's still running. What? Once we're done. The Donatrace box over there, you can try it out. You can try it out. That's why we might have to do that. So, I mean, I dig it.

I love the idea, but it's clearly intangible to most of your potential target customers. I mean, we're trapped in a cube doing our work on a monitor. We're not standing up doing VR. So kudos to the early adopters. We finally come up with a use for Google Glass, and it's a canceled product. Yeah, I know. Exactly. So maybe Dynatrace can go buy Google Glass and just implement it that way.

There we go. But other than that, it's been a fantastic experience. I hope the listeners at home, I think we were technically a little messed up, but that's meta. We're going to fix some of these recordings and clean them up like we do every year. Then we'll put them back up and you'll have another download then if you subscribe to the download. Yeah, that'll work. It's awesome. More meta.

More meta. Woo-hoo. I'm so meta, I'm meta. You're meta meta. All right. So we will catch you guys later listening online. Thanks for listening for a few days. If you're not a Dynatrace user, all the trial stuff, I think it is.

Just go to dynatrace.com. Dynatrace.com. All the trials there. Go to the AWS App Store. Oh, that's right. You go to the AWS App Store. Marketplace.'s right. You go to the AWS App Store. Marketplace. Marketplace.

Yeah. There you go. You can get everything you need. But to sign up for the free trial, can we sign up on there? Is that what you said? Probably redirect you. Give it a shot. Yeah, it might redirect you.

You can drive innovation and modernization with AWS. I mean, our SaaS model resides in AWS. Yeah. So I'm sure you're right. They'll fire up your trial, and away you go. That's beautiful. When you sign up for the trial, it's like a few minutes, maybe a minute or two,

and here's the login for your SaaS instance. All right. Anyway, so we may or may not be back on later. If we're not, thank you all for listening. And a big hug-ops to all of our listeners. And a big thanks to James. Big hug-ops. And who are you, Mark?

I don't know. For making this happen. I don't know. It's a good one. For allowing me to be, because of you guys, I get to be here again this year. What do we got in the future to join us? We're going to get in the cadence of some episodes. We've got a News of the Damned.

We didn't really do any News of the Damned this week. So we have Super Bowl weekend. Maybe we do a Super Bowl broadcast. I've got to tell you, there's going to be some News of the Damned episodes coming right after the Super Bowl. So next week. Next week we're going to be alive

and well. Right. And be talking about patterns. And remember talking about patterns. And remember, the big pattern this week is probably that left hand, right hand between marketing and operations. Not communicating. You've got a big commercial coming. Yep. Cultural clash.

Yeah, you're spending $5 million on the commercial. You don't push any of that money over to ops just to make sure everything is happy and secure. And, you know, some people watch the Super Bowl for the events. I'm going to be watching for the commercials. Well, actually, maybe not because that's Sunday, and I think I'm going to be busy Sunday. Mm-hmm. I'm not going to be busy on Sunday. But you're not out at any shows or anything happening local in Denver there?

Nothing? Is there a Dynatr or anything happening local in Denver there? Nothing? Is there a Dynatrace meetup group in Denver? Nope. Should there be? Nope, because then I'd have to go to it. Unless it's during the day. There you go. I'll be at TISCWA in the end of February.

So look for the Triangle Information Systems Quality Assurance. And then we'll be at STPCon. It'll be the next live broadcast. And that'll be in Newport Beach. Newport Beach, California in April. I think it's April 9th or something. And you know, Brian, if you really feel like taking a trip, just come out to

Newport Beach and hang with us. We're going to be doing some live podcasts. Thanks, everybody. We'll talk to you later on the PerfBytes slash Pure Performance podcast. Yes, thank you. Bye-bye. Thank you for listening. Bye, everyone. Thanks, Mom.