PurePerformance - Dynatrace Perform 2018 Tuesday Night Session 1

Episode Date: January 31, 2018

We're live. Okay. All right, just hold, don't cover this part, hold it down here. And hello, Rao. We're back at Perform 2018, Dynatrace, Las Vegas. It's Tuesday night, and I am here with Rao, who came by, saw one of our SoundCube Bluetooth speakers and said, how do I get one of those? So he stepped to the challenge.

Hello, Rao. Nice to meet you. Nice meeting you. Something should inspire to say something or do something, right? Yes, exactly. That's good. So let's hear this. You have a very interesting story because most of the times, as we know, people see performance degradations and they know there's a problem. You had an opposite situation.

Yes. Explain this, what happened here. So I've been a performance engineer for the last 18, 19 years by passion, by profession. And so I came across a very interesting performance, I would say a problem, because everybody, like you just rightly said, the whole Dynatrace conference or any APM looks for performance degradation and how to solve the problems. But when I had the performance dashboard up and running in a production command center, one fine day the SOC was, all the guys were there and they're watching the dashboards,

and the performance baseline was around two and a half, between two and three seconds. Right. Not bad. Not bad for, you know. It's not Google, but it's not bad. It's not bad either. So one day, and we had only the response time on it. So it just dropped down.

It dropped to a few hundred of milliseconds. So this was very interesting. The guy said, oh, we are running really fast, really fast. But being a Dynatrace guy and a performance engineer, like I said, by passion and profession, it sounded too good to be true. Yeah, because you said it got down to from three seconds to about 100 milliseconds. Yeah, from around between two and a half to three seconds. You've probably seen these things, Mark, as well. A transaction going from three seconds, normal, down to about 100 milliseconds.

Yeah. Right? You think we just made an amazing, in a production system, we just had an amazing, In a production system on a normal day. Your first reaction, right, as a long-term load profession

would be, we just had a wonderful performance improvement, correct? No, something started airing out. Well, let's hear, let's hear, let's hear around.

Something started, something that used to take time is no longer taking time. That's a 404s or fast. I'm a big fan of you, so I love these too much, but I've been following you for a long time. Two decades, close to two decades.

Am I even close? That would be my first gut instinct. Actually, there are no errors. No exceptions. Well, could be swallowing the exception somewhere in the stack. Dynatrace could catch them. True. True dat.

True dat, homie. Oh, it's like that now, huh? I'm from Philly. I keep going. Tell the story. You stumped the joke, man. But what happened is, see, when you build dashboards, people are pretty much interested only in the response time. So they build dashboards only with the response times.

And then there's a little bit of what you call request throughput, but not all requests are captured all the time. What happens in Dynatrace is with the business transactions, so people are interested in putting in dashboards only their business transactions. So the cash flow is happening, receives are happening,

sends are happening, users are happy, user experience is happy, everything is nice. Or it looks nice. They're normal. I would say nice, but nice is always perspective. Something good for me, not good for... But they're all normal, no glaring things at all. Or it looks nice. They're normal. I would say nice, but it depends. Nice is always perspective. Something good for me, not good for... Yeah, right. But they're all normal.

No glaring things at all. Okay. But only the response time only dropped from that seconds to 100 milliseconds. It was a big... Actually, I was supposed... It's huge. Yeah, it's huge. All right.

And I was supposed to give a presentation. I was on a waiting list, so I have the deck also still there to especially showcase that. All right. Then something definitely is weird. So we started digging. We went into the UEM, and we found out there were a couple of IPs, not couple, probably several hundred IPs, only doing a small one-by-one GIF, and they're only doing that. They're just trying to only hit that part.

So it's skewing the numbers. Exactly. Because some functional issue that was only grabbing this one by one, like a beacon GIF of some kind. Somebody deliberately trying something on the website, only trying to access that one by one GIF

and they have hundreds of hits. So for some reason those are being labeled the same as the regular request? Yeah, they're not regular requests because if somebody hits the home page, if somebody hits the home page, it would load all of them and it would take two to three seconds. But somebody, when they're trying to access only that one-by-one J, the response time is just 100 milliseconds because it is just the cache

and it's coming out. But the greater transaction gets skewed because it looks now, it's like, oh, my gosh, it's super, super fast. So the average response time off actually got completely skewed, and we are performing at 100 milliseconds. Look, I'm not going to toot my own horn because I'm not a horn player. But I said something that used to work, that took time, stopped working, like a 404, well, they're not calling the whole page. It's not a 404. You're right, though.

That's good. No errors. There's no 404. There's no exceptions. It's a valid HTTP request. Yes, and that is why even in it. See, that was a very, very, very interesting case I wanted to share in Dynatrace anyway, but you guys are lucky you got that case.

No, that's good. But I've been a performance engineer also for two decades, and I've been following you many times. Good for you. Yeah. So this was something that I went to the NOC Street, Network Operations Center, and they said it's not even hitting that threshold of denial of service to actually alert there in the tools.

So they didn't catch it either. So we showed them. Oh, yeah. Yeah, because it's not that high. So that's interesting because it's not frequent enough to actually that's interesting because... But it is high enough to see, not frequent enough to actually... And probably not, depending on the DDoS, probably not coming from the same location. Same IP, exactly. Not coming from the same IP, same location.

So that was very, very weird. And so when we went into the network and security, we alerted both of them. We said there's something weird that is happening. They could not block a set of IP because it's coming really randomly. Right. So that was really, they could not block a set of IP because it's coming really randomly. Right. So that was really, really weird. But at the end of the day, what they did is they instantly renamed that one-by-one GIF to something else, and then they stopped it because there's no way you can stop all those IPs.

Right. It was so random. But the interesting thing there is if you put that, if response time is too good to be true, yes, it is wrong. It's not true. Too good to be true, it's not true. What's interesting about that is you're finding a possible attack based on a

decrease in response time, not because the network things got kicked off. I was recently talking to, we were talking to Harold from... Harold. And we were talking about some of the IoT stuff. So we have an IoT agent and all that. And if you recall, a little over a year ago, there was a DDoS attack done by web security cameras and all that. Right. Now, we didn't have Dynastrace monitors

for IoT at that time. But if people had, one of the characteristics people noticed afterward was if you were analyzing, you know, this is still a computer little program on those things. Right. The memory consumption

used by those programs on the web thing, as this update occurred and this malware got injected, increased. Increased. So if they knew

what the memory size was supposed to be and were monitoring the memory size of these IoT devices, they would have seen an increase and said, we don't have an increase. Increase. Right. So sometimes it's not always the obvious metric that's going to capture an attack of some sort, like a decrease in response time.

Who would think that could trigger? But you investigated, which is like... And even if people, because there were no alerts, no exceptions, no errors, nothing to trigger on, it was not high enough for the DDoS to trigger. The network security could not catch because it's coming from multiple IPs. Right. And again, so that was a very, very classic example of how a performance engineer goes through, and I'm a performance engineer like you, and I'm sharing the story. No, no, I'm thinking you could also alert on total page size.

That was a... Like if it falls out of trend. Like if you're running in the test mode, you could look at the trend for the total page size. It depends on the app because some pages naturally have high payload, low payload, depending on what they're doing with catalogs, images, media, whatever. Since you're accessing only the GIF, it's not even a page. So you won't get that. Here's the thing, though. I'd have to check with one of my colleagues.

Did he win? Oh, yeah. You're definitely going to win. You get a sound cube. It's a Bluetooth sound cube. Come on now. So, by the way, what you missed was before you came back, he was like, well, I'm just here by myself.

I'm like, well, let's do it. But he was like, what if I take it now and come back tomorrow? But I actually wanted to meet Mark more than anything. No, no. So here's the interesting thing, and I'd have to check with one of my colleagues. I'm finally starting the list. You're at the top of the list to win our prize. You know what our prize is?

No. A Dynatrace UFO and an Amazon Echo. I have both, but I still want one more to play with. Then you could give it away to a colleague who would appreciate it. No, I'll tell you. The Dynatrace UFO that we are trying to still build has a limitation on hardware because of the enterprise security. But I want it as a part of the company. But if I win this, I'm going to take it home and build my home with that. So I don't want to give it away to a colleague.

Because I've already given it to a company, not a colleague. So here's the one thing I wanted to say about this, that I'd have to double check with a colleague with. But I know in Dynatrace, if you have a sudden drop off in traffic, of even a single request type, we're going to alert. I also believe that Dynatrace AI will, if you have a sudden increase in traffic of a single request, so since it's that GIF

file that I was looking for, I think Dynatrace might have flagged that for you as opposed to using Atmon. I'm noticing excessive high throughput on a certain resource. This is not our normal traffic for the time and it might have given you an alert, which is...

And that is exactly what I am hoping when we transition to Dynatrace and AI enable it, that's what I'm hoping to capture and put it as a traffic. Interesting. Dynatrace, though it is for monitoring, I have used it,

or rather I've enabled it in security. I've enabled it in testing, obviously. So Dynatrace has a lot more than performance monitoring. And tomorrow there's going to be some announcements about some AI Engine 2.0. I'm very excited and eager to hear all about that. Me too. I've only heard hints, so I'm very eager to hear myself. Especially the 2 billion.

Okay, I don't want to spoil it. I know that. No, no, you're not spoiling it. Well, no, if you know it, then it's... And somebody leaked it. We don't know any spoiler. I know that. No, no, you're not supposed to. Well, no, if you know it, then it's... And somebody leaked it. Yeah. We don't like leakers. Unless it's normal biological function.

Yeah. It's a teaser, not a leak teaser. That's right. Exactly. Exactly right. Good, good. Awesome. So how was the rest of the show?

What sessions? How are you finding the rest of the... It's great. Any favorite sessions? Did you do hot sessions? No, not this year. Last year we did. Not this year. But I'm looking forward for the year. Did you do hot sessions no I not this year last year we did not this year

but I'm looking forward for the year did you do a puzzler with us last year he did his talk already you spoke up with the people one no

it's tomorrow that's what I registered for oh you have a different one today I was on the main stage you're on the main stage I saw and I've been at CMG I heard your talk

and I also gave a couple of them that's what it was not this year but a couple of years back I gave the performance engineering the agile way and all that I've been a speakerG, I heard your talk, and I also gave a couple of them. Oh, yeah, yeah, that's what it was. Not this year, but a couple of years back, I gave the performance engineering, the agile way and all. I've been a speaker there, too. Good for you. But anything memorable so far with Dynatrace Perform? Yes.

The ecosystem that they are building, you know, whatever, back in the early 2000s, or I would say mid-2000s, I'm part of the TCS as a team, which we built a tool called Genser. If you know Genser, we open-sourced it. And bytecode instrumentation, first time it was exposed, so we built a tool that can do that, and it could replay the JVM, and this was a decade back. It could replay the entire JVM of whatever happens,

and it's still there on open-source. Forge.net, it is there. At that time, the biggest challenge that we want is, can we replay the entire IT? Yeah. That was a very complex thing. If you can imagine almost 13, 14 years back, that's an extremely difficult task where multicore

was not even there. And you had Wiley and you had, remember, Performant? Yeah, Performant was there. Performant was the... Yeah, Quest. Yes, Wiley was there. Performant was the... Yeah, Quest. Yes, Wiley was there. And it's very difficult for them to do because we don't have enough CPU computing to do all that.

And we did a JVM replay. It was just too early. Too early. And I always wished, can we replay the entire IT? And they're coming up with it now. And that was a very good replay, IT replay. And that was really exciting for me because I was looking forward for it.

Watch out, T-Leaf. Yeah. T-Leaf also has that. Don't quote me on this, but I think one of the big differences, I don't think T-Leaf can handle single-page apps. No, it cannot do so much, but again, they got the concept. They were there.

They definitely built a big, strong... How long have you been there? You've been there more than 10 years. You own the place. You either own the place or you go home. Right. Either you innovate or perish. Kumram, which we bought, they took a whole new approach. I don't know the underneath.

What is it again? Kumram? Kumram. Kumram? Kumram. That's the replay. So a few months ago we announced that we bought this company, Kumram. I just like saying it.

Development teams in Barcelona. And I like saying 84 lumber. 84 lumber. Yes. It's from last year. Development teams in Barcelona. And I like saying 84 lumber. 84 lumber. Yes. It's from last year. 84 lumber. Yeah. But, yeah, we went and searched for, like, the innovators of a replay technology.

And this is who we came up with. So I'm really curious myself to find out what's under the hood of this, how it's going to work. Yep. There's so many questions I have about this thing. Yeah. I just think, yeah, I'm really excited about that one. Awesome. So nice performance story. Yep. Yes. Thank you about that one. Awesome. So nice performance story.

Yes, thank you so much. Thank you very much for doing it. I know we had cross paths before, so it's good to see you. It's a really nice meeting you, Mark. Do you still listen to the podcast? You should listen to... Do you listen to Pure Performance?

You listen to his podcast too, right? Ours is not so good, right? You can say no. You can say, you know what? Yours is terrible. Exercising, that's what you should do. Driving, commute. You have a commute. You take the train. True. While you're sleeping, you can listen, yours is terrible. Exercising, that's what you should do. Driving, commute, you have a commute, you take the train.

While you're sleeping, you can listen to it, and subliminally you'll observe all the information. That's right. That's a good idea. You know, if you play it backwards, you actually lose the things you learned. Unlearn. You unlearn. Unlearn the learn, which is what we should do. Just play it backwards.

Exactly. It's a pleasure, guys. Thank you. It's a pleasure to meet you. Thank you, and congratulations on the SoundCube. Yep, I look forward for the Echo. You have to go test that thing out. Yeah, sure.

No, you don't have to do it now. Yeah, but I definitely look forward for the Echo and UFO. You think you're going to win? Because that is something that I really want to play and probably... But you think you're going to win? You're feeling the feeling? Yes. It's like Vegas.

You've got to roll the dice, right? Yes, yes, yes. So that I can present it back to you again, what I did with it. Yeah, yeah, that would be cool. There's a lot of different things you can do. You know what, James Pulley? You mentioned the – no, it's James. You use the UFO in your home studio when you're on-air recording or not recording.

You mean the on-air light? Yeah. So you're going to use the entire – Use the UFO as all red. You're on the air, off the air. I just signed up for a new home, so that'll be a tool to automate my entire home. Yeah. There you go.

So that's what you do in it. If this, then that. You can put the UFO on IFTTT. Okay. Excellent. Thank you. It works with home security, not enterprise, so definitely I can put it at home. That is awesome.

Thanks, Ryan. Awesome. And enjoy the rest of the show. Brian. Yes. Oh, you know what? I'm not simulcasting. You're not?

What happened? Oh. I unplugged myself before. How did you unplug yourself? I'm doing blank. I'm going to turn this off. Who knows? Anyway, I think we'll take another little break.

Yeah. No. Wait a minute. How am I getting audio in here? Oh, because it's picking up the live. No. Dude, how is that

doing? Because it's coming, it's taking, it's taking this stuff. Off of the headphones? Off of the speaker here. Yeah. Well, I'm going to stop that.

Oh, that's fine. That's weird, but it's fine. I'll leave it for now. No, it's getting it off the Alpha. I think it's just saying it is. The Alpha's not plugged in. But how is it being responsive then? Because it has a built-in mic.

I think it's just coming in from the built-in mic at this point. That's really strange. Anyway, alright, enough of the meta. Enough of the meta. This is when things get really crazy usually at Diner Trace Perform because... Because what? Because we end up drinking a little scotch and we ask people to come and give their performance stories. Yes. Yes.

And we give things away. And there were a couple of stories I don't have on the list here That I have to go through and remember I also I have to stop my simulcast meta Or we'll just copy it over Yeah, we'll do it later Anyway

So Brian, this is We're coming to the close of day one We are coming to the close Which is really day two for you and me Yes, there have been long days You notice Here's the subtlety Third year we're doing this live podcast.

Yes. Is there any... If you do go to... You didn't... You weren't at Perform. No. This is your second Perform. Yeah.

Now, do you notice anything demonstrably different this year over last year? Well, there's a large AWS booth. Very tall. They figured out... They probably asked, what's the ceiling height? Yeah. I did notice that. Okay. Well, they probably asked, what's the ceiling height? Yeah. I did notice that. Okay.

Well, that's a very physical thing that you would witness. It seems a lot more lively. Yeah. I mean, last year was pretty good, but it just seems... Ecosystem of partners seems bigger. Uh-oh. Here comes trouble. Klaus!

Mark, can you believe it? Hold on. You need a microphone. He's going to help you. I'm going to see where this microphone... Phone comes on. Oh, this is really loud. Yeah, come on now.

Okay. Whoa, what's the difference in the volume? What do you want me to leave? Mark. Can I leave something? Mark. Mark. Mark.

Mark. Mark. Mark. Mark. Mark. Mark. Mark. Mark.

Mark. Mark. Mark. Mark. Mark. Mark. Mark. Mark.

Mark. Mark. Mark. Mark. Mark. Mark. Mark. Mark.

Mark. Mark. Mark. Mark. Mark. Mark. Mark. Mark.

Mark. Mark. Mark. Mark. Mark. Mark. Mark. Mark.

Mark. Mark. Mark. Mark. Mark. Mark. Mark. Mark. Mark. Mark. Mark, Mark, Mark, Mark, Mark. Can you believe it? It's the first time ever that the two of us are talking to each other here at Perform on your podcast. We never, ever had the chance.

And it's now like four times you are with us. This is our third year at Perform. Now, the first year you did a drive-, and you said a few things on the mic. Right. But we didn't get to talk at length. I think last year you gave me the middle finger, didn't you? No. Yeah, last year I think you just walked by, said nothing, gave us like a dirty look.

No, I'm kidding. No, I think I was just passing by. You were trying to impress me with being like, I'm I was just passing by. You were trying to... I'm trying to impress you with being like, I'm the super stressed guy. You were trying to lose weight to fit to your lederhosen. Yes. Exactly. I have my lederhosen as well.

Actually, I left my lederhosen this year. Lederhosen is at home. In Orlando? The lederhosen? Remember the big outdoor plaza and everyone got a little intoxicated? Yeah. The food was quite good, though. Yeah, absolutely.

That was beautiful. Beautiful. You know, right now there's a full moon out. Really? I've been outside. Yeah, I walked outside and had a phone call. There's an outside? Oh.

Yeah, there's an outside right up there. Do we have to walk like 15 minutes to get to the outside? No, no, no. Actually, you just go over there. There is this patio. This is awesome. Go there. How are things going?

How's this perform for you? This perform for me is... You were on the main stage there, weren't you? No, I was not on main stage. No, actually, I don't do main stage this year. You just don't. I just don't. Before we continue, Klaus, for our listeners,

do you want to tell people who you are? Because I know who you are. Mark knows who you are. Everybody knows who I am. So who's listening to your podcast that they don't know who I am? Well, you never know. New listeners.

Yeah. Okay, so my name is Klaus. What are you doing now? Even if we mentioned you in the past. What I'm doing now is I'm trying to do a little bit of technology strategy for Dynatrace. Okay.

And for Perform. Perform is kind of my baby. Yeah. I grew it from the very first one where we had like 60 people to now over 1,000. I remember talking to you the first time. It was like less than 100 people. Yeah. But it was good.

It was like a good 100 people. Now, was Perform back in the Dynatrace solo days you're talking about? Dynatrace solo days before Confluent. Because I went to one as a good hundred people. Now, was this performed back in the Diner Tree Solo days you're talking about? Diner Tree Solo days before Comfey. Yeah. Because I went to one as a customer. Yeah. The one that one of our favorite competitors was out handing out leaflets. Exactly that one.

That one. Exactly that one. What was that? That hotel. That's pretty cool. You know you're disrupting something if you can get that out of your competition. Yeah. Yeah.

Definitely. Definitely. You're doing exactly the right thing. I remember. This is a new fashion style. Putting both up there, yeah. And now, yeah, we put it together. Andy, your listeners know Andy, right?

Andy Grabner. We're also co-broadcasting. We don't know anything. Nobody knows Andy. We're also co-broadcasting on the Pure Performance podcast right now. Although not really. Wait, has he been on Pure Performance? Yep.

He has. Twice? Twice in a row. We've been on Cafe at least a few times. He's been on Cafe once. You've been on Pure Performance at least once, I think. But I don't know if you've been on the actual full-length show. Because you have two shows, Pure Performance and Pure Performance Cafe.

One is the shorter show. So that's the competition. The competition is for the longer show. It's like the Saturday Night Live Tom Hanks kind of, you know. Well, Alec Baldwin. He's not hosting, though. Steve Martin. So I'm going to get back on.

I'm going to beat you. I have to be on more than anyone. Is there anyone else done a threefer? Marcus Heimlich. Not Heimlich. Heimbach. Heimbach. Yeah.

He's been on twice, and we've scheduled a third recording two times and had to cancel so far. He's about to break to number three. Here's my issue. They count it as sessions. I've actually been on more episodes. I've been on three episodes. But we only sat down with you.

That's not fair. If you're a listener at home, that's three individual episodes. Totally. Here's the reason why it's a problem. If you're a listener at home, that's three individual episodes, right? Totally. But here's the reason why it's a problem. Because if I do change it to episodes, I'm going to have to go back through the entire history and recalculate all that. I don't want to do that. You would need to do that. Anyway, so you're doing tech strategy.

Tech strategy for Dynatrace. What is the future of the UFO? The future of the UFO. Because I have ideas. You have ideas? Power over Ether. First of all, we have to shrink it. The mini UFO.

Mini UFO. The personal edition. The personal edition, yes. You have to wear it. We have to make a wearable out of it. What do you think? Like a medallion? Yeah, yeah, yeah.

Like flavor-flavored. Medallion would be more cool. Like flavor-flavored, right? It's like, hey, green. Kind of medallion? Yeah. Medallion would be more cool. It's like, hey, green. I think you could do a big watch version. I'm thinking Buck Rogers in the 25th century. Dr. Theopolis. Yes. That's the name of my cell phone, actually. You could do a watch version because you could have the UFO tell time.

You know, the hands of the clock. Yeah, we could. So you could have, as a watch, it could tell time and then do other things. Tell you about the Bible and say this. What if you make, oh, oh, oh, oh, great idea. Great, I got it, I got it. It's not going to be UFO anymore, though. It's going to be a shock collar.

So if it turns red, you get zapped. Oh, I don't want to wear one of those. shock collar. So if it turns red, you get zapped. Oh. I don't want to wear one of those. Well, you know, there are certain cultures in certain parts of the world where I think that would be totally acceptable. So I'm glad that I'm

living in Austria. It's also a form of wearable. You could wear it in other places, too. That could be interesting. I think that's... Yeah, exactly. I totally get where you're going with this. We would be disruptive.

People would pay attention to us. We would get press. Totally. But I try to disrupt with something that is a little bit less... The beauty. Yeah, moral. Moral is the right term. I mean, ethically correct.

And actually help a little bit more. But outside of just miniaturization, which I dig from a... I do like the at my desk personal feedback. Because we still operate individual engineers, individual check-ins in empowered individuals in the chain. Right? They can see the shared accountability and impact of their actions, but it might be nice to have certain things that are just for you before it goes to the shared world.

So you could instrument individually on the pipeline with a personal UFO, and if you don't respond there, eventually the big UFO is going to go red. Yeah. So that's empowerment. You know what would be really cool? Maybe you can help me here. I'm looking for a meme. What can I put in front of the business guys

so that they start caring about what we care today? This is a thought that I'm having. The PM UFO? The PM UFO or even the CEO UFO? Ah. CEO UFO. You put somebody's CEO UFO. The Cufo.

We've got all this fancy automation, right? Things are going. But we're promising responsiveness, increased frequency, flexibility, push whenever you want to push. Don't push. Right? And the manual intervention can come on high. We were going to push this, push this, push this. No real hardware changes required.

Give the UFO manual override button to the executive branch so they can send a special signal to all the UFOs, stop what you're doing. Or go faster and if maybe special lights there's a, if you meet this sprint, there's more money at the end of the sprint.

I mean, you could give more from a corner office standpoint they do things like we just found out something for our corporate intel on the competition. We need to accelerate a feature or not. So maybe you give them the throttle, push the spinners faster. The pipeline throttle. Well, you know, Mark, there's a chance you might accidentally uncover an edge-level epileptic, you know, once that thing starts strobing.

That's an HR difficulty that they'll have to work out. No disrespect to actual people with serious epilepsy. I mean, he's right. He knows the one. So I have another idea for it outside of the business side. But right before we were talking about the developer side. Yeah, we were talking about the developer side. Right, so for people who commute via their cars to work, right?

You know, like the breathalyzer, you can't start your car until you breathe into it. If you, like, had, you can't start your, if you go to, if you leave work, you check in your code and you go to start your car and your thing is not green, your car will not start. Because it's car automation. And you know you've got to go back. That's an interesting one because it's a little bit more impactful.

Have you seen automotive-grade Linux? Yeah, so automotive-grade Linux. You can embed a UFO widget right in the center console. Maybe the smart cars of the future can get that in. Take me home. We could leverage our agents there. So it's like a closed system. Take me home. No, go back to the office

and fix your problem. Here's another idea. Another one for you. Hardware enhancement to the UFO. Add webcam. So you could see So you could see that an empty office. Interaction, right? Or you can see the fear on everybody's face or only when it goes right

does it record yeah and maybe make it part of a fun thing like if you get a bunch of people and they get it to go green it has a countdown to the team who just got green to do it and it'll post it to the social media from the camera because they'll all a UFO selfie when you get it to go green and you're like, yeah! Capture the celebration. Or if it goes red and you see somebody standing there going capture the despair. The despair, right?

You could capture that as well. That's really good ideas that you're having here. Webcam on the UFO. Webcam there. We have a miniaturization wearable thing now. And we have the UFO. Webcam there. We have a miniaturization wearable thing now. Yeah. And we have to throttle.

I think that's wrong. Yeah, we forget about that. Next year before, you're going to see all these ideas in real life. Let's see. Set aside the UFO. We do know some things AI with Davis were maybe going to escape just Alexa.

Does Davis have a future out in the wild as well? Actually, a lot of customers are already adopting it. Already asking. Asking for that. Actually, Davis kind of became the UFO for the C-levels. Just ask him, hey, is there a problem? So you put voice recognition and a speaker inside the UFO? Good idea.

They become one thing. Actually, Alexa is a good topic. Yeah. You know that we can monitor Alexa skills? Ooh. I think I did hear that. No, you haven't? What would be interesting from a failed Alexa skill?

Alexa, open the doors. You know what I mean? No, I mean, let's say you created a mission-critical app whereby, in some business case... Actually, we already created it. We added to our Easy Travel an Alexa skill for Easy Travel. You can book your travel by talking to Alexa. Through voice recognition. Through voice recognition, yeah, absolutely.

Demo test today during my presentation. If you're going to start using voice recognition as a means of a work objective, productivity, you know, getting a task completed, yeah, you'll need to monitor the device because it's not the employee's fault if something's faulty within the voice recognition. I'm sorry, Klaus, I must reboot now. Yeah, I've heard that today. I think that would have really come in handy for Jeff Bezos when he said, you know, Alexa, please buy me, pick up something from Whole Foods. And they bought Whole Foods because of a mistake. That was a bad meme that was going on for a long time. I don't think he heard your joke.

No, don't say it again. It's too late now. I'll tell a cow joke. No, do not. Well, wait a minute. You're only allowed one cow joke. You can ask Alexa to tell a joke. He has cow jokes.

You have cow jokes? Lots of cow jokes. And the joke's on you when you hear it. Okay. Do you want one? Do you want a cow joke? You're our guest right now. Yes, I need a cow joke.

All right, go for it. Okay, so a cow and a dog walk into a bar. Car, okay. A cow and a dog walk into a bar. Cow and a dog walk into a bar. And the cow says, hey, barkeep, I'll have a vodka tonic. The cow's like, all right, dog, what are you going to have? He's like, yeah, I think I'm going to have a white Russian. And the cow is like, wait, wait, wait, wait a minute.

Does a white Russian have milk in it? And the dog's like, yeah, it does. The cow says, that sounds delicious delicious i'll have one too so the joke is the joke's on you for thinking this is a joke see this is my routine i got a ton of them i can keep going so we have that's two in a row that are terrible here and we have steve barton and happy feet here that's terrible here. And we have Steve Barton and Happy Feet here. That's terrible. All right.

And we have Tor Johnson over here. I can't wait to hear the next terrible cow joke. Tomorrow. Tomorrow. June and tomorrow. Tomorrow morning over breakfast. To a day. How's your experience of the facilities? The food?

Thank you. You made us VIP. Oh. Laura, apparently we rank. So what does that get you? I'm not a VIP. I have no idea. Upgraded swag.

We're PerfBytes, three years running. We're VIPs, which means we have access to the VIP lounge. Hold on. They have free massage. You get $1,000 every time you say the word Dynatrace. I don't know if you know that. It's awesome. I'm taking your bet now.

You know what? I was not aware of these benefits. Mark's been holding on to all the money and the massage for you. I thought you knew about the... I had no idea. Dynatrace, Dynatrace, Dynatrace. I'm just kidding.

It's a great place. It's awesome. Dinotrace, Dinotrace, Dinotrace. I'm just kidding. No, actually, it's a great place. It's good. It is. It's awesome. It's totally different to last year. Is your elevator shaft 15 minutes from here? No. Actually, mine is three minutes walking.

Okay. So you got one of the... So I had one of the fruit rooms. Andy brought us some Austrian whiskey. Actually, I think maybe you brought it. Or Alois. No, Alois brought it. It was not me.

Yeah. But you know about this? It was pretty good. Affen? I enjoyed it. Affenteller. Affenteller. It's a new...

It's a young distiller. He's new to doing it. Yeah, totally. But it's good. It's apple. The first thing he said is it's apple. That's the first thing I smelled. Yeah.

Not Affenteller. Yeah. It's Affenteller. Not ophthalm teller. Yeah. But ophthalm. It's ophthalm teller. Actually, it would be monkey. Monkey cells. Monkey cells. Yeah, like the cells that you have.

Yeah. That's weird. Oh, monkey cells. Yeah. Monkey cells. Monkey cells. That would be a word translation. Yeah.

His battery died. His mic is dead. He's taken two seconds. So, Klaus, we've had... I mean, obviously, there's some other announcements tomorrow that we're not going to go into. But from the things announced, I mean, no one here is listening, but people at home, we'll still keep them in waiting.

But there was a lot announced today. Wow. Right? We had the management zones. We had the log analytics. Log analytics, the RUM KPIs, the replay. Yeah. I know what my favorite is.

What's your favorite of those? Oh. My personal favorite. It helps me a lot with customer interactions. Therefore, my personal favorite is the management zones, actually. A lot of people have been saying that. It's one of those core assessments. It's deceptively cool.

It sounds like, ah, boring. No, it's a disruptor. Total disruptor. It's really cool stuff. Yeah. It's one of those core functions that... Remember back in AppMon? They asked us, hey, how can we divide things better? How can we handle visibility better?

But we said back in AppMon, it was like you lose visibility by enabling it. And this with management zones, and that's why I love it so much, is you can limit visibility without losing the connectivity. Right. So somebody else still sees what's going on and still can figure out the root cause. And you still have access

to all of the traditional users, groups, roles, the things that you would have connected to Active Directory or whatever directory service. But now you have the ability to slice and dice very differently the entire. Plus, is there some efficiencies within the engine for all the query stuff? Because you're not quite doing full search queries, so you maybe have some better performance built into the stack internally, yeah?

Yeah. So I think that's a good, people will appreciate that, especially larger installations. Totally. They will appreciate that, especially larger installations. Totally. They will appreciate that. And look at our, the customers that are here. It's like insane. Insane. You know, I had a panel discussion.

Let me ask you, is there any customer that has 100,000 agents? Yeah. 100,000 agents, 200,000. Really? So we're, right now, if you think about a single customer with between, what is it, 150,000? 110,000. So the first one to cross 200,000 on one agent. Yep, they are on one agent.

Okay, there you go. So that's the largest one. So there you go. But that's two really really big installations yeah just just take a look at i had a panel discussion this afternoon yeah user about user analytics future yeah and i was organizing it and actually it was at the point in time when I was on stage, I realized we had there a KBC Bank, a MasterCard, a TIAA, a T-Mobile, a Sherwin-Williams, all of them. No, you were not on my panel. So sad.

Yeah. But I had all of them. I could have taught those guys something. This is like, they're just incredible. And they have ideas. Hey, we need this, we need that. This is the future where we are going. And actually, some of it really relates to the other two announcements that we had around real user monitoring and session replay. This is a total game changer for Dynatrace again because we haven't had that visibility.

And we have now a totally great data set. And one of the panelists, you know what they say? Business doesn't know what business doesn't know. We're sitting now on a wealth of data. We're on a wealth of epistemology. There's the known knowns. There's the known unknowns. And then there's the unknown unknowns.

Yes, it was unknown that Donald Rumsfeld was the first performance tester. Yeah. Yes. You know, I've got to say, replay is what's got to be hot. Let me put this to you, Brian. Yes. What about the unknown knowns? I can't tell you about those.

Right, see? I can't tell you about those right see so you bring up replay and that's the one that's got me most curious I mean I understand management zones tremendously important but in terms of excitement of wow what can we possibly what's the potential that we can do

with this? I think replay just looks like it's going to be a lot of fun to get my hands on, start playing with, start figuring out. Definitely has some direct competitors. Definitely in the testing space, but tea leaf operations. Oh, in fact, we were talking to, who did we talk to earlier, who was saying, I want to use replay on my Selenium tests so when one of my tests fails, I can go back and watch it.

Not even really as a monitor. That was Neotis. That was Heinrich Rexha. I want to have a visual record of what happened during the test so I can hand it off to the developers. Which is totally not even what I mean.

We can watch a browser-driven test. Yeah. Awesome. Total game changer. Also think about, hey, you are arguing with your marketing team

about tracking pixels. Wait a minute. And the impact. You argue with your marketing team? I think every person has it changed. I have an infinitely harmonious relationship.

Performance, marketing, we love one another. It's beautiful. It's like the age of Aquarius. I love it, too. They always call me when they want to run some promotion. Yeah, yeah. They always say, hey, I don't know much about computers, but you might want some more of those VIMs.

You need more VIMs? Is that what you need? Yeah, yeah, yeah. That's what they say to me. Yeah, yeah, yeah. Because I just put a four megabyte image on the homepage. Woo-hoo! Holy shit.

And that was performance gone. And this is actually, I had one. I added a default shopping cart to the landing page? Yeah, why not? Oh, nice. Actually, that probably was on the tech guys. They just said they said shopping cart, and they wanted the image of a shopping cart, not an actual shopping cart. And in some worst cases, like you click on that button, and a guy shows up at your front door with an actual shopping cart.

I thought you said the customer wanted a shopping cart. So we had Amazon send them a shopping cart to their house, like drops in a junk. Right? An actual shopping cart. Isn't that what that means? Exactly. I'm trying to understand these technology things. It's hard.

And that's what a lot of key performance indicators, key performance metrics I love that one here. The flexibility of that is just tremendously. I mentioned on main stage in PayPal we're working in this new agile to durable teams. Very entrepreneurial independent, small, autonomous

teams. Fits with, you can choose DevOps. If you have a durable team, you want to do waterfall, whatever, you have more autonomy, right? Some of them are still going to be held accountable to define what's my KPI. And from a business value perspective, they're still going to have to, how are you going to measure this feature, this thing you're building, in very small ways? If you just came to them and said,

you can only really do the key performance metric based on two things for AppDex. Now, you have a durable team, and we can speak this KPI language more specifically to how are you measuring the

tolerating, frustrated types of things in that model or other models to be able to measure this more specifically for the business. So to me, that made perfect sense even in my world. Thanks. What do you think about the SaaS vendor announcement? The SaaS vendor stuff?

Yeah. Did you miss that one? Oh, the browser plugin. Everybody going into the cloud, even with all their serial systems and whatever. I think this is a game changer. Is it? You've got to get on the mic. So many

different companies are moving from metal to platform as a service on to software as a service and they lack the visibility under SaaS that they had previously under other solutions, the only thing that they have available, absent some sort of browser plug-in, is up-down. And it also forces their relationship with their SaaS provider to be one-dimensional, up or down.

Now that I have performance metrics, I can shape that relationship with the SaaS vendor to say, not only are you going to give me availability, but you're going to give me performance on specific pages which are important to my business. And previously, I've examined this question. The only other path was to go through a web proxy, pull the logs for the W3C time taken, and collect that data. It's a lot simpler to have a deployable package to be installed in every browser that's in the enterprise with a particular domain under examination and then be able to collect all of those statistics. And it's going to provide additional insight.

For instance, let's say Birmingham, Alabama is always slow for whatever reason that you couldn't see otherwise. Totally. Totally. So here's my thought. Lode Runner product manager, former Lode Runner product manager. Former Lode Runner product manager. So we figured out how to take things at the transport level protocols, right?

And you know our licensing model had this you sell support. Even though it was all maybe TCP connections, we didn't have the TCP virtual user. We had Tuxedo, FTP, SMTP, HTTP, 16 other different versions of branded protocols, and we sold the kits, whatever your app needed. Here's my question. You guys are licensing this as part of a UEM or ROM. It's the SaaS plug-in, and you get out of the box all this list of SaaS vendors. Why didn't you make this massive money-making idea of you have to buy a license for Office

365? You have to buy a license for Salesforce. You have to buy a license for... It would be totally against our core principles. Make it easy for our customers. For everyone. For everyone to get the data, to consume the data. This is it.

That's why actually the whole real user monitoring part is covered with this single session count. A visit is a visit is a visit. Actually, maybe I have to correct you. This is marketing term language. It's totally wrong. It's a session is a session is a session

is a session. Whether it's a mobile, it's a session, it's a session, it's a session. Whether it's a mobile, it's from Alexa, it's from the SaaS vendor, or from your traditional web, it doesn't matter. That's where it's coming from. So you're right. And you're putting an emphasis from those principles on the practitioner who one day says, I've been supporting everyone on Office 365 and now they're telling me,

stop that, move to Google. Doesn't matter. Go ahead. Go to Salesforce. Go ahead. You still have the same data. Go ahead. Just move it. You'll be one less barrier but actually more of an enabler. So here's the performance my users enjoyed on

SaaSPod Vendor A and we're going to do the same thing with their competitor and we So here's the performance my users enjoyed on SaaS platform vendor A, and we're going to do the same thing with their competitor, and we have the agility to move there because it's a better deal or because the users want that. Whatever. Go for it. Or you have durable teams, so new durable team spins up. They want to be a Google thing. These guys are Amazon and Office 365. Those guys are Azure.

You don't have to say, oh, well, I'm sorry, you can't do your work because you need a license for that protocol. Sorry, that's the load runner model. So I just thought of a very meta use for it. When you buy it, you go ahead and monitor your usage of your tenant, feeding it back into your own tenant, feeding it back into the data you consume,eding it back into your own tenant. Feeding it back into the data you consume. Feeding it back into your own tenant.

Which creates a paradox in the universe that destroys all matter. So I have a question about the SaaS app. He didn't get that. Okay, let's look at this from a security perspective. Let's say I have a negative list. I have a list of domains which are allowed, but I want to understand the user experience every time my user community goes to something that's not official.

Let's say they're cruising, I don't know, a Microsoft site and you're a Java shop. You want to understand what, hey, someone's going to Microsoft and what their user experience is. Now, they might be going to something more offensive than a competing vendor. They might be going to an adult website, not that that ever happens inside of a corporate enterprise. Could you use this from a security perspective and just say blacklist, whitelist, and when we have a blacklisted site, we actually just go and collect metrics from it? Currently, it's a whitelist thing.

So you name it, you get it. But it's a good idea to come up with blacklists, kind of monitoring as well. It would help you in a security area. It's the Russians. The Russian websites? It's definitely a Russian issue. Okay. Don't you think?

No, I don't think so. Now the NSA is listening to our podcast, which, hey, guys, woo-hoo, do you want to learn about performance? Actually, I know the FBI has this fantastic storage array that's really, really cool. They have some amazing storage big data capabilities. Yeah, actually. You ever go internally in the, like, the federal government? No, I'm not allowed.

I'm not a U.S. person. If we have done that, we're not allowed to stay on the air. Oh, but they have some really cool stuff. Actually, yeah, I was surprised when the whole leak came up. Yeah. Well, I was surprised. I was like, holy crap, we could learn something from them. Yeah.

I mean, they must have super-performing software and hardware. And they do sort of some super-performing stuff on sort of old-school architecture, right? Right. So, classic spinning disks. Those spinning tapes are going... Biometric matching. Yeah, but you can tell the federal government would be federal government you see those tape things running over there? Yeah. We need

150,000 times the throughput. And they're like, great, I got a barn in Nebraska and I'll just fill it with those machines. HP, how much you want for all that? Okay, here you go. A whole division was born and you can go find a barn in the middle of Nebraska that's filled with nothing but those tape machines but they are getting 3 gigabytes per second

out of that. Here we go. That's it. I made that up. I just want you to know. Absolutely, I don't know any secret. You heard it here first. No, you didn't, actually.

My name is not Mark Tomlinson. My name is Klaus. Klaus Pulley. Klaus Pulley. That's right. Yeah, just throw him off. You'll be fine. From South Austria.

South Austria. South Austria. Right? Is there such thing as South Austria? No. There is a South Austria. Central Austria. Actually, there is just about. They have spicy food in South Austria.

And everyone there has a draw and they play banjos. Yeah. Yeah. Totally. Totally. That's it. Do you know anyone from Austria that plays the banjo? Can you do accents?

No. Do you like a Southern accent? Like an American. A Southern American. You know do like a southern accent? A southern American? You know, you can talk like James Littleby. My name's James Pulley. Hi, my name is It doesn't. No, it doesn't work. So it's my accent is an

Austrian accent. I'm just curious if you could like break through, like do the Austrian accent doing the southern accent. So I'm just going to put on. No, I can only do the Arnie accent. Super strategic technology directions strategy, cloud strategy, tech guru. I'm just going to put the

mobile app UFO on the list. Mobile app UFO. Because I think it's kind of trapped in the format that it is. It is today, yeah. And just adoption wise, people people like, you know, I can experience. IoT is such a great thing these days.

Then you'll get an alert. Then you'll get an alert on your phone, which no one gets. I think it could go viral. Okay. And you would get more people on board. And then people who love the idea would start saying, let's get a real UFO. I would put in a vote for give me a mobile app experience that is the UFO.

You could make it light up. You could make it do cool stuff and deliver some of the same functionalities visually. So could you put this in the Apple App Store or the Google App Store for the Vufo, the virtual UFO? The virtual UFO. The Vufo. But the best

part would be is you could have it send you an alert. Yeah. Amongst all the other alerts you have. On change. Well, you won't be getting a lot of other alerts because you only get one for a problem. And you can select your notifications and maybe change the color scheme. Just change the color of your screen. It would be another way to bring people into the experience of the idea of it.

Vufo. Vufo. Add it to the list of predictions. Actually, it would be Mufo. See, when people wonder where new ideas for Dynatrace come, it's from sessions like this. We heard earlier there was like the hotel session. This was what it was.

It was the four of us standing around talking about what do we need and coming up with crazy ideas. This is exactly how this stuff happens. By the way, if people don't even know about the Dynatrace UFO at this point, they can go to just search GitHub. Is it GitHub? GitHub. Dynatrace UFO.

It's an extremely fun video if you go to YouTube and search Dynatrace UFO. Oh, yeah, yeah. Which is that awesome video. It's a UFO. Who's the one who's doing the video? Michael. Michael. Michael. Yes, give him a shout out. Who's the one who's doing the video? Michael. Michael. Michael. Yes, give him a shout out.

It's a great video. That's great. What else should we know? What's coming? Can't spoil. Tomorrow. Tomorrow there's big stuff coming. We're still broadcasting tomorrow, so we can talk about it.

We picked up the things this morning. We talked about log analytics. We talked about management zones. We talked about the browser this morning. We talked about log analytics. We talked about management zones. We talked about the browser plugin coolness. I kind of like the flexibility of the key performance metrics actually.

That's one of those ones where you're like, well, yeah, of course we have that. But then you're like... So I predict we're going to have a very large AI discussion tomorrow. I think we're going to have one tonight. Oh, at the whiskey attic. Because my intelligence is artificially predicting that.

I think maybe we are talking AI ops. Huh. I ops. I ops. I ops. It's already taken. I ops. I think it's AI in development.

I think you invade the static analysis space. Okay. So you're removing the ops totally? No, because there's no difference. Okay. We're pushing the question downstream. So we're pushing the pattern analysis downstream earlier in the process. We identify the pattern. We recommend the in the process, we identify the pattern,

we recommend the solution, it is a least-cost fix. Sounds great. Another term that I want to throw here in the round, APIs. What do you think about APIs? Useful, useless, what is it? First of all, it's been a battleground in the testing world for a long time. It was a differentiator between tools, versions of APIs, all that kind of stuff. And look at, what was it, SmartBear?

Really was ready API. They had a whole program, all the tools aligned with it. And I think a lot of people that fanaticize current REST API architectures, if you go back, here's a stupid static tunnel of an EDI transaction coming in some weird flat file. That's an API. I mean, it's

an application. It's another application. I tell you, give me this data. I mean, this is nothing new. I would love if the developers would actually hold it static long enough to test it for more than three cycles. Yeah. So you want to establish a baseline, basically. Well, you know, if I'm going to compare test to test, if the developer keeps changing all of the parameters,

how am I going to have a framework and performance to actually do this performance testing if everything changes in every build? I would like to treat an API like a child, which is an API is born. It has experience. It doesn't have experience. We have telemetry data on it over a certain period of time, or we don't. And so we're informing a rules engine or AI based on what we know about this new API. And then if that child, new child, has siblings, and maybe we have blended families or extended cousins, the dependency mapping piece comes in handy.

We already have a transaction flow from the old world, but we have all the fantastic mapping for the topologies now, right? So I think the next thing would be to understand trust of the child because as the API is new, I may not trust it to take on new workloads and then would predict, you know, I know

you want to run a batch job that's going to hit this new API, but my sources tell me, Davis tells me, we don't think this API is mature enough. We don't have enough information to know whether it will scale or whether it will crash. So I want to know a little bit of history on that child along with my view of its capability. So you want to know when the API becomes a teenager, right?

When I can start trusting it with the car keys. Are you trusting a teenager? I don't have a teenager. You don't have a teenager yet? I don't have either. Mine are three. But this is the interesting thing. When are you getting your license

to run then? What is it? Is it the driving license? Exactly. If this is a new API and it's new to production and we think somebody has big plans for it, are we asking too much of it too soon? Can happen.

That would be an interesting context. Totally. When you bring up APIs, are you talking about APIs that customers too much of it too soon. Can happen. Can happen. Totally, yeah. When you bring up APIs, are you talking about APIs that customers are building in, or are you talking about APIs within Dynatrace for interface? Throughout the term APIs. When I think of APIs, I think about interface APIs. So you just simply threw out the term

God and asked for a religious war. Right. So here's my thinking. So where my mind goes with APIs is operationalizing Dynatrace. And what my thinking goes through now is we have enterprise products.

I have no idea what's being announced tomorrow. So I'm not even trying to predict. Making it up. And we already have some of this to an extent. Having more and more extensible APIs to the product where it's a hybrid of an enterprise product with an open source outside community where all different kinds of interfaces are being built into it, managed and stored in GitHub by users, shared and built, all these additional pieces that can be made. An example, Davis and GitHub. Exactly.

Davis is an API. Exactly. Davis is leveraging. Exactly. You know, even something as simple as some of the JMX monitors I'm creating for Hybris, I'm throwing those up in GitHub. And, hey, you guys have better ideas for some of the JMX metrics that will be better for this application? Go for it. Let's do it. But in the meanwhile, we have the rock-solid enterprise application being built by the Dynatrace team

that's backing it all. Not necessarily supporting those API creations, but having almost full visibility of the data. And as we were talking with Heinrich earlier, so Heinrich

from Neotis, they just put out a new plugin where they're going to push their data from their load testing data into Dynatrace. So you're analyzing and all that. So the idea there is we don't care what tool you're using to consume the data as long as you see it all together. It's this openness of data sharing. You have your tools. They have specific purposes.

And it lets everyone get all the data together to be better. That's awesome. Thank you for the whole session. I'll leave you hanging with the information. We're talking about APIs tomorrow. Okay. We can't know about it yet. You can't know about it, but there will be something coming.

The tease is what you're suggesting. That's the teaser. That's why I threw it out. So after you come off stage tomorrow. No, I won't be on stage. You won't be on stage? So after the tease is satisfied, we expect you back here

to tell us what the heck you really meant. No, but you'll go to, before that, perform.dynatrace.com to see the live stream so you can watch whatever it is you're talking about happen first thing tomorrow morning. I think probably. Maybe. This is a good question.

Actually, I couldn't follow the main stage changes. Maybe later in the day. Maybe in the morning. No one can tell. No one can know. What's going to happen is a very large UFO is going to come down. Davis, can you tell me when this announcement is going to be? Burn is going to come down out of the UFO with flashing lights and smoke machines and everything and be an interpretive dance.

You know it's Vegas. Everything can happen in Vegas. He's going to do an interpretive dance of the API. Without lederhosen. Feature. Right? Yeah. I think I nailed it.

I like it. Yeah, you nailed it. Totally. Perfect. So everybody's nailed it. I like it. Yeah, you nailed it totally. Perfect. So everybody's spoiled now. Thank you very much. They're going to be wearing the Intel full body suit. No product placement. Yeah, the weird hip hop

dancing. So thank you for joining us. And I will say, you mentioned your daughter, I think? No, I have two sons. Two sons, but one has just turned three. Just turned three. The other one is four months. Because you were a new dad at the first perform when we first met you. Right? Or it was happening.

Yeah. It was almost. Yeah, yeah, yeah, yeah, yeah. A little more than three years ago. Yeah, yeah, that's true. Yeah. Wow. Yeah.

Now I'm feeling old. There you go. Yeah. I just turned 44 yesterday, bud. He's turning 50. Thanks. Thanks. Thanks. Thanks, guys. Now I'm feeling old. There you go. Yeah. I just turned 44 yesterday, bud. He's turning 50. Thanks. Thanks. Thanks, guys.

Now I'm feeling much younger. Yes. Hey, thank you. And Marcus, it's totally appreciated. It's been a pleasure. And the race is on. Yes. Thanks, Brian.

See you tomorrow. Who gets on the podcast more often. I'm going for two for. Three for. Four for. I'm going to have to reevaluate. Reevaluate. Reevaluate. Oh, you're fine.

All right, Klaus. Who else should we talk to? I'm going to go talk to the bathroom. All right. Thanks for sharing that with the audience. The kids at home. If you're really interested in that. Oh, it's fine.

We're good. We had a few stories. Yes, that were good. Other customers came and shared. Yeah, I noticed we have far fewer speakers, as in like Bluetooth speakers. Yes. To give away. I was trying to remember all the different people who came to give a story.

I think one was Harish, wasn't it? Harish was one. We should have written them down. I know. So Brian Brimsfeld came, right? Yes. Yes, he did. He did that really rambling story where he got a stereo pair.

Yeah, and Jeff Hoffman was there, and then who else? Here's Jeff Leonard. Did you want to share a performance story with us and take home a Bluetooth speaker? I'll be as generic as I can. Did you want to share a performance story with us and take home a Bluetooth speaker? You can make it as generic because all we're really interested in is the weird, crazy performance architecture story. All right. And keep in mind, this podcast does not come with subtitles, despite the fact that there are two people from North Carolina on the podcast. It keeps happening.

So I should speak very clearly is what you're saying. Yes. In a very deep voice. Very clearly and slowly so people outside of the South will be able to understand you. Oh man, now I'm really in trouble. So what is your story?

Let's see. I did have one a bit back where we had something where something just creeped up a small percentage, and we showed that that can back up everything at the mid-tier level. Yeah. And they had a defect for a while with that, and we had them set with that, but they needed to better. The overall architecture, just a three-tier web kind of thing, web app database? It was off a service bus.

Okay. Yeah, so a service bus to back-end kind of thing, web app database? It was off a service bus. Okay. Yeah, so a service bus to back-end kind of thing, right? So then what was happening with the back-end with that, it would just creep just a small percentage of milliseconds. So you had a bunch of serialized requests. No, you're not supposed to guess. He's supposed to. Okay.

No, no. I'm dropping back. We're supposed to be like, let him get the story out. I'll put my governor on. You don't want to shut him down. It's all good. And by the way, that was my southern part. It's all good.

So basically, so what was happening was their mid-tier was having issues, and they were thinking it was purely at the back-end level. But what happened was the mid-tier was so busy because it was waiting on the previous request to come back, and there was really nothing looking like it was slow. And if you really went underneath the microscope with it and really looked at it, it just creeped about 20 milliseconds. And the amount of requests it was making was so high that it actually backed up all the connection pools.

And then because it would back up everything in the back end, your front end, well, not on the database side, on the side of the mid-tier going to the database. Okay, so incoming to the app server itself. Yes. Okay. Max threads? Yeah, well, no, it was odd.

So basically think you're coming into like a front end of an application server. An application server is going to Service Plus. Service Plus is going back into a back end system, right? So then what's happened is when it's in that mid, you know, the subscriber publisher part on the SOA, it's stuck in that level there, and it's making that level look bad coming in and going out. So where are they pointing the finger when you came in uh they're pointing the finger at the app server the es the bus yes both both and i'm like no that's not it and they're like uh no what you got a problem here

is this is increased by 100 uh probably about 20 milliseconds it's only about 100 milliseconds or 80. And so once you get in this level here, it's backing up all your connections, bopping in there, and they're all held hostage. And you're tuned for an ideal tuning. Solution to the problem? Solution to the problem is pad some additional,

even though you're not using it all, for moments when that gets inundated with something bad that's going on in that environment that they don't know about. Transient spike, something. Exactly. Since they didn't have anything to thwack, something that might be a rogue process, they would consume everything and slow it down just slightly. What was the downside for the customer? Was it just slowness, or were they losing business? What was, like, in the greater scheme of things?

No loss business. It just made things really slow. People are just waiting around. So it was a loss of productivity. No loss business. It just made things really slow. People are just waiting around. So it's a loss of productivity. Yeah, exactly. Like your front end would just become miserable because it had so much waiting going on. So you'd end up getting, nothing would return

or things would spin. Things of that sort. Because it was out of connections. Now if you're in there and the connections that were getting served, you're great. Last one in, orphaned. Exactly. Because it couldn't hack anyone. You know, it's too busy.

Do fiends. Why didn't they find this? So you just waited the ESP. Exactly. Yeah. So why couldn't they find this? Well, the thing is pretty complicated. It seems kind of straightforward.

Well, it would be if you think it was just the way I explained it, but it was dozens of different things, of different applications, consuming a common bus. Exhausting the bus as they shared different services, but it didn't matter. It's part of the infrastructure. Yeah, so it looked like the calling app server was the problem, but the calling app server wasn't the problem. It was two tiers down. Ah.

Yeah. That's a good one. Oh, believe me. It was two tiers down. Ah. Yeah. That's a good one. Oh, believe me. It was pretty hairy, and no one believed me for quite a while. You haven't mentioned the customer at all? No. So it really was an all-publisher problem, not a subscriber problem. Exactly.

Yeah. And the last thing you would think, it would be that, because it was only a 20-millisecond difference. Yeah. Now, is this JMS? Or what was the underlying ESB architecture? You know, being old, yeah, it was JMS. It was messaging. Yes, you got JMS and maybe some kind of Apache MQ stuff.

Yeah, well, exactly. All the above kind of thing, right? I mean, Oracle bought up some of that technology that's in it. I'll just give you that much of a hint. Well, one of the things I found is that people would grow an ESB and they would use heterogeneous different things. I got a little

RabbitMQ over here, a little ApacheMQ, a little WebSphereMQ, some old MQ series running some Oracle queues on that thing. And they think, queue's a queue. I can just plug this stuff together, right? But all of the underlying protocols that are connecting these things have preferences, have defaults that

aren't exactly... I really like talking to another one of me. I don't like talking to my competitor, another vendor, or whatever. So all those esoteric, like, oh, I'm talking to... I have to do three round trips to do the same thing as if I'm talking to your API.

And that's stuff that people can really whack to the bus. And this one's even crazier because this is IBM kind of crap. I'm sorry. It's not crap. It's just an IBM kind of thing that when they do their gets, it's an incremental get. So it could be a thousand things to get the one thing you want. Yep. And that's what it was.

And so then when you add 20 milliseconds on top of an incremental get, that really adds up. You're getting 20 milliseconds added. To 1,000 requests. Which is almost like an N plus 1 problem, right? Exactly. And therein lies the problem. So I forgot to give you that extra piece there. That's the most important piece.

That's very, very cool. All right, Jeff, you've won yourself. We've entered you in our long-term drawing here to win Dynatrace UFO and an Amazon Echo. You also won the Bluetooth sound speaker. We only have two left. Awesome. After this, I don't know what we're going to give away. We'll have to give away

Neotis t-shirts. Did you get a beer koozie? I have plenty of koozie, but we'll put this in there right now. You can't see this, but he's got the old school. That's not going to work. Think about Thanksgiving. You need a good half a dozen of these things.

You know the funny thing tonight? I'm in the South. I have plenty of these. With all the beers tonight, the koozie's been tremendously popular. People are like, I've got a beer. I've got a koozie. Oh, who needs a beer koozie? He needs one.

Time and place. Yeah, we'll give it. Jeff, thank you. That's an awesome story, though. All right. I appreciate it, guys. The thing that's deceiving, it sounds common. Oh, I know this. If you've been any season performing, then it snakes thing that's deceiving, it sounds common. Oh, I know this.

If you've been any season performing, then it snakes away. You're like, wait a minute, that's not the thing I thought it was. It's the incremental get because it could be 1,000 or 10,000. And when you add 10 to 20 milliseconds on top of that incremental get, it eats your lunch. Next thing you know, you're sitting there for three, six, eight seconds. And when you look at a normal graph, the little tiny graph, no one's going to see that. Oh, yeah. No one's going to see it.

Awesome. Yeah. All right, cool. Thanks, gentlemen. What's your favorite thing here so far at Diner Trace Perform? The hands-on. Hands-on. Yeah, which?

Yeah, actually, well, since I was kind of new to some of the new front end with all this, we kind of wanted to do the entry level, and that was all sold out. So I'm like, all right, so I'll pop in the mobile one, right? Yeah. I'm like, yeah, I don't do any mobile, but I'm sure it'll apply because it's just an end client, and I've done ADV before. Right. So that wasn't bad.

That was great. Got a lot from that, and that applies to actually all the other areas. Yeah. So and then did advanced web monitoring, too. That was great. Yeah, yeah. So the hands-on was worth time. Which is advanced web monitoring on Dynatrace, not AppNone.

Not at all. Have you used the old AppNone stuff? No, I have not. No, so you're really in super- Yeah. Oh, yeah. I got to dive in the deep end day one. That was fun.

That's pretty nice. That's kind of rare, actually. There's still a lot of folks like me. I'm still an Appmon user just trying to get things into the Dynatrace world. So that's good. Yeah, it was handy. Of course, now I got in kind of maybe early evening and had to finish downloading everything. I'm a wire shark guy, so.

Awesome. Thank you, sir. Thanks, guys. We'll see you again. Bluetooth speaker. Bluetooth speaker and a cozy. So, do you think it's... What time is it? We've been going for... I think it's maybe time to wrap up. Actually, is it close to 7?

It might be good. We'll give a little wrap up for the day here. Yeah, two minutes before 7. Yeah, yeah. So. Yeah, all right. All right, day one. Day one. We started in the morning.

We posted the Chris Morgan interview for Red Hat. He did a main stage. That was good. Yep. We covered our four massive things first thing in the morning. We did. We gave away pretty much all of our Bluetooth speakers. Except one.

There's one left. I might take that one home myself. That's probably not fair. We talked to Heinrich. Talked to Heinrich. Got that on the air. Got Chris on the air. We actually did some conversation with the AWS guys.

So there may be an interview coming down the road. Maybe I'll let you guys post that. Not the AWS guys, just Steve Pace. Just Steve Pace, but, you know, we'll see. That'll be kind of cool. Just got to finish that. But that may come posting down the road. While we didn't speak with Brian Folk of Folk Consulting, he has some announcements to make in about a week.

Did he come, or maybe talk to him tomorrow? He has some things that didn't quite come together in time for the event. So, you know, we'll post it to the Perform 2018 tag as soon as it's available about a week's time. That sounds fantastic. Yeah, I'm tired. I'm tired, too. I need some sleep. Yeah.

But we're going to sign off for the day, the end of day two. James, any thoughts, reflections, anything piquing your interest? Yeah, the browser add-in, I think that's a huge. That's a big thing. Very, very subtly tossed in, kind of under the covers as an announcement. But I think that's extraordinarily large for organizations that are moving to 100% SaaS model. And they need to have something other than a black box up down. Right.

Okay. Oh, and don't forget, though, right? We talk about if you think black box up down. No, we're wrapping up. What's your thoughts for today? I was just going to say there's always synthetics. My thoughts of the day is it's been a long day.

A lot of information. I wouldn't say overload because it's been well-paced. No, no, it's perfect. But it's been a lot of fun. So we had a hot day yesterday. A lot of fun. A lot of cool stuff today. I'm really looking forward to the announcements tomorrow. Yeah. We didn't talk enough about Chewbacca

and the Stormtroopers. And Darth Vader. Right. And Dave,'t talk enough about Chewbacca and the Stormtroopers. We did not. And Darth Vader. Right. And Dave, who was like a Jedi. He was a. Dave the Jedi. Ozzy One Kenobi.

That's not. Ozzy One Kenobi. Oh, that's not. Ozzy One Kenobi. I missed that. I'm so glad. I'd have to tell him that. That's right up there with 84 Lumber.

84 Lumber. Just to mention it. 84 Lumber. And Bob Stoltzman. I mean, guys, you can't not remember Bob. What about Bob? I don't know. So we're going to sign off for the day, everybody.

Join us tomorrow. Again, same time in the morning. Same time, same channel. Right after breakfast. Maybe we'll see how we're doing in the morning breakfast-wise. And then a couple of different segments, some more interviews, some more stories. And then we'll wrap it up. There's even another party tomorrow night, I think. Yeah, there's a band playing.

I don't know if we're going to be podcasting. I'm going to get on stage. You and I are going to jam in the breaks, right? Oh, yeah. All right, cool. Yes. I'm good with that. I've been practicing my beatbox.

Now we're going to go have a conversation with Quincy. You want to? No, no, no. Okay. We're going to wrap it up. The podcast that never ends. Thank you so much, everybody. We're signing off. See you tomorrow. Live from Las Vegas.