PurePerformance - Dynatrace PERFORM 2018 Tuesday Lunch

Episode Date: January 30, 2018

Live Lunch with Perf Bytes and PurePerformance

I thought you were going to do some intro music. No, you're going live too. I thought you were going to do some intro music. What happened? We're back for lunch. No, I skipped the intro music. It's not fun. I guess I'll stand up too. Frederick, how are you? I'm doing good. Fred, have you ever done a podcasting before?

First time. First time podcaster. First time. First time caller east of the Rockies. What would you like to share with us? Exactly. So you have a performance story for us. I do. And we're at lunch. Did you have lunch? I did have lunch.

What did you have? Just the salad? A little of everything. A little of everything. Same here. Risotto? You tried the tortellini. Complaints on the tortellini.

No, I didn't get the tortellini because they were completely out of tortellini. That's insane. So everyone was waiting. I was like, oh, I guess I'll just... You never have enough tortellini. The risotto wasn't bad. All right. I enjoyed it.

So just to set the scene, we're in a giant ballroom with all kinds of people around us that are eating food. But you thought, I want to share my performance story so that I can get a cool Bluetooth speaker. Well, you were first coming to look for what swag we were giving away, right? Yeah, you got me with the cozies. Yeah. Yeah. And then we roped you in with the tell us a story. All right.

So what performance devilry have you seen today? You said it was today you've seen something great. Right. Standard day, open your inbox, check your emails, and then you start seeing an email thread and you look at it. So that's the first mistake. Yeah, first mistake. From Vegas, first mistake.

Cool. So bottom line was is that one of our developers just pushed out new code using PCF, pushed it up to the prod region, and they started to test. And they started getting errors. Right. So they have a hook into one of our third-party services that was already on the cloud and already in production. So they started asking, why am I getting this error? Nobody knew.

So then we started getting our architect to look at it, and he looked into logs, and then other people looked into logs, and then somebody eventually found out one of our certificates expired with the vendor. So we don't know how long we've been in prod not calling the vendor because our certificate expired and apparently it didn't stop our processes from continuing they just kept right on going like whoops oh well right so then the first question i got from my architect was why didn't we know about this my response was apparently we didn't have alert in place for that yeah and apparently your code didn't throw an exception in the log either. Right. Yeah.

So we have a few holes to plug up. But being here at the Dynatrace conference and seeing the new version that they have with the one agent, you deploy it, it detects those anomalies, it looks in all the logs. Right. My first thought was that could have probably found it right away and opened up a problem ticket. Yeah, if you're making that third-party call and that's coming back.

Right, because you would have saw some kind of error in the log and it would have saw that. So log analytics would have helped this new kind of idea? Problem ticket would have helped. Or even the basic log alerting. Or even just the basic transaction might have seen a difference in the response from that and said, hey, your response used to take 500 milliseconds and now it's

taking one millisecond. And this is actually part of our critical path, so there should have been a business transaction monitor on this. For an exception, where you could have, the response time would be really fast. If it's like, I don't have a cert, goodbye.

So it could show that, hey, response time's great. You're just failing. Yeah. Right. Right? Oh, that's a good one. What was the actual cost? Like the downside to this? Did you have to go back and fix data?

Or you've been paying the vendor and they're not getting any calls and they didn't call you up? Yeah. That's another good point. The vendor never called us up and said, hey, no traffic's coming through. But here's another invoice for the month. Yeah. That's the way it works. Exactly.

They're like, oh, yeah, well, I guess we did. Nobody's looking. That's the problem. Right. Exactly. Nobody was looking. But, yeah. And I know where some of our gaps was because we haven't gone to the new version of one

agent. We're using Atman. Right. So we have Atman on all of our Liberty IIS boxes, but we haven't really put it into our PCF areas. Okay. So this came out of our PCF region, which is why we never saw it. Right. But, you know, classic performance example.

Something's happening in prod. We didn't fall over. The end user didn't get any hard error. Yeah. But it was part of our critical process. Nobody knew about it. Yeah. So now we got some backtracking to do to figure out how long is this.

And it's part of our pricing process. So now we got to go back and we got to see, okay, how many pricing transactions do we run through without this third party? Without the check. Right. And how many told people they can get it for really cheap? Oh. Yeah. So there is.

Lucky you. There could be a revenue downside. It's free for you today. Lost revenue, yeah. But those are the exact questions my execs are going to ask me. You bet. Yeah. Wow.

And you're in Vegas. So tonight you'll be sitting up in your room working all this out, I imagine. I have people back on the East Coast starting that. Yeah, I was going to say. Right now. You should have people doing that for you. Good. Yes.

So East Coast-wise, where are you from? I'm in Maryland. I'm in Philadelphia. Okay. So I actually work with some of the PayPal teams down in Timonium. Okay. That's where we're at. We're right across the street from them.

Okay. Awesome. Great. I'm just glad we could travel to Las Vegas and both be jet lagged just a little bit. Exactly. It's snowing back in Maryland. It's the perfect time. Really? Yeah. There's fling back in Maryland. It's the perfect time. Really?

There's flurries and it's happening. I used to live in New Jersey and East Coast snow comes and dumps. It stays around. It stays cold all summer. You have the dirt black snow that's been getting all the exhaust absorption and everything. Four years ago, I moved to Denver and everyone was like, oh, what are the winters like?

What are the winters like in Denver? It must be really cold and snowy. I'm like, well, today it was 80 degrees in February 2nd. And, well, tomorrow it's going to be negative 3. But three days after that, it's going to be in the 60s. Oh, and by the way, we had snow yesterday, but the sun came out and melted it all in the afternoon. That sounds like Maryland. Yeah. Wait five minutes, the weather changes.

My rule of thumb is the mountains make their own weather. They do. That's actually really crazy. That's always good to know. When you're looking at the ski resorts, anyone, you can't just look at mountain weather. You have to look at the weather for the exact resort because they're all different. I think we need a Dynatrace dashboard for this. We do. No, forget the dashboard. I need an AI alerting

on weather unpredictability. So let me ask you, Fred, of the things you saw this morning outside of the issues that you're dealing with currently, from the things you saw this morning, the new announcements, obviously I know you're not using Dynatrace yet, but did any of those new features

sound really interesting? Oh, many of those. The AI, the anomaly track. Well, that's exciting. So the newer ones, meaning, well, that's all part of Dynatrace. And, yes, it is very exciting. I'm talking about things like the replay.

I guess management zones wouldn't make too much sense because you're not using Dynatrace, so you don't know how that fits in. But would replay be helpful? Or maybe, as Mark was saying, the different key performance indicators, do you use things like time to first byte or something else besides some of the standards? We do have metrics for all those as well. Yeah. So we have agents all across the country, different regions, different size transactions.

So performance is really hard for us because one policy with one vehicle is going to go fast. One policy with 100 vehicles is going to go fast. One policy with 100 vehicles is going to go slow. Right. Right. But our biggest thing is, can the user experience go through first time without any delays, errors, the angry clicks because it's spinning? We have that problem now where if it clocks too long, users will kill their browser session, start up another one, hit the same account, lock themselves out, step on their own toes.

Yep. So. Too much freedom there. Right. Too much. So taking a step back then to you, where you were going to go before I cut you off and tried to steer you in another direction was you're at Atman now, which a lot of our customers are still using Atman quite a lot. And they're starting to say, hey, we've been hearing a lot about this Dynatrace. You were kind of talking about a lot of the things about just Dynatrace itself

that were interesting to you. You mentioned the AI, which is obviously a really, really cool piece. What else about, you know, obviously making a switch in any kind of tools, a little bit of heavy lifting, you got to learn some new ways and processes. What about Dynatrace is making you say, say hey i think maybe we do want to make that switch you were kind of going there before what was the second part just continuing where you were going before as in terms of what about dynatrace is making you think maybe we do make a switch over from atmon to dynatrace to full dynatrace well that was one of the reasons I came out here is because right now we have a lot of Atman,

and we're doing a lot of microservice work. We're moving a lot of things to the cloud, and that's been our gap for coverage because we just don't have it configured there. And at the rate that we're creating new microservices, the licensing for the Atman is going to explode for us. Then I've got a small team right now, and I'm going to need a large team

to manually go in and configure Atman for all those different microservices. Right. That becomes a headache pretty quickly. With 200 plus developers constantly putting new change in, I don't have enough resources. So I came here looking for Atman to figure out how we could use it better,

not really actually realizing that one agent was on its way. But then hearing all of that addressed all of my concerns that I had with Atman. Yeah. The nice thing with PCF as well is you can not only monitor your applications in PCF, but you can also monitor your PCF infrastructure. So for the people whose job is to maintain and keep the lights on your PCF cluster, you can throw it on all those components and be looking at the healthier Diego cells, all your other different...

There's so many technologies, it's so hard to remember all the components between PCF, this technology, that. But it's just looking at that as the infrastructure, too. So that's also fully set up. So for your performance story endeavor, you have got yourself a sound cube. Thank you. A Bluetooth speaker sound cube. It looks like it would be mono.

It's not mono. It's stereo. Nice. There's two speakers inside? Who knew? But, yeah, thank you very much. Enjoy the rest of the conference. Should we find other people who want to give a story?

Sure. Let's go ahead and do that. Thank you very much. Should we walk around? Yeah. Awesome. James is just hanging out over here. James, you want to man the booth while we go take a wander? Yeah. No, we're not going to wander. No, we're not going to wander.

Oh, we're not going to wander. Okay. No, no, we tried that, and it got weird. It did get weird. It did get weird. I'm going to have to edit that show. That's really... It didn't work.

Mostly because people talk like this. You know, I should write a show. Hey. Which is really annoying if you're on the other end of a live broadcast yeah this part of the show is good hold on yeah back and forth I'm coming to you in stereo now

two mics what kind of audio effect do you think this is going to be there are people throwing up people are going to throw up now that's really bad they're getting car sick Mr. Wilson if you're listening to this live you will not hear this People are going to throw up now. That's really bad. Audio acrobatics. They're getting carsick, Mr. Wilson.

Yes. If you're listening to this live, you will not hear this. Here's a guy who wants to give us a performance story. You're walking over. You're looking. A Bluetooth SoundCube speaker. Yes. Really?

Can you talk to us about JIRA performance? What's that? Can you talk about performance at JHA? Sure. How's it going? Do you have any horrible terror stories of things that went horribly wrong? At JHA or somewhere else? Well, anywhere.

Put the mic right up there. Don't be afraid. Yeah, we've had a lot of crazy performance stuff that's happened. Any memorable story? Or someone died, maybe? Does that have to be in production? Dying in production. Or halfway?

Yeah, someone got killed because of a performance issue. I tested an application at a client where I was supposed to be in a performance environment, but I ended up in a production environment

ordering medication for real clients and real hospitals. Did you really? Luckily, I was a smart aleck, I guess, since I probably don't want to cuss on here.

And I was ordering Vicodin and stuff in kilograms instead of milligrams, so someone caught it. So that was a pretty interesting one I've had. Serious. And it was like printing out somewhere in the pharmacy? Inside the pharmacy, one of the people caught it, and they kicked it back to the doctor asking if they meant milligrams. And then once she went to the next order and seen it was another one with kilograms, they decided something was wrong. Wow. That, I think, is one of the best stories I've heard so far.

I almost want to give you two of these. They're right there. That's insane. Another time, I was working at a bank, and we were doing a data center recovery. We were moving over from one environment to another using our production backup. Right. And we got sign off from all of our managers that there's no way we're going to hit production. And I started a foreign exchange script and sent $70 million to China that we didn't recover

all of. Ooh. Well, you know, they do talk about like modern currencies and economies. It's probably just a wash. $70 million is a wash. No big deal. Yeah, you know modern currencies and economies. It's probably just a wash. 70 million, it's a wash. No big deal. Yeah, you know. That's fine. It's a real estate investment, chump change, maybe some secrets.

Not only do they have our IP, now they have another 70 million in cash. And did you keep your job after that? I did, but the development manager over that area that signed off was no longer there a week later. See, that's what you'd expect to happen. Right. But you survived. Yeah, me and my whole team survived. It was a little bit scary when you first heard what happened.

So there's two pretty catastrophic things that you survived. Yeah. So perhaps he could be working for the NSA. He could, but the thing is, with the two catastrophics, before you were asking about a death or a dismemberment, I think you're a step closer now to being part of one of those events in production. Maybe a server will fall out of the cloud and land on somebody. Have you ever blown anything up, like actually melted equipment?

That's a good one. You can get actual smoke to set off the halon. That's fun. That would be really awesome. People could die because they tell you when that thing goes off, get out of the room because there's no oxygen. Or just hold your breath because it only has to last like 15 seconds. If that happened, I doubt my key card would let me out. Oxygen. Yeah. Or just hold your breath because it only has to last like 15 seconds.

If that happened, I doubt my key card would let me out. They're not supposed to have a key card on the inside to get out. Isn't that right? I mean, they can lock you in, but you. I've had ones where you had to have key cards on both sides. That's very scary. All right. Congratulations.

You've won yourself a stereo sound cube. We only have, what, five of them? Do you want any cozies? Yeah, we have lots of beer cozies. All right. Congratulations. You've won yourself a stereo sound cube. We only have, what, five of them? And do you want any cozies? Yeah, we have lots of beer cozies. Take some cozies. Take 100 of them. There. Oh, no. You've been denied.

All right. Thanks a lot, Mark. Thank you, and enjoy the show. Thank you very much. That was ridiculous. That was marvelous. That was lunchtime in Las Vegas. And you know the story with the Vicodin, right?

In kilograms. But I can totally see that. This is why... When you're doing tests... He's caused the opioid crisis. Right. When you're running tests... Single-handedly.

You're always having fun, like, oh, what am I going to put into these fields? What am I going to order? Whatever, right? You always do something fun because that's the only way you can entertain yourself while you're doing these things. Yeah. So you're like, oh, yeah, let me do Vicodin. Oh, let me order the kilograms. You always do something fun because that's the only way you can entertain yourself while you're doing these things. So you're like, oh yeah, let me do Vicodin. Oh, let me order the kilograms.

That'll be fun. But the system allows you to order Vicodin and kilograms. But it was also production. It actually took, there was no edit on the GUI to stop you from ordering two tons. Are you sure you want to do that, Dave? If I've got 600 kilograms. Yes.

Quite amazing. Close the doors, Hal. Playing doors on Spotify. I love that. Oh, oh, oh. I think we're going to have to take a wrap up this session. No, because I have to go on the main stage. I can't start now.

You can. I have to wait. No, no. You have to warm up. No, no. Loosen up a little. No, I'll just, because I'll just talk too much. I'm a pod. No, no. You have to warm up. No, no. Loosen up a little. No, I'll just, because I'll just talk too much.

I'm a podcaster. Don't get me started. I really only have two main answers I have to give, so that'll be all right. All right. All right. Anyway, lunches, we're coming to a close. We're going into the second half of day two. Well, really, day one of the main conference. Because that was day zero.

So it's kind of like an array. Yesterday was day zero. Today is day one. Tomorrow is day two, but there's three days. You have a session coming up soon. 2.30, right? 2.30, Andy Grabner. So listen, if you are listening live, 2.30 session, Andy

Grabner. Perform.Dynatrace.com Yeah, and he's going to be doing an awesome thing about doing a lot of CI pre-production work with Dynatrace, which is really, really critical because we had so much of that with Atmon, and he's been working his butt off and dancing his butt off, I should say. Well, and here's the thing. He's got it nailed.

You see his slides with the big circles? Yeah, they're yours. And then you come to my session, and you see the original circles. Yeah, they're yours. And then you come to my session and you see the original circles. Now they're Andy's circles. Yes, well, you know what? Andy's done some great work with our CICD stuff, implementations. No charge for me.

No charge for me. I just give it away for free, like the podcast. Exactly. Just give it away. If you love something, set it free. If you're a long-time listener of the show, one thing I've noticed is... Just let it be free.

Mark does like to take credit for a lot of things, doesn't he? Oh, you never know. You never know. I've got connections. Yes. Maybe in the Musad, but I would have to kill you. You would? Yeah.

Of course, I would be trained to do that, but that's kind of off the subject. This is Joe Hoffman. Hey, Joe Hoffman. Want to come by for a sec? Come on. Look, you coerced him. Tell us a good performance story. Any recent sensational, fun, performance disaster story that you know of?

Something crazy? You're on. It's on. Maybe something you witnessed at a customer or you maybe you screwed up someone's system oh that's a lot easier story to come up with and success story uh time i screwed up a system does he come through yeah okay yeah i i remember one i remember one is a little while ago but uh we went in there we installed the agent this was a long time ago.

And they were like, this is great, this is great. And at the time, the product had a capability to actually make changes to the system. This was back in the early days of WebLogic. Yeah, yeah. So the system detected there was a problem, made the changes to WebLogic, restarted WebLogic, all automated. Yeah. We thought this was great.

Automatic remediation. The customer said, don't ever touch my system again. We're like, oops, that was a mistake. So we took that out of the product. Years go by. Now I'm with Dynatrace. I take the same concept to a customer, different customer. I go, what do you think about the idea that we could actually automatically detect a problem, automatically make the change, automatically rebalance the system if necessary? And they're like, that's the greatest thing in the world. When can I have it? Do that right now.

And I'm like, well, we had that 10 years ago, I'm thinking. But it was before its time. Yeah. And so I think the interesting experience in my mind was this realization that it's not just about the right product, but it's about the right timing in the market. It is. Readiness. And it brings the fact that the community is aware of it. The community needs it.

The community is willing to accept it. They're willing to say, I'm going to hand this over to an automatic process. But I think the times have changed, the attitudes have changed, and now people are like, oh, my God, that's great. Oh, my God, this is wonderful. And so it's about timing, market timing, and I think that's what I'm really pretty cranked about. Dynatrace is this idea that what we can do now today, the world's ready for.

In DevOps transformation as of recent, like this is over the last year of everything we've talked and done with Dynatrace, the idea being that you are really attacking, it's one of the principles, learn your systems thinking as part of DevOps. Okay, it's not learn your

systems with your brain turned off. So what's interesting to think of the human evolution from the early days of a web logic self-healing. I know Cisco had done the self-healing system. F5 looked at it infrastructure-wise. The HP guys looked at it. It had it in, I think, some of the early HPUX stuff had the ability to do its stuff. And then it was the human limitation of saying, how do I think about this?

No, no, no, don't ever touch my system. It's a certified X, Y. Our definition for what needed to be controlled required a human being to control everything at their will, at their hands. It's about trust, right? It's not just trust in the system, trust in a vendor, trust that their system was repairable.

People don't think about that today. It's like, sure, it's a spin-up a, trust that their system was repairable. People don't think about that today. It's like, sure, just spin up a dozen. If they died, no worries. I don't care. I got five more in the pocket. So running. Just keeping coming. That's a fundamental change.

It is. And the cool thing that I really like about Dynatrace is this idea that as that evolution has happened, we've stayed with that evolution. We've stayed ahead of that evolution. So now we're like, oh, yeah, we're built for that. That's what we've thought that through for years. We've been thinking about that problem for years. Rather than, oh, I guess we should think about it, we have thought about it. Yeah.

It's a proactive approach. Very cool. And from this morning, there were some of those new announcements of the new stuff coming through. What are you looking for? You know, the management zones, the replay, the performance KPIs. What are you looking forward to the most out of the new stuff coming through? Management zones. Yep.

There's some really cool stuff, but that's just one I think every time I showcase this to a customer or talk about it, they're like, oh, my gosh, we have to have that. And we have current customers that right now have multiple instances of manage simply because they can't have manage. They can't tag them and separate them. Within their organizations how they have to and management zones gives that to them. Multiple different

hosted accounts. So it's not the sexiest one. Replay is really sexy. It's a really awesome feature. Management zone isn't as sexy as a feature but it's so critical. Incredibly critical. And I think it's also very critical to the expansion of the product throughout the enterprise

if we're going to truly go enterprise wide and think about this everywhere concept as many customers are trying to think through how do they do that with managing the information, managing access control it's a core feature that needs to be there I mean it's always been there in the mindset of we're going to put it in the product

it's now that time, we've done it so seeing that in the field is going to be to put it in the product. It's now that time. We've done it. It's now reality. So seeing that in the field is going to be super cool. Cool. Well, thank you very much, Joe. Hey, guys. Great. A lot of fun.

You have obtained yourself a sound cube. Very cool. Thank you for your performance. Thank you. You can also double as a fidget cube. Ah. Well, that could be dangerous. Thanks, guys.

See you around. Thanks. All right. Rock and roll. That's a good idea. a sound cube fidget cube. So the more you fidget with it, the more sound it makes? I guess, yeah. It could be a sound and light fidgeting cube.

Now they have squishies. So it went from fidget spinners, fidget cubes. So I have a kid, right? So I know these things. Now it goes to squishies, which is basically just these foam objects. It'll be shaped like a piece of cake or a peach or whatever. I never graduated from the pooch ball. And they're really soft, so you can squeeze it down, and it'll slowly expand.

Some of them quickly expand. It's like memory foam. They have different tensions on them or whatever you want to call it. And this has replaced the fidget cube and fidget spinner amongst kids these days now. Mostly because a fidget spinner can kill you. Small kids. If you put blades on the end of it, yes. Not child safety.

Okay. No child safety. All right. That's lunchtime. Yes. Tuesday lunch. Thanks for joining us, James. You can't see this, but Andy Grabner was here dropping off a bottle of scotch.

Half liter. I believe it was Austrian whiskey. so it would not be scotch. Austrian whiskeys. Yes. We'll be back a little bit later. Thank you, everybody. All right. See you guys.