PurePerformance - 070 Exploring Real User Monitoring with Ben Rushlo

Episode Date: September 10, 2018

Ben Rushlo, Vice President of Dynatrace Services, specializes in the Digital Experience. In this episode, Ben talks to us about Real User Monitoring. What happens when good bots go bad? Can Real User Monitoring (RUM) replace your traditional site analytics? If you have RUM, is there any reason to also use synthetics? What performance metrics are the best when it comes to monitoring the end user? How does RUM help you understand the performance of business? Tune in to episode 70 of PurePerformance for answers to these questions. https://www.linkedin.com/in/benrushlo/

It's time for Pure Performance! Get your stopwatches ready, it's time for Pure Performance with Andy Grabner and Brian Wilson. Hello everybody and welcome to another episode of Pure Performance. My name is Brian Wilson and as always with me is my co-host Andy Grabner. Hey Andy. Andy? Oh my gosh, something happened to Andy. I think maybe the Der Commissar got him. If anybody catches that reference and why would maybe reference der commissar with andy go ahead and tweet it at pure underscore dt haven't done that in the beginning of a show anyway andy is not available with us today uh he had something

come up so we have a fun show anyway you're gonna have to listen to me talk instead of andy which i know is going to be a real pain for everybody out there. But I have a great guest who I, this is not one of those way over my head topics, so it should hopefully be an awesome show anyway. I'd like to introduce my guest. We'll get right into it. Ladies and gentlemen, and people of all ages, Ben Rushlow. Hello, Ben. Hey, Brian. Good to be with you today. Thank you. So, Ben, for all of our listeners out there, why don't you tell us a little bit about yourself and why we should listen to anything you have to say? I often ask myself

that question. Yeah, I think a little bit about me. I'm Ben Rushlow, as you said. I am a vice president of services at Dynatrace, which basically means that I have a services organization that helps folks with what we call the digital experience management or DEM parts of the Dynatrace solution. So our expertise is really focused on real user, synthetic, I come out of a synthetic background, and obviously we're going to kind of advance into session replay. But I've been with Dynatrace, I have to say through an acquisition, so I can't say I'm an original Dynatracer, but through an acquisition of a company called Keynote, I think a total of about 18 years. And in that time, yeah, always in services. So working with lots of different customers, kind of seeing how internet technologies and performance management

has evolved over 18 years. It's really been a fun experience because every customer is different. So yeah, that's me. I'm based in remotely in Phoenix half the year and up north in Arizona the other half of the year when it's too darn hot to be in Phoenix. But yeah. You said up north in Arizona. Yeah. So it's people don't really get that. Arizona has like these amazing mountainous pine tree, you know, kind of climate areas or, you know, geographic areas. So. So, yeah. So. So Phoenix is great from about November, maybe through April at the best. And then it gets unbearably hot and crazy. And then, you know, I, I, uh,

I go up North and get out of the heat, uh, to a little cabin we have that, that luckily has internet and connectivity, everything we need. But, um, but yeah, that's, it's going from the desert and the place we have up North about 6,500 feet and, you know, pine trees and stuff. And it's awesome. Yeah. It's cool. I did not know. I myself did not know that, but I also, before I moved to Denver, I didn't know that Denver weather was awesome. I like most people thought all it did is snow all year long in Denver. And that's very far from the truth. If nothing else for your listeners, we'll learn about Arizona geography. There we go. There you go. For anyone who hasn't been there. And I haven't been there yet. I've been in New Mexico, but not down to Arizona yet.

So today we're going to be talking about a lot of RUM topics, RUM real user monitoring, right? And I just want to do a little precursor to this, you know, for everybody listening. This isn't specific to Dynatrace, real user monitoring. There's a lot of tools that do real user monitoring. Of course, we would love for you to be using Dynatrace real user monitoring because that's what funds this show, even though we have no funding for the show. But, you know, it's, this is all about the real users, the front end performance, right? So it's not specific to any one tool, any, any basic real user monitoring tool should be able to do all this kind of stuff. So just that intro with it, just so we don't think

it's, you know, we always try to keep it to agnostic. And that's the case for today. So with that, Ben, you gave me like a list of 10 awesome topics. I don't know how many we'll get through, but if we don't get through them all, we'll definitely have you back on for another show. But let's let's start with I mean, do you want to just go down them in order? Maybe we can tackle them that way. Or did you have a specific one you really wanted to start with? with um yeah i mean i think let's just tackle them in the order they're there i didn't actually put them in any order but that'll actually make it more interesting how they kind of flow yeah we're going to one topic and yeah so the first one's bot traffic right now i've seen this i've seen

this a lot but i'm gonna you're the expert on this but basically right for everybody anybody who's thinking what's a bot right and i don't think anybody out there is thinking that. We've had some interesting conversations, I think in the first year of the podcast, about some bot traffic and things people can do for them. But yeah, these are basically programs coming in and hitting your website. Some are good, some are bad, right? But you have told, you know, from my point of view, which is just strictly like the performance metrics or even sizing environments, I just think, OK, bots are something we need to know about. And you want to make sure you are for a performance point of view. Maybe you put that data or maybe you send those to a different server.

But from when you're doing this analysis point of view, you have a whole different take on bots, right? What is where does that go? Yeah. I mean, I think what's interesting is you sort of have, you know, in my view, you kind of have, you know, bots who are raising their hand saying, Hey, I'm a bot, right. Which is, which is super helpful because in that scenario, you know, it's easy to filter them out. Right. So, you know, most tools, rum tools and other tools, right. You know, can look at headers, unique headers and say, okay, this, you know, this person is nicely identifying themselves as a bot. And you could actually use Dynatrace as an example, like we have our synthetic product and we try to identify ourselves as a synthetic bot, you know, synthetic is a type of bot, right? And you can put screens, you know,

I had a call with a customer yesterday and they were, we were looking at their bounce rate and, you know, it was higher than I, than I thought. And we were getting this out of the rum data and a bounce for those who don't know, right. It's a single page session. So a user comes to a single page and then leaves. And typically you don't, you don't want a lot of bounces. I mean, depending on the site, right. Um, because bounce means, Hey, people are engaging. They're not staying. And, you know, this is, they're in the travel And they said, well, actually, that's really normal because we have all of these bots coming in and scraping our site for prices, right? So that's one example. But, you know, I had another customer I was working with and they heavily partner with, as most companies do, with Google around paid search and search engine

optimization and things like that. And we were seeing in the RUM data all of this traffic coming from Mountain View. And as we kind of dug in further, huh, strange, it's all from this Google ISP. And as we dug in further, it was all from a single kind of resolution mobile phone, right? And that's the second kind of type of bot I think that's actually more troubling is if they're not raising their hand identifying themselves, right? And you could put the screen scrapers in that. You could put what we found with this Google bot, which I'm still not quite clear what Google is doing. We've had some discussions with them, but it hasn't really totally illuminated what they're doing with that. And I think the

list could go on where there's traffic that is perceived by the business owner or by the technology folks looking at their site in the rum data as legitimate, that when you kind of go a few levels deeper, you start to realize this isn't legitimate. And it's not something that's, you know, they're not raising their hand saying, hey, we're a bot, please exclude us. They're intentionally trying to maybe hide potentially and not identify themselves, therefore skewing the analytics, skewing the data, right? You know, and it's a bit messy. And it's become such a big issue, I think, that I was presenting this data that I talked about, about this finding with this Google bot traffic to Google. And Akamai was on the phone.

And for those who don't know, Akamai, huge CDN vendor. We do a lot of stuff with them. They're a great company. Very kind of good in terms of pushing the performance envelope. And they said, actually, we're building a tool, an AI kind of intelligence tool, and they demoed it when I was on this call, to look at our traffic at the edge, right? Because they have all this traffic coming to their servers, and also try to identify these bad, non-identifying bots, right? And I say bad, I don't mean that they're, you know, like, they're not like, you know, trying to take

malicious bot attacks, right? But yeah. Right, right. But I mean, like, non-identifying, you know, not, don't have the bot name tag on. And they're also trying to build AI around, you know, around how to know who those bots are and then be able to kind of either treat them differently at the edge or obviously filter them even out of their own analytics. So I just think it's an interesting topic because for me, I hadn't really realized, I kind of thought of bots as, OK, they're identifying as bots. It's great. I get it. I understand it.

But this whole kind of second class of unidentified I think is interesting because then you sort of start to think about, can I trust my data if I don't exclude those? Or how do I really make sense of what my actual customers are doing versus what these bots are doing. So I think it's an interesting topic that people should be aware of as they look at their analytics data. And I would imagine that that has an effect on Adobe Analytics and Google Analytics as well as ROM analytics and CDN analytics,

anything that's capturing things that are supposed to be real users. You know, two things I thought of while you were telling that, number one is I can't believe Google's doing a bot without identifying, right? Because, you know, we always think about these malicious bots, but, you know, all the real players would identify themselves as a courtesy. And the fact that they're not, I feel it's a little bit negligent

because specifically, as you're saying, people have data that they're analyzing based on knowing that they're excluding known bots and here comes someone like google or if maybe it's another big player who's trying to do something in a clandestine style you're you're skewing people's data and that's kind of not not really cool uh so i'll give a little ding against google for that but um what i think is, you were talking about the good bots versus bad bots, right? And historically, bad bots have been the attack bots, the bots that would generate traffic on your site used for like, you know, DDoS attacks and all this thing. And probably, you know, almost every other bot that you know, all the good bots identify themselves.

But the definition of a good bot was one that was not attacking your site. And the definition of a bad bot was always historically one that was doing something malicious to your site. Whereas now, I think we get a second definition of a bad bot, not one attacking your site, but one who's skewing your data by not identifying itself. They're not doing anything bad, you know, scraping for pricing. That's, that's what you get. If you're like a sales site, right, that's going to happen. Usually people identify themselves, but if they're not, it's not that malicious one. It's more of that, uh, you're screwing up my data, bro. I hate saying bro, but hey man, what's going on? Yeah. And for some business people who, you know, as we become more of a data driven,

and I think this is good, right. As, as, as, you know, digital platforms become more data driven. And I think this is good, right? As, as, as, you know, digital platforms become more data driven and, and are relying on this data, right? As they start to do AB testing and multivariant testing and personalization, if you throw in, you know, 5%, 10%, in this example, I was giving, I think it was like 30% of their traffic was these Google, Google hits and you, and, and they're behaving in a totally different way, obviously, because they're they're not human, that that has a big effect on the data pollution. Right. Or, you know, potential to pollute the data, data quality. And I think, yeah, that it's it's not a minor thing then, because now you're rolling that data up and saying, oh, our A.B. test of this sort didn't work right or the personalization is not yielding the results you want or actually, you know,

we did a big release and the bounce rate didn't go down. And it's like, yeah, but if 30 percent of that data is is bad, then then you're kind of you can't really get the optics on on what you know what those other changes are doing. So I think it's a it is an interesting topic. I mean, I don't know if it's like the biggest thing that's happening that's interesting, but it's certainly one that surprised me. And it's something to keep an eye on, I think, for folks out there. Yeah, I think even bringing up the topic of, you know, rum is usually licensed by visits or traffic and everything, any rum vendor, right? And what we all do well is we exclude the identified pots, right? Now, if someone's not identifying, that's screwing that up too, costing people more money. Anyway, so that's,

yeah, that's pretty crazy. I love that someone's, Akamai's trying to do some sort of AI. Now they're trying to, I just want to stick with this topic a little bit more. You mentioned Akamai's trying to use AI to identify patterns of what might make a bot. Do you, have you come across any patterns that people might be able to tell? I think you mentioned something about single resolution phone or something like that. Like what, what have you seen at least out there that are kind of giveaways to a non-identified bot? If you, if you have any examples. Yeah. I mean, I think it's, um, and this is stuff that we've passed our product team, you know, to think about them. Well, maybe we could build something that's a bit better at catching these things. Um, but it's things that, you know, because real users are so beautifully

odd and variant, right? We're all doing weird things on the site. It's really looking for consistency and patterns that are almost too consistent, right? So it's traffic from a single ISP, right? Or block of IPs, you know, that's a little bit odd. You know, single city that has, you know, twice as much traffic as anywhere else, single resolution. A lot of the bot stuff we see is, you know, single city that has, you know, twice as much traffic as, as anywhere else, single resolution. A lot of the bot stuff we see is, you know, hitting single pages. Um, and so the sessions are all single page sessions, which are, you know, bounce sessions. Um, and so, yeah, I think it's, it's probably looking for patterns, uh, of uniformity that are

beyond what you would expect with, with human behavior, which I know that's not probably super helpful. It's a big, you know, big topic. Yeah, I get what you're saying though. That's what we're seeing basically. I think that's what Akamai is trying to identify as well. Now, my guess is once you, if you can build a, and I don't know if they're doing this, but I'm assuming they are, if you can start to build up that intelligence and then have even, you know, some sort of person integrate or interrogate that data. But then you could kind of market as, OK, if we see this again, right, this is likely a bot. You could build up hopefully, you know, some database kind of views where it's like, OK, we've already looked at this.

We know it's a bot. Next time it comes up, we'll throw it away. Right. And I think if you can do that, I don't think these folks are trying to be like I don't think they're hiding. I mean, they're they're not really aggressively hiding. Right. Like I wouldn't expect all of a sudden Google is going to say, oh, you found us. And now we're going to start spinning up, you know, spinning up bots from another location and trying to use a, you know, use a different ISP. I don't think that would happen. Right. So I think these are kind of, like you said, in between that good and bad. They're not really malicious. They're just not identified.

If we can start to identify them with data, then, you know, I think you could start to exclude them pretty easily, probably. Right, right. Interesting. Never thought of it. And, you know, whenever we talk about bots, I think about them and then I forget about them, which is what they should be. But it's always really fascinating.

So if you're done with that one, I guess let's go tackle number two then, right? So what are you trying to say with the second point here? I'm trying to paraphrase, but I'm not sure if I'm going to completely get it. So you're trying to say without rum, right? And again, this is not being salesy, but you're, well, without RUM, you're missing what's really happening. But without RUM, are you kind of saying that in terms of if you're just using an analytics tool that's capturing data or if you're not capturing anything? Yeah, so I think the way I see RUM, because I come obviously from a performance background, not a user behavior background. And I think what we've seen, and I, you know, I'd love to get your perspective on this and feel free to disagree with me because, because people do all the time, but, you know, it's like these, you know,

technologists and performance people have relied on, you know, kind of non-user behavior analytics tools. And that would be, you know, synthetic is a perfect example, Chrome dev tools, traces, webpage test, you know, et cetera, et cetera. Right. And they're looking, you know, at an APM tools and whether those are kind of gen one or gen two, whatever, you know, they had all this data right around performance, but they were missing like the user behavior impact. Right. So, okay. We improved the page, but did that actually affect how many people engage with that page or bounced or, you know, or we're making assumptions about the pages that actually matter. And that's where I kind of, I think I'm getting to in this point

too, is that, you know, we work with customers and, you know, their technology teams, and if they don't have RUM, they're unlikely to get direct access to the pure user behavior data coming out of, and really for most customers, it's coming out of two places, right? Adobe or Google Analytics, I mean, pretty much. They're very unlikely to get access. I don't know if you found that as well, but I mean, our customers cannot, even if they're high-level technology leaders, struggle to get from marketing, you know, those analytics pieces, right? And if you don't have those analytics pieces and you don't have rum, then you're kind of in the dark and you're sort of saying, okay, well, I think these pages are the most important.

And obviously people are pretty smart about, Hey, if I have a retail site, okay, check out, you know, card, et cetera, it's most important, but, but you'd be surprised where you're kind of the gaps are. And, and the example I was thinking I was thinking of when I wrote down this not very well phrased bullet was we have a very large auto customer who, of course, they're measuring what they think are their key entry pages, the homepage, et cetera, et cetera. But at the same time during a month, the marketing team is marketing team is doing huge campaigns and those campaigns can be offline, right? So they can be on TV and, or they can be in print or, or obviously they can be online as well. And you know, what we found with looking at the rum data over, over the month was, oh wow, you actually, your top entry page was this campaign campaign landing page. And the technical team really didn't know that that was happening. Now, you could say, okay, that's a dysfunction in an organization,

but I find most performance issues are dysfunctions in organizations. That's a fair point. But by having the RUM data, even though that data probably is very similar in some fashion to the analytics data that was coming out of Adobe at this customer, the technical teams were now able to have visibility into how user behavior and kind of the technical pieces interplayed. And so they were able to say, oh, OK, all of a sudden we see this big spike in traffic

for overall, but also on this particular page. And by the way, we're not really measuring that page to make sure it's up with synthetic, or we're not really looking at it from a performance optimization perspective. And by having the RUM data, I can see that and now actually be a bit more aligned as a business. So I think that's what I guess has always surprised me. As soon as you turn on RUM, pretty much with every customer we've done that with, they find things they didn't expect. Like, oh, we're getting a lot of traffic from, you know, bots, like we just talked about. Oh, we're getting a lot of, you know, traffic from these marketing campaigns that we maybe weren't fully aligned with marketing around. Oh, we're getting traffic outside of the country that we thought, you know, we thought most of our traffic was all in the U.S.

And strangely, you know, we have a ton of traffic from India, right? I mean, I think those are the sorts of things that you could go on and on, right? Top rows, JavaScript errors, et cetera. But if you don't have RUM, you're kind of missing the full picture. And even if that data exists in another silo in the organization, like in Adobe or Google Analytics, you know, you're still kind of a bit blind, I think, as a technical person or a performance team. And so I think that's kind of what I was getting at, I guess, with that point, probably. That's interesting because, you know, obviously you said it yourself, it seems to be more of an internal dysfunction if somebody has Adobe or Google,

and the teams are not getting access to it. But I do agree with you, too, definitely in a large point, because when you're talking about looking at data from separate tools, that starts becoming a little more difficult to piece together. So let's say every organization did give every team member who needed access to the analytics data. So they're seeing performance issues and they can go across, check to the backside. One thing we do see over and over, and I'm sure customers and anyone else listening to this podcast, who's has multiple tools is every time you're crossing from one tool to a second tool,

there's a chasm you're hopping over, where there's a complete disconnect between the data, that some of it's easy, you know, some of it's easy enough to piece together if there's a promotion page, or you can look at the promotion page on the back end, then you can look up statistics on the back end. But you start start falling you start going more into the correlation as opposed to causation realm so yeah i mean i definitely would agree having it all in a single type of tool definitely makes it a heck of a lot easier a lot more convenient and probably gets used a lot more then but i do want to put you on the spot right we have vendors like ourselves who make rum tools and then we have vendors like ourselves who make RUM tools, and then we have vendors like Google and Adobe that make analytics tools. And one of the questions that kind of pops in my head a lot is, if you have your RUM tool set up very well, everyone has access to it, you're connecting that to your performance data, do you still need an analytics tool?

Yeah, it's a great, it's a good question that we get asked a lot, right? Because I think a lot of times when we're talking about ROM, particularly when we have business people in the room and we're trying to kind of get them maybe to do a POC or, you know, trial proof of concept, um, their, their, their question is their first kind of pushback might be, well, we already have Adobe, right? And so in that case, what I try to do, and I think this is totally accurate, is say that, you know, what we're doing with ROM and as a company is really performance first. And what Adobe and Google are doing is really behavior first. And that those use cases are actually somewhat overlapping, but they're kind of more adjacent than they are completely kind of overlapped. Right. And an example of that

is, you know, a lot of the stuff with with Adobe is, well, first of all, Adobe is an interesting one, right? Because going from where they were with Omniture, which was very much just analytics to now, most of our customers are using kind of their full suite. And that includes things like they have a dynamic tag manager. They have a personalization, kind of what they call audience manager. So they're kind of personalizing content. So for that, it's almost an easier thing to cut is like, well, if you're really bought into the full Adobe suite, right, then, and they have a content management system, then it's really not competitive or, or, or overlapping

at all. Right. Um, yeah, yeah, they have an analytics piece. And what we found is people struggle to get some data out of that at sometimes, um, we've, we've been told many times that our data kind of is, is easier to consume, but it's, it's, it's, you know, you've bought into the Adobe ecosystem and therefore, you know, we would really position RUM as performance with an understanding of behavior because that's where those intersections make sense. But we're a performance tool first, right? Now, for other customers, they actually have replaced their, you know, Google Analytics. I think that's where we see it more probably because that's less of maybe a major ecosystem,

though Google does have tag managers and a bunch of other things. Because they're able to say, well, okay, for us, this analytics data coming out of the Dynatrace tool or RUM is actually good enough, plenty enough. And plus we get the performance stuff and the integration of the larger kind of ecosystem of our tools. So we have seen that, but I think we're, you know, I still see that we're adjacent, not directly competitive yet.

And I'm speaking now of the Dynatrace tool, I guess, versus other RUM tools, because we're so grounded and centered in performance. And, you know, these other tools are really grounded and centered in behavior. Are, you know, if I look at how to crystal ball and predict three, four years in the future, do I think, you know, I could be telling customers, maybe it's not even three or four years, maybe it's one or two years. Hey, you don't need this other stuff? Yeah, maybe. Because I think we're getting smarter and better about adding you know um funnel analysis and conversion analysis and we're going to have the

ability to a a b testing and out you know and those are the things i think the business really buys those tools for and so as rum evolves and i guess whether it's our rum or other rum you know as it evolves more into that business area, I do think you could make the case for not having it. What I think is actually happening, which is pretty funny, is I'll look at sites and they have Adobe and some of them have Adobe and Google Analytics and they have a rum tag from Akamai, which, you know, they bought, obviously they bought Impulse and they have our rum tag, right? And you kind of say like, what, you know, what, but the reality is like, each of those are serving different use cases and, and, and different, um, internal constituencies,

I guess. And so, you know, I'd love to see a view where it's more consolidation, you know, but the reality is I'm almost seeing the opposite, right. It's like people are using, you know, more and more analytics, which, which also means I think good analytics is really important because, because, you know, there's a, if there's a plethora of data, which one do you trust, right? Which one do you actually use for these different use cases? Right, right. And a lot of it comes down to understanding how the tools are collecting the data and looking at the data. I think this, this segues then perfectly into number 10, since we're talking about comparison of different things,

let's jump into the, as you, I liked how you put it, the fallacy of synthetic versus real user, right? So if we're talking about real user versus analytics, and there are some use cases for both, obviously, let's talk about synthetic versus real user, right? So for synthetic out there, just a brief explanation for people who might not have come across it that's when you're having basically some of these good robots as we talked about before uh driving traffic against your browser it might be a single page hit or it could be a click path through but it's going through and from specific points geographical locations think about them as like a control test. Any kind of time you're doing a

laboratory test, you always have a control. So this is always from a very specific point in time or sorry, point in geographic region, broadband, whatever. You're hitting that constantly, or not constantly, but at a timed interval, as opposed to your real users, which are all that various wonderful goodies that your real users give you. And I guess as you're mentioning here, and people ask all the time, if I have real user monitoring, Ben, what do I need synthetics for anymore? Yeah, it's a great question. And when we've kind of deal with and, you know, at a for a time, when I worked at Keynote, you know, we only had a synthetic tool. And, you know, we only had a synthetic tool and, you know, other

folks were developing ROM tools and, you know, it was sort of like, you know, you felt like at the time everyone's going to throw away synthetic, right? Because, because all, all, you know, all you need is ROM. And, you know, I come from that background, so I have some bias, but I've really been sold on kind of the beauty and amazing pieces of ROM and the ability to see kind of every user, every browser, every page, every geo. Right. Like that, that visibility is like going from kind of black and white to 4K. If what you were using, you know, synthetic for was to understand performance. I think that's the biggest shift. Right. So it used to be that we would say, hey, synthetic is a proxy for your user's experience, right? And people can read a

lot into user experience in that phrase, right? And, you know, for some people, if that meant, well, synthetic is a proxy for how my customers are performing, I think that is where RUM really has taken over, right? Because synthetic is one browser, as you said, or even if you have multi-browsers that most vendors are going to single browser, you know, um, you know, a subset of geographic locations, a subset of pages, even though, even though those pages are really critical, the reality is you can't really measure the crazy performance that we all bring to the table with our different setups, different browsers, viruses, my grandparents' computer, you know, my kids' computer, Wi-Fi, you know, mobile. And so I think, you know, what's really happened is RUM is the voice of truth for performance in production, but Synthetic still has a very strong, and I think will continue to have a very strong and compelling use case around

availability testing, making sure that kind of I can get in from the outside testing everything from the browser through my network, DNS, you know, my routing, et cetera. I think it has a very strong use case around that. That's probably the strongest. I think it has a super strong use case around QA or pre-deployment testing, because RUM really can't fit in that. If you don't have real customers hitting a, you know, QA or pre-launch site, you really can't fit in that. If you don't have real customers hitting a QA or pre-launch site, you really can't do performance testing of that or you can't get an understanding of performance.

I think it has a really strong use case around comparing yourself to your competitors because, I mean, not to take a segue, but I will, I think that's a big challenge we see with customers is they collect performance data, but they have no idea if it's good or bad. I've had high-level executives tell me, oh, it no idea if it's good or bad. I've had, I've had, you know, high level executives tell me, oh, it's fine that our pages take eight seconds to visual complete. And you're like, well, why, why do you, why do you think that? Oh, I just think that that's fine. I've, I've, you know, I think that's good. And it's like, well, wouldn't it be

better to make, wouldn't it be better to make a decision based on actual data, right? So if you're in the bottom percentile of the internet at eight seconds, that might drive you to make a different decision than if you're in the top percentile, right? And so having competitive context of what is good, what is bad, right? Whether it's in your industry or it's across the internet, because you could make the case that maybe you shouldn't only compare to your industry because users aren't just thinking about that when they think about your site performance. But having that competitive context, super, super helpful. And it's hard to get with like a rum tool or an APM tool.

You can't really say, I'm going to deploy my rum on my competitors. It doesn't really work that way. So, you know, so Synthetic has a really strong use case for competitive baselining as well. I mean, again, I think the strongest is really for availability monitoring. I had a customer this week, again, not making this stuff up, where they run a booking engine, airline booking engine.

And they were not able to capture these issues that they were seeing with RUM and a back-end APM tool. But they were with Synthetic. Now, obviously, with RUM, they were able to cross-correlate. And you could see, like, sessions were dropping and conversion was dropping. So it's not like RUM has no availability play. But as a primary front line, it doesn't have the visibility into, hey, my pages are returning 500 errors or I'm not able to get through my border router or I'm having DNS failures. Like you can't pick that up very easily with RUM. And so I think synthetic is still really strong there. Last thing I'll say about that is, you know, what I've told customers,

and I think this is true, is if you were spending, you know, 100% of your dollar on synthetic, right? And this is just talking about synthetic and rum, not the other APM components. But, you know, obviously, I think you're probably going to start evolving that where maybe 30% of your dollar is going to be on synthetic and, you know, 70% is going to be on ROM. And, you know, maybe that's where you'll be in a year or two years in between now and then you're going to kind of move in that direction. I think that is true. I think we're going to see customers spend less in synthetic, pick the very key entry points, the very key transactions, you know, for availability, maybe for, you know, pre-launch, maybe for intermittent competitive benchmarking. But I think that's sort of where we're seeing the industry evolve to.

Very interesting. Another thought I was having too with it is when we're talking about, you know, the one use case I really love is the control, right? Because I'm a fan of science and controls are very important. And this works on two levels, right? Because you have synthetic that's testing your front end, but you can also use synthetics to test your back end components, right? You got any API endpoints and everything that you're running. So take these two scenarios, right? Scenario one is if you have a, you know, when you're testing for on your front end, as we said, when you have all these various end users with all their malwares

or anything else going on, maybe they're, you know, we've seen in the past where people's browsers are really slow and you take a look at what's going on, they're downloading tons of ads and the customer is like, we don't put that many ads on our site. Well, they had their browser been hijacked, right? So like you start seeing all these kind of things. So when you start seeing some bizarre behavior slowly creeping in from your users, if you have a million users on your site in an hour, right, and three or four of those or just like a small number of those users are starting to show some anomaly, right,

you may or may not write it off as, okay, some people have these, you know, this is the end user, right? There's going to be some people who have something screwed up. Whereas again, with that control of the synthetic, your synthetic should only fail if there's a failure because you're not adding anything to the browser. You're not changing anything with it. It's coming from the same point all the time. It's that control factor. So a single failure on synthetic, I believe, but you're obviously the more synthetic person, is so much more of an indicator than smaller. It's much more of a stronger warning than a small amount of errors coming from real users

because there are so many things that could be impacting, you know, so many outside factors. Now, taking that to the back end side, you know, if you have a back end payment system and, you know, maybe it's really early in the morning or overnight and not too many people are on your site, if you're, you know, doing more of a geographically based store, one or two users, again, failing on something payment, you might be like, all right, something happened there. But if your synthetic is failing once, a second time, even, you know, that's pretty big indicator you have a major, major problem. So that's one of the reasons why I love it still too, because you have, because of that variation that you have in your real users, because there can be so many

other factors that can be creating, leading to a user's failure or a user's error real user's error those synthetics typically should be much more of a louder warning bell at least to me but no yeah i think i totally agree i think the the controlled laboratory environment allows you to detect very small changes even even hundreds of milliseconds of changes, which in real user, it's sort of an interesting paradox, and I'm sure if I was a better statistical wizard, I would understand it more, but it's, on one hand, like you said,

real users are all over the map, right, in terms of their setups and kind of performance and issues. And on the other hand, if you get a site that has lots of traffic, that sample kind of gets washed and you really can't detect major changes. Right. And then during the evening hours, when real user traffic typically, unless you're running a 24 by seven, you know, site drops, you almost always get a performance increase. And that's really based on the sample. Right. So it's based on the less sample. So it's it is hard if you're trying to say, OK, I want to really detect micro kind of changes as I'm tweaking my front end, let's say, or my back end.

Or to your point, I want to really understand errors. And you're right. I think a synthetic error has a lot more weight because you kind of have a controlled environment. It is harder to do that with rum. And I think that's what people, I mean, to be honest, I think this is going to sound bad, but having worked with customers for so long, I think, I think one of the things that was push people so hard to rum against synthetic was synthetics hard, right? The scripts can break, the sites change, you can get false positives, right? And then people threw rum on their site and they're like, Oh, everything's green all the time. It's like, you know, it's all just, everything's green on

my dashboard. Now I think that is, it's, it's a, it's a fallacy again of like, I just don't want to deal with the messiness of synthetics. So I'm just going to use rum and, and, and then probably not look at the right metrics or maybe just kind of not really realize that there's some gaps that rum can't cover, but I'm just going to be happy because things are green, right? And so I think that's, you know, the other piece is synthetic requires a bit of an art and a science to get right and to make sure the scripts are hardened and to make sure you're not over-measuring or you're not under-measuring to deal with site changes and stock and inventory. And all that stuff is hard.

And so I think, you know, we help customers do that, but some customers kind of give up and say, well, as much as I like it, it's too messy. And, you know, I'd rather just have this rum data, which I don't really have to do anything except for put a tag on the side or, or auto inject or whatever. Um, and I think that's kind of lazy, no offense to our customers, but I think, I think you think you have to kind of work harder. And that's, I mean, of course, you also need an APM piece. And like, that's where bringing it all together and as much telemetry as you can get, right, I think is helpful in really understanding what's happening on the site. So. Right, right. You mentioned something in there that kind of makes me want to go to point number seven.'re talking about metrics and and what data to look at

and i think we've seen um over the years many many attempts at people trying to define what you labeled as the perfect performance metric i know ourselves we we like to talk about you know this with the visually complete speed index these are the new ones that you use um and i think a lot of these new metrics are really awesome and amazing but I think the point that you're making at make trying to make here is that the performance perfect performance metric is I would probably wager to guess that it's the one that you need for the specific thing that you're monitoring yeah I mean I love this topic because because it's one that I get to kind of go head to head with our customers about. And it's always fun.

And you really get to learn a lot about a customer's culture and organization when you have these discussions. Because I've literally been in conversations where it's like, well, the reason we use Dom Interactive is because our CTO came from, insert Silicon Valley, you know, a company name and they use that, right? And, you know, that, and that's, you know, that's, that's why we use this or, you know, our business people, they want one metric and they really can't deal with, you know, kind of uncertainty or, or, you know, the ambiguity of multi-metrics. So, you know, the ambiguity of multimetrics. So, you know, that's why we chose XYZ metric, right? And I think it just illustrates, you know, again, it kind of to my point about, you know, people want easy and the green, you know, the green

dashboard and, you know, something they feel like they can, you know, take out the complexity. And which is kind of ironic ironic because if you think about how the internet has evolved and I'm old enough and I don't, I think you're old enough to, to, to, to have been around for most of that. Right. It's like, how complex are these sites and apps now? Right. In terms of the underlying technology, the cloud technology, the CDN technology, the integrations. And yet we're, we're striving for like like we think like somehow one metric is going to make sense of all that right and right it didn't work 10 years ago and it doesn't doesn't work it doesn't work now so anytime i go into a customer and they have either a pet metric where they feel like you

know and i i would say some of the dom ones are pets metrics like that don't always totally make sense to me like oh dom interactive and it's like yeah yeah, but that's very early in the W3C kind of timing cycle. The time to first paint everyone talks about now. Right, right. Or they just have a single metric. I think we're working with a large auto manufacturer in Europe and they chose visually complete. And that's smart, right?

I mean, in the sense that I think the thing I like about visually complete and speed index is it's getting us closer to something any human, and I would say like passes my wife test, right? Any human. Let me just respect your wife there, of course. Exactly. Any non-technical human, right, could understand. I think that's been the problem when we had metrics like, which we were a big proponent of, of load event and, and I still am actually, I think it's a good measure because it probably most of the page

functionality in JavaScript has fired. Probably the page is visually ready. Um, and that was way better than end to end, which used to be kind of the old metric back in the day, back in the day, because that started to include lazy loaded stuff and third party tags. And of course that's, that's kind of nonsensical. Um, but you end, it's like, try to explain that to a business person. Like, oh, well, it's this timing and the browser has really loaded all the content in the DOM. And then it's like, I'm sure Andy or you could probably do a much better job of explaining it than even I could. But for a non-technical business person who is trying to use these metrics to either hold their technical teams accountable or to drive improvement or to understand user experience, the closer we can get to something that makes sense

for a non-technical person like Visually Complete, the better. The challenge, long-winded way of saying, okay, this customer in auto manufacturing, you're abusing this, the challenge is it doesn't work on every page, right? And so it's like, it's not perfect. There's situations because of embedded frames, because of a shadow DOM, because of X, Y, Z, where visually complete isn't the end all be all metric. A chat box can come in late and that could skew visually complete for multiple seconds. And then you're going to the business saying, well, Hey, your page takes eight seconds to visual complete. And they're like, well, I just tested it on my browser. There's, there's no way in hell it's taking eight seconds. Like, okay, well that's because that's not the right metric. So

I think the, you know, what we try to do with our customers, a lot of it's just education is let's use a combination of metrics that, that actually helps us get to what pages need to be optimized, right? What's outside of normal. And for us right now, the three kind of key ones are visually complete, which I think is still a great primary metric, easy to understand above the fold render. Speed index, which is kind of a way of saying how fast am I getting to visually complete with most of my page? And I think that's kind of encompasses what used to be, you know, first paint, things like that, you know, really it's a measure of kind of perception and, and how quickly am I getting some content to the, to the customer. And then I still think load event ad is useful because it does exclude typically if you build

your pages, right. It excludes third party content and lazy loaded content and things like that, but it does give me another kind of view. And so the three of those together are really useful. Now, two years ago, I would have said something different because visually complete wasn't being a focus. And I'm sure in four years, I'll be talking about, you know, it's Google's talking about first meaningful paint and first meaningful interaction. And it'll probably evolve again.

And the point is that don't get obsessed with a metric, you know, for the sake of a metric, get obsessed with the idea of using the, as you said, the best metric for the, for the, the right situation, you know, for the particular situation in a way that will help you drive improvement, right? If the metrics you're using are to help you get a bonus, you know, or to show your boss that everything's okay, then I would say that, you know, don't use it. I mean, that's just crazy, right? But a lot of customers look up. Yeah, exactly. Or just make them up. We will have internal discussions, you know, when we're, you know, we have lots of great customers, but we have some that are challenging, of course. And, you know, you kind of sit around and have a

beer and you're like, you know what? I think this customer would be more happy if I just gave them an Excel sheet where I just made up the numbers, right? Like, I just think, you know, just give them what they want, which is I want green, right? And it's like, they don't need to pay for all this complex measurement and telemetry and data and rum data and synthetic data. It's like, just, they just want to see that, you know, things are fine. They want to live in a fantasy world. And, you know, I think obviously that's not the norm, but I think you can kind of see that when you get to customers who are like, well, this is the metric. And it's because it's the fastest metric and the business likes this. And it's like, OK, but the whole point of performance management is really to drive improvement for your customer.

And so find a metric that actually represents as closely as we can or or a set of metrics, the customer, and then use that to drive improvement. So you can see, I get very excited about that topic. Yeah, no, I think it's a, it's a great topic. And what I think it really comes down to is right. There's yes, there are some better, there's definitely better metrics than others. Right. But what I think it comes down to for an organization is number one one you need to really kind of split your pages out into things for different types of performance right performance of a page load you're going to probably want to measure differently than an xhr call totally even something with a login which has a redirect and all right there's a there's a lot of different

kind of things that go on in these and to to me, what it really comes down to is capture data about, capture multiple metrics on all these pages, because you're not going to figure it out right away. But over time, as you're looking at these different metrics and you're making improvements to some of these metrics, it doesn't matter. The goal is not to make improvements to the metrics. I don't really care if my speed index increases, if my end user satisfaction is not better, right? This is tying it back to the rum data, tying it back. And you've talked about, you know, time, uh, business and performance data way back when I was doing something with one of our integrations. I don't know if I made this up, but I, but I liked the term,

the performance of business, right? Um, you've got to look at the performance of your business. And if you do make an impact, if one of those metrics drastically gets better or drastically gets worse, does that have an impact to your conversions, to your bounce rate, to your user satisfaction or any of those things? If it doesn't have an impact, you can probably start scratching it off the list of an important metric to measure. But if you do see that change have an impact, you know, then suddenly, this is one of the most important metrics to do on it. So as you say, there's no easy button, if you're really want to be serious about your run performance. It's not just about the metrics, but it's what the change in those metrics and how that impacts the end user. And that's what

run is all about is, you know, real user monitoring. So I think it's definitely, yeah, people want it to be easy. And if you get the data right, you can make it easy. You have data streams out. Hey, maybe you can even use AI for some of this. Hmm, I wonder where that idea comes from. But there's data. Turn it into information and work with this. Because most people just want to have the checkbox to say,

we're collecting the data, but you know, how are you using that data? And you can, you can talk about metrics to your face turns blue, but you still have to know how they're impacting, you know, that, that your end user to see whether or not they're, they're useful and what you can do with that data. Uh, anyway, no, I totally agree. Can I hire you to work on our team? I mean, that's exactly the, you know, kind of stuff that work on our team? I mean, that's exactly the, you know, kind of stuff that we have to edge. And that's so much of what consulting or services really is, is educating customers so that they can kind of transform their own organizations, right. And get over their kind of issues and silos and hangups around whatever it might be.

In this case, we're talking about metrics and data, but, but that's exactly right. And I think you said it very clearly. It's like, you have to tie it back to the business. And for so long, I think that's really what's been missing in the performance space. Now, obviously you all, you hear these things about, Oh, Google did this study five years ago where they slowed down, you know, intentionally slowed down some percentage of their traffic. And they looked at how users engage with search and, okay, that's great. That's Google. That's awesome. That adds a lot of value to the industry. But generally, most, if you talk to most large companies that weren't in Silicon Valley, they didn't have any understanding of how performance was really, you know, transforming or affecting the business. And I think to your point, being able to see in a single kind of view, conversion, bounce, engagement, right?

All that's super critical because that's really what matters at the end of the day, not a number. And yeah, I totally agree. I think it's right on. One of my favorite examples is the idea of the clicking on place order, right? People think, oh, we have to speed up our pages specifically because speed translates into more conversions. And usually place order takes a little bit longer than other things on their site. So quite often people will look at, oh, place order is running slow. Let's improve the speed of that. But what we've seen is a lot of times once you get to someone clicking on place order.

They're already caught, man. They're already into it. Right. Yeah, exactly. They're placing their order. It's the pages that lead up to it, which are more important. Granted, you don't want your place order sitting there taking 15 seconds and they're thinking like, what the heck's up with this, right? But that, you know, but that's also, again, tying back to what is the performance of my business. I'll use my little phrase again to know which ones are actually having the impact on those conversions. So not just making improvements, but being able to measure that improvement. That's kind of like the shift left, shift right, that Andy always talks about looking at not only, you know, how do we take metrics from

the left and push it to the right, but how we also take metrics from production and let that influence what we concentrate on pre-production to make things better. Yeah. I think the, the thing that I think I like about what you just said is it's, it's, it's the performance metrics and business metrics in context, right? So like the context in there is the, someone's already six clicks into a conversion funnel, they're hitting place order. Like, does it matter that it takes a second versus, you know, half a second, right. To come back, maybe not because we understand the context of where the user is within their kind of digital funnel or journey. Right. And I think that's what, that's what we need to bring more to, and we're trying to do in our, in our area

to the metrics. Right. So, so another, another example of that, that I kind of drives me nuts is, you know, people are like, well, do you guys have, you know, uh, data on your tool that shows, you know, very, very cleanly the relationship between performance and balance. And I'm like, yeah, we actually do. We have one graph, but I know who they're, they're thinking of other competitors. And, and what I say to them is like, that's not a simple, and anyone who can tell you it's a simple equation, like doesn't understand the context, right? That, you know, you're not always going to find this relationship. You need to know things about referrers. You need to know kind of where the traffic's coming from. You need to know where they are in a funnel,

if you have a funnel, right? And really all sites have some funnel there, even if they're not typical conversions. It's way more complicated. And if you think you can just, you know, make it simple and, and, and kind of go and say, well, here's my one graph that explains how performance and balance intersect. It's not that simple. And I say that as someone in our tool where we have that graph. So I get it. Right. Um, but we're exploring kind of deeper connections and we're finding like if you have the refer data, you actually might be able to make more sense of that. Well, why is it? Because if I'm driving a Facebook campaign and, you know, people are clicking through and, you know, knowing that that's a segment of traffic versus, you know, organic search, you know, those people that are clicking through, we might expect to bounce that have nothing to do with performance. Right. Because just we know that if you're driving people to campaigns, they have higher bounce rates. And that probably isn't driven

by performance, right? But maybe if we understood this other, you know, cohort or segment of our population is coming in organically, there we might find a better relationship, right? A performance to abandonment or performance to engagement. So I say that to say like all of that context is super helpful. And I think that's where we need as an industry to kind of add more business context and user context. RUM is a great starting point for that into all these performance discussions. Completely agree. Well, listen, I know you have a hard stop coming up here. We got through about half the list. I am not going to summon the summerator because I would not be able to do what Andy does. Come on, do it. So for everybody who was curious as to whether I was going to summon the Summarator because I would not be able to do what Andy does. Come on, do it!

So for everybody who was curious as to whether I was going to try to summarize, my summary is RUM is really awesome and has a lot of really cool stuff, but as with everything else there's a lot of complications with anything performance. Once you start thinking about it, you start thinking about more and more and more and you start getting overwhelmed pretty quickly. But the important thing is to at least have the data so that you can turn it into information. So I guess I did a little summary there. Ben, last words before you take off?

No, this is great. It's a lot of fun. I didn't know what to expect, and it was great to chat about topics that are near and dear to my heart. So thanks for letting me do it. We'll try to schedule another time. We can get Andy back on and finish up the rest of the topics. So I think they're all really cool ones still. So thanks a lot.

Thanks, everyone, for listening. And if you have any questions, comments, tweet us at Pure underscore DT. Ben, do you do Twitter or anything like that? I'm so lame. I don't. I don't do social media. That's fine. That's fine.

You're probably on Wikipedia all the time. So just guessing from your, your outdated piece there. Yep. Yeah. Well, uh, thanks a lot. We'll talk to you soon. And, uh, thanks everyone for listening. Bye.