PurePerformance - GraphQL, API Gateways, API-Led Growth – How to make APIs Observable with Sonja Chevre

Episode Date: June 5, 2023

APIs are powering and empowering software innovation as they enable new use cases on top of existing services. Observability into API usage to answer questions like: how APIs are called, what APIs do, where APIs fail, where APIs are slow, where APIs are misused … has to be on top of mind for architects that decide to build or use APIs.In this episode we welcome Sonja Chevre, Group Product Manager at Tyk, who recently gave a captivating talk at KubeCon about using OpenTelemetry to get insights into popular API frameworks such as GraphQL. We are discussing common challenges for SREs such as that APIs often hide the status of a call behind an HTTP 200 or that debugging individual calls is really hard as details of the call are not exposed by default to telemetry data. We also cover topics such as API-led growth, API as a product as well as open standards such as OpenTelemetry and OpenAPI. Here the list of discussed links during the show:KubeCon Talk: https://kccnceu2023.sched.com/event/1HyVc/what-could-go-wrong-with-a-graphql-query-and-can-opentelemetry-help-sonja-chevre-ahmet-soormally-tyk-technologiesAPI Management vs Gateway discussion: https://www.linkedin.com/posts/sonjachevre_apimanagement-apigateway-apisecurity-activity-7061596404854521857-N_cO/API as a Product: https://tyk.io/blog/unlocking-the-potential-of-api-as-a-product/OpenAPI: https://www.openapis.org/OpenTelemetry: https://opentelemetry.io/

It's time for Pure Performance! Get your stopwatches ready, it's time for Pure Performance with Andy Grabner and Brian Wilson. Hello everybody, welcome to another episode of Pure Performance So you have no proof? Well, as you just said, I have the recording. And it's got the video recording, so I can do the whole thing. Speaking of the recording, we had a challenge with the picture today. So the challenge for the listeners is, where's the broken pure path? So if you get to see the picture for this episode, find where the pure path is broken. The distributed trace, I should say.

Distributed trace. Distributed trace. Distributed trace. So Andy, I had another dream I got to share with you. Actually, it was a nightmare. Really? Yeah, and I'm sure you've all had this one before. You're going to one of the first days of college, university,

whatever you want to call it. In my case, it was high school as well. And you don't have your schedule. You have no idea where you're going. Do you all ever have that one? You can't figure out where you're going. Is this familiar to you? Follow the crowd. But you need to get to your specific class. The funny thing I've got to say before I go down into this one

is going back to when I was in college, I'm sure the younger people listening to this, they have it on their phone, their email, whatever. We just had a piece of paper. We literally got mailed a piece of paper with our schedule, a thin piece of paper, a flimsy piece of paper. It wasn't laminated. And that was it. You had nothing else. If you lost that piece of paper, you were screwed. They'd be like, oh, go to the bursar's office, which is the

dumbest name in the world. Go to the bursar's office and get another copy. And the only way you could find where the bursar's office was was if you had your map, which was on the folder they gave you. But if you were like me, you're like, I don't want this stupid folder. And you throw your stupid folder away. And now you don't have a map of the campus anymore. That was the only way you can get a map of the campus. Because again, this is before the internet, before the computers.

So anyway, it was always a nightmare. So anyway, in this nightmare dream, I had to get to my class. But there were all these different buildings with all these different classes in it. But I needed to get to my one specific class. So there was a pair of classes and then a specific class. The class I was trying to get to was gateway management in modern media. But I didn't know where it was. I didn't know what time it was.

And I spent all this ridiculous amount of money to be in this place. And it was just a nightmare that I'm sure we can all relate to. Anyhow, you were in there and I couldn't find the class. And I was like, Andy, where do I go to find my class? He's like, do you have your schedule? And I was like, I don't have my schedule. And he's like, go to the bursar's office. And I'm like, don't know where the bursar's office, Andy. He's like, it's over there. So anyway, I eventually got to my class and I got to look at my, you know,

my API. Sorry, I just said the wrong word. Oh, I said API. Anyway, we're speaking of APIs, right? And some other things. So that's the tie-in. And anyway, that was my nightmare. It's just hilarious

to see what you come up with I wonder what's in your coffee in the morning honestly some non-sugar sweetener and cream and some peppermint extract hey but you brought up a good thing

talking about APIs should we talk about APIs should we talk about API gateways should we invite somebody on the podcast that we talk about API gateways? Should we invite somebody on the podcast that actually knows about API gateways? And API management? See, that was the university, was the management, because it had all the different kinds of classes.

So, without further ado, Sonia, welcome back to the show. Sorry for having you endure our stories and our fun in the beginning, but I think Brian is doing a great job. Yeah, that was good.

You know what? I have a dream like that that comes up sometimes, where I show up to an exam and I'm not prepared. And I know I have to learn, and I wake up like, I had an exam and I'm not prepared. So I kind of relate to that kind of university stress. Yeah. And you don't have those dreams when you're actually in university.

It's when you're an adult and way past it, right? It's the dumbest thing. Anyway. Hey, Sonia, we had you, believe it or not, almost three and a half years ago on the topic of understanding distributed tracing, trace open sensors open tracing and open telemetry there's a mouthful of telemetry telemetry telemetry um sonia for those people that don't know you who are you uh i'm sonia i'm a product manager in tech i used to work with you guys in dynatrace i really really miss you. You're awesome.

But I decided it was time for me to see another domain, and I'm now working at Stack. We are an API management company. We have an API open source gateway. So if you need an API gateway, take a look at our open source gateway. And since I come from the observability world, I have seen everything that happened with open observability, open sensors, open tracing, coming to open telemetry,

and I have brought that whole world and expertise to tech, to API management, and I think there's a lot to do here. And that's the pickup. We can talk about it today. Yeah. And especially coming back to the dream, right? If you think about it as an API gateway, somebody that is on the campus and you can go to,

and Brian goes to him and says, I know I need to go to this class. And then the API gateway actually sends you directly there because the API gateway knows exactly, hey, he needs to go to that room, so go over there. But the API gateway also knows if the room is already full. So maybe then rejecting it. So, right.

If he's allowed, even allowed, you know, like expertise. Is it an advanced class and he's still a beginner? Yeah, he knows that Brian obviously is not a good fit there also. Or because he he was there yesterday and the day before and he always made troubles. He always caused the class, he always disturbed the class. So now the API gateway is not allowing him through anymore but giving others a chance to go to the class. I'm always interrupting the class with my dreams.

Sonja, all jokes aside, we met recently at KubeCon in Amsterdam. You did a talk there and we'll definitely make sure to put the link in. The talk was called What Could Go Wrong with a GraphQL Query and Can OpenTelemetry Help? And the interesting thing about this is

I want you to talk a little bit about it, but we also had a talk recently on pure performance. I think it was actually the last one or second to last with Justin, who was also talking about how he implemented OpenTelemetry in their organization and how they're also very GraphQL heavy. And they saw a lot of challenges with GraphQL. But also on the other side, they saw a lot of great things with OpenTelemetry, like how

GraphQL is actually taking query requests on the front and then distributing it to all the backend services. We talked about the classical M plus one query problems, capturing too much data. It felt like all the way back in the early days at Dynatrace when we talked about the M plus one query problem in Hibernate. Yeah, every time I talk about it, I have the picture of Andy in the box.

Yeah, it's really interesting. So GraphQL is really interesting for me because last year I learned a lot about API management that of course everybody knows REST API, so you send a call and you get that HTTP response code. So 200,

everything fine, and 400, 500, then you have an error, you need to look into that. Your monitoring solution observability will flag it out so you know you have an issue. you need to look into that. Your monitoring solution observability will flag it out so you know you have an issue. GraphQL works a little bit different. It feels more like a database, actually kind of a database-like language for API

where you can ask for things that you need, but you don't need to get back the whole data. You can really ask, I want only my username and my user title, and you don't get everything that is available about the user. So that's really good for mobile application. That's good for versioning, so you don't need to manage all those versions. You can add gradually things to your schema. Also, the user experience is really good.

As a developer, you don't have to know everything. You just can write your query. You get what you need back. But the problem is that GraphQL, if you send a query, so there's kind of two different errors. There is, if you have a problem with an upstream service, then usually you will not receive back a 200 error code. So you can already understand there's a problem,

but then you kind of need distributed tracing to understand where is the problem, which services. But then if it's a schema problem or something else, then you get a 200 error code back and you cannot, you know, you don't know that there is a problem. So if you're exposing APIs and you have

external consumers that are calling you APIs, you don't even know at the moment that they're having issues. So in our talk with my colleague Emmett, we explored some functionality, some recommendations, and adding error codes in the semantic convention of OpenTelemetry to be able to better detect error in GraphQL. Because if there's an error, you want to know. And you want to be able to fix that or improve your examples if it's an error side error.

So let me ask you, I mean, we love analogies and I have to come back to the dream. So same API gateway. If I go to it and I'm not a student, but I am maybe the, what's it called? The headmaster of the school? The principal, the dean. maybe the, what's it called? The headmaster of the school? Dean.

The principal, the dean. And the dean basically says, I need to know how many students do we have in the class right now across all the classes that are older than, let's say, 18 years old. So I could phrase that as a query. And then obviously, GraphQL would then make sure that it goes to every single room and says, how many are in here that are older than 18? And some come hopefully back with an answer. Some may fail. And then kind of the aggregated result would be,

yes, we have five classrooms. And for four of them, we have the results and the number is, let's say, 20. So the result will come back correctly, but maybe somewhere embedded it would say but there was an error in one of the services or maybe the number is not fully fully up to date right and then the question is and this is I think where OpenTelemetry really comes in nicely if I understood you correctly you could really then trace in what did GraphQL

actually really do to which classroom did it? To which backend services did it go? Which one returned the result in which time? And was there any failure in a particular instance? Israe? Yes. So GraphQL, you can see it as a bit of an integration layer. So you have multiple services or data source that have different domains. So it could be you could have a service

or data source for the students. You could have one for the different locations, the different domains. So it could be you could have a service or data source for the students. You could have one for the different locations, the different classrooms. You can have one for the teachers and then the GraphQL service would help you put all together to understand who should be who in which classroom.

So maybe you would have a GraphQL service to help you find your plan and understand where you should go. But maybe there's a problem with the teacher service. So GraphQL will return the information to you, Brian, that where you should go for which subject in which class, but there will be a problem with the teacher. So the teacher would say blank.

And on the HTTP layer, then GraphQL would say, okay, 200. So you would think everything is fine, but you looking at your plan, you would say, who is the teacher, who needs to be there? And that information would be missing. And in our talk at KubeCon, we explored the possibility that you have with OpenTelemetry.

So the default instrumentation of OpenTelemetry at the moment for GraphQL doesn't add that extra layer of errors. So you wouldn'tmetry at the moment for GraphQL doesn't add that extra layer of errors, so you wouldn't catch it at the moment, but you can do some custom instrumentation to add it. And this is also some information that we are adding a tag in our own API gateway that produce GraphQL endpoints. So we are going to add this information, this error information for OpenTelemetry out of the it's only a quick question is when you say graphql is now creating open telemetry data are you then talking about the graphql client and server-side libraries or who is it exactly the

server-side library so when you work with graphical you have different provider of GraphQL engine. So you have the usual, like Apollo, for example. You can also do it with Node.js and just the Express framework. So you have different where you can do it per code. You can do it low code, like we do it with our gateway at stack. So you have multiple possibilities. If you do it by code, then you can go to OpenTelemetry, go to the website

and you know there is the list of all the instrumentation library. So you can add OpenTelemetry instrumentation for your GraphQL. For example, in the talk we did at KubeCon, we used Express and Node.js and then we added and then just tried all the examples with that

and then we saw, okay, that's a shame because when there's an error it's not flagging OpenTelemetry because OpenTelemetry still looks at these specific instrumentations, still looks at the HTTP 200 error, but there's not that added

information about the error. It's really easy to add, so we added it in our prototype with Node.js Nexpress. Something you need to know. And I think that's something we are going to suggest as a semantic convention for GraphQL because it makes sense for everybody to have a kind of agreement on

how does a GraphQL error look like, so that later the observability vendor like Danatrace can make use of it and surface better insight for the user. So it's something that we all need to work on as a community, I think, making GraphQL instrumentation

in OpenTelemetry better. And then another question. So because this is for me, technically I'm just interested in this. The API gateway in your case, Tyke, what is your role in all of this? Are you actually providing a GraphQL kind of mapping where you can say if somebody makes a call to the GraphQL endpoint and you are basically distributing the load and you kind of play GraphQL server, or is it more that you are inspecting the payload and then based on that payload, you're extracting metadata to enrich the OpenTelemetry space? What can I...

It depends on how you're doing GraphQL. So we are an API gateway, API management. So if you think back about the API gateway, the role are really things like security in governance. As you say, you're going to control the access to the APIs. Also to make sure that you expose the right APIs. Maybe you have internal APIs and you have some things that you don't want to make public

to the public, so you can remove something, do some kind of transformation, change the API contract. And then we have... And we can do it with any kind of services you have so if you have already a GraphQL endpoint we would just forward to it

a proxy to it and just help you with the security maybe also some GraphQL specific security so with GraphQL we can have some permission on the fields, so in GraphQL you have certain fields and you can say okay that user is allowed to get information

on that field. Another user is not. So there is some specific gateway feature for GraphQL. But then we also have our own GraphQL implementation. So we have our own component that's called the universal data graph.

And this is something intact that you can configure, define in the user interface, link different services, and define a schema on how those data aggregates together. So in the case you are using tech's own GraphQL implementation, the universal data graph, then you are going to expose those information about errors with OpenTelemetry.

In case we are proxying to a GraphQL endpoint that you already have, we won't be able to add that extra layer. So that's something that would need to come from your own instrumentation. Folks, if you listen to this and you're looking into GraphQL, I can highly encourage you, watch the full talk from KubeCon.

I know sometimes in the podcasts, especially as we're talking about technical things, it's always easier to also see things. And I think in your presentation at KubeCon, you also did some great demos. And I think the role play with your colleague back then, right? And I remember one of the things where he was then asking you for, hey, it's great that in observability, I now see all the requests that are coming in on slash graph, but this doesn't tell me anything because everything is all of a sudden hidden by the query endpoint. And I want to see

which queries are actually taking long, which queries are failing. And I think these are all very important considerations, whether you're using GraphQL or any type of API, kind of, if you design your own APIs in a way, right, you always want to make sure that your APIs are A, observable, but you always have to write metadata for every single request because if everything is hidden behind one endpoint, then like your colleague said, that P95 doesn't do me anything because it's basically just like, oh, the average,

it doesn't do me anything because it's an average across everything. And I have some queries that are super fast and some that are super slow because they're doing something completely different but i don't get this information if i don't get the semantics on this on these calls yeah so i think that's really two false cards you need to make sure you have the time to data unit the api the the query so depending on the use case but but really all the semantics you need, and then you need in your tool to surface them in a right way. So some tools like Jaeger aggregates everything and really look at the HTTP layer, which is not enough for GraphQL.

So you really need to make sure you get that added granularity on the API level, if it's API on the GraphQL query or on the GraphQL field. And this is something that I think, so there is a basic that is clear. We want to have the errors. In terms of performance, I think we as a community still have some work to do together

to figure out what is the best way to expose the most meaningful information to OpenTelemetry for GraphQL and what is the right granularity. And maybe it also depends on your use cases. Maybe you have a GraphQL and what is the right granularity. And maybe it also depends on your use cases. Maybe you have a GraphQL service that is used only by your own frontend. So the queries, you only have a set of 10 queries that are always sent. And then it's really interesting on the query level. Or maybe you have an API that is open to the public

and then you need to track it for API clients or any other level of granularity. But, you know, everybody think about how can you get the right level of observability so that you can understand failures, usage, performance. And I think the other aspect that you talked about in the beginning, what is actually really happening behind a query that you're executing? How does it branch off and what does it really impact in the end? And this is where distributed trace analysis comes in to really see, okay,

these are really expensive queries because they are querying the data in a really inefficient way or making too many roundtips or capturing too much data. And this is like the analogy with Hibernate, what I always bring up. Exactly. analogy with hibernate what i always bring up exactly like never dies yeah yeah what else is there so you are obviously working from you know you have a lot of experience now besides observability on on api management do you have any other suggestions especially if people are listening in today and they say hey hey, we are building new systems. We have services that will expose APIs.

What are some of the key questions? What other questions are out there maybe that people need to ask when it comes to observability and operating it in the right way and giving also enough information for developers? Yeah, so we work with teams, API platform teams, and they are responsible for the API gateway, for API management governance, but they start to leverage that platform engineering point of view. So they want the developer

to be able to generate new APIs, to generate the configuration, to push it to production, and to have the insight they need. So we see, and sometimes those teams, they use different kind of observability tools. Maybe a team's acquired or work in a different way.

But the API gateway is kind of the central thing. And most of the things that get the calls first. So that's kind of the first component. So it's really important to understand your end-to-end timing. And those platform teams that are now looking at things like the OpenTelemetry Collector to get insight from the gateway for the API and make sure that you send the API information from the gateway from Team 1 to the observability tool of Team 1. And the traces of Team 2 to Team 2 kind of splits the observability data

and have really observability pipelines for the teams and make sure that the observability data they land in the

right tool of the right team that is responsible for the end-to-end of those APIs and at the same

time also using the open telemetry collector to remove sensitive information so API platform team which

cannot have the full picture but just make sure they send secure data or remove anything sensible to their own development team. Also maybe different sampling strategy. So we see really

an interesting centralization of the API gateway but then the configuration and the API making sure that teams are self-serviced, they can push new APIs, and they get the feedback that they need. So this is one of the things that we see.

Another thing that's super interesting is the whole API as a product. So we see more and more financial teams or any kind of teams see that there is a benefit and there is a really opportunity to have API products to sell your APIs for integration or for data. But then the developer experience is even more important because those external developers are trying different APIs. They want to be really quick and integrated.

And I think observability is a right place to understand what are those developers doing with your APIs. Are they getting any issue? And even as a product manager, you can go there and have a kind of analysis, what kind of errors. Is it 400 errors, something has moved? Or is it something in the documentation? So you can, with observability,

you can catch many issues. And some are not always operational. Sometimes they're really developer experience and that tells you a lot about your api quota so for instance brian and i obviously with our history of having fun openings we could come up with a with a choke or actually maybe not with a with a with a with a dream service right we build a service with with an API and you can generate your next dream. And then we need obviously some input data on like a topic suggestion.

So we could then, first of all, we need to make it easy for developers to use this. But then we can monitor, first of all, what weird input data comes in from people. That would be interesting. And then also see, right? And then we can see- Are they coming back?

You know, do they want more? Are they coming back? That's a good one. Are they coming back and are they coming up with some strange input that actually then causes our backend system to behave abnormal because we cannot do anything with that input? But I think from a developer experience, a joke aside, I think there's two aspects. From my perspective, there's multiple aspects. One is you need to make it super easy because you're selling a product and you want to make sure your product can be consumed easily

and therefore provide examples, make it super easy that people can just call your API. But then I guess you also need to make it transparent on what they're actually doing. So I think from an observability perspective, I also want to bring back the data to the developer how often have you how well consuming this api maybe even give some recommendations on how to use it better more efficiently because apis are also as it is a product there's costs involved with it right both on the developer side that need to pay eventually and then also on your side because you need to host the infrastructure um and then obviously from from you're hosting, if you're the seller of this product, you want to get insights on how

you can optimize it. Because if you see everybody makes the same mistake, why is that? Ah, because it's in our tutorial. Everybody makes the same... Yeah, we forgot to update it with the last version. Or you want to monitor the versions. With API, you can have different versions

and at some point you want to deprecate it. Are people still using that version? Is it going to cause issue if we deprecate it? Who is using it? Can we, is it two customers, we can go talk to them or is it still hundreds and we have no way of, we don't want to cause an issue on their side. Yeah.

I need to take a lot of notes here now because I obviously in the end have to do a write-up. But I really like the APIs, the product, making it observable to benefit both sides, right? The user, so the dev, but also the one that hosts it. Yeah, that's really interesting. And folks, if you listen to this, there's so many links that we post. What, Brian, sorry. I'm laughing at you talking to yourself on the podcast.

That's great. I know. That's how far we got, yeah. Yeah. You were saying if you're listening to this? If you're listening to this, I have a lot of links. Also, Sonja, you sent us a couple of links

leading up to this. Also one to a recent post you did on on linkedin that got quite uh quite the attention i would say um and it is about what's the difference between an api gateway and an api management and that has lots of lots of interactions repost comments likes um Can you fill us quickly in? Yeah, so sometimes, you know, we have an API gateway that is open source, and that's the thing where people get to know us first. And sometimes people that are not that familiar

with API management, they will come to me and ask, how do you make money? And so it's important to, so it's important to... So it's really interesting to differentiate what is an API gateway, what is the role, and what is the bigger value of the whole API management. So an API gateway is a component that solves technical problems. Things like you want to control who access your APIs.

Is it allowed? Is it not allowed? You want to get observability to understand which APIs are being used, how they perform. You want to have some features that doesn't make sense to add in each service like authentication, weight limiting. So the API gateway can be a central component that does a lot of stuff for you without you as a developer having to care for it. You can write the business value and then all those helpers, weight limiting,

or transformation, you can do it in the API gateway. Also the API interface. So maybe you have internally, you have gRPC services that you want to expose as well. So those are all the things that an API gateway can take care of you.

But API management is so much bigger than that. It's really about the value that you are providing. So is it about... And we say it's kind of more people... And we say it's more about... And we say it's more like a people problem than just technical problem like an API Gateway. So some of the common use cases that we see is... and that our customers are asking is, how can we enable the

reusability and the discoverability of our APIs? You know, if you are in an internal organization, big organization, how do we make sure that people know where to go and they can get the API? How can they learn about it? But also, how can we publish API for monetization and for external consumption? And then, of course, we talked about API as a product. Now there is this whole trend and push toward API-led product growth. So how can we make, can we grow as a business? Can we use API to unlock part of the business?

That's really interesting. We see a lot. You see it also with AI, right? AI, all those AI tools, what they have is API that power them, that enable you to monetize them. Which are the APIs that provide? So you, and you're looking more on, well, with the API gateway,

you're looking at observability, like performance. But when you look at API management, you look at the big picture. So you want to understand which APIs are providing the most business value. You want to understand how to share API documentation with things like OpenAPI also. You want to understand how

you can encourage the developer to follow the best practices in the API development lifecycle. So it's a really how do we use API in our organization and it causes a lot of interesting organizational and people challenges. That makes it super interesting.

Together with the tech aspect, that is also really interesting. I kept taking more notes, but I kept my mouth shut because I don't want to feel like I have to be reminded by Brian that I don't have to talk to myself in a podcast. So folks, I took some notes and I will add links to OpenAPI to this as well. Yeah, so I know we're hitting the end of time there. Andy's got to catch a bus.

Don't let the pigeon drive the bus, if anybody knows that book. It's a kid's book. And Andy's the pigeon. But I can use my app to actually tell me, because my app is using probably the API of the local bus company, to tell me if the bus is even on time because the weather has changed. Maybe the rain is impacting the train schedule. But another example, right? If you provide data to an

API, like traffic information, there's so many different things you can build on top. And this is why I really like the whole idea with an API as a product, API-led product growth. I mean, obviously, thinking about what we do at Dynatrace, it's very similar. And also, AdOps are building vendors, right? We want people to use our data that we have or send us more data using our APIs to push and pull. It's going to be very important so that more people can build new

capabilities on top of existing capabilities or data sets that are exposed to an API. That's how you innovate, right? That's how you innovate. So I think another interesting topic, obviously we got to wrap up here, but we'll be talking about security around API exposure, right? Not just hacking into it and getting into it, but using that data. Just really quickly, I've noticed every once in a while, I'll get a delivery notification for a package or something. And depending on the company, every once in a blue moon, it'll be like, oh, you can find where your truck is.

And it'll show you on the map where the truck is. But I'm sitting there thinking like, well, wait, this is a great way for people who like to steal packages to follow the trucks and just go get stuff. So it'll be interesting to see and you also heard about some of the data breaches from Facebook where they just take all this data because the API is there and then they use it for nefarious

reasons. So all those are kinds of considerations which will be interesting. Anyway, the only thing I want to say, because I know we've got to wrap, is I'm going to have to go back and watch your talk, Sonia, because today's topic was all new for me and trying to figure it out first time from here was quite a lot. GraphQL I know nothing about so I'm definitely looking forward to watching Today's topic was all new for me and trying to figure it out first time from here.

It was quite a lot. GraphQL I know nothing about, so I'm definitely looking forward to watching your, what was it? What conference was it? We'll have the link below. What was it? Did you say it was? It was TrueCon.

It was TrueCon. Yeah. So definitely look forward to watching that. Yeah. But a whole new set of topics that I really didn't know anything about, Andy. So thank you, Sonia, for sharing that all. Andy or Sonia, you have anything else you want to add to wrap up before we...

Just if you're interested in API observability, what we can do with OpenTelemetry, we're looking forward to improve with the community. So feel free to reach out. And if you have ideas or input or feedback, they're really very welcome. Awesome. And I can only say, look at the picture that we took from the podcast and find the broken distribution trace.

Yes, and I think the best place to go do that will be on the Spreaker page, S-P-R-E-A-K-E-R. We always talk about the show notes. I'm not really too good at updating the notes on the Dynatrace site. But if you go to the Spreaker hosting site, that's always got the links, the pictures, and all that kind of stuff. So check that out there, and you can always contact us at pureperformance at dynatrace.com And thank you for...

Andy's got his jacket on, and he's making fun of me again because he's an evil bursar. That's your new villain, an evil money... They collect the money. Anyway, thank you for having us on and I hope everyone has pleasant dreams. Thank you. Thank you,

Sonia. We'll have you back. Yes. I would be happy to. Thank you.