Coding Blocks - Tracing Specifics – Know your System with OpenTelmetry

Episode Date: September 5, 2023

Get a behind the scenes intro to some of the interesting conversations we have before we even get into the content. We’ll be jumping into the meat of this episode and looking at the specifics of tracing using OpenTelemetry. Before we do that though, we should probably find out what special 2-liter containers Outlaw uses […]

It's actually hot in here, but I think that's because I was outside in the sun for two hours without a hat on the day. Yeah. I'm burning up. You look a little toasted. Yeah. I didn't realize the sun was actually coming down like it was. It reminds me of our San Francisco trip.

Yeah. And mountain biking. Oh, that's nice. Nice. It was awesome. All right. Let's do this. I might have to like,

if I, if I disappear like during the recording, it's because I drank two liters of water on my ride. So, you know, I'm just saying like, I might be flush and need to like empty, but I think you're going to have to translate that to the empirical system of things. Cause nobody here understands what two liters is. What the fuck are you talking about? empty but i think you have to translate that to the empirical system of things because nobody

here understands what two liters is what the f**k are you talking about nobody knows what you've never seen a two liter at the store how many ounces you actually drink man you've never seen a two liter coke but i don't remember what a two liter is nobody buys those things because they lose their fizz after the first day like they're they're complete you buy cans or or you don't buy anything the cans are disgusting no dude they have like a metallic taste they're they're the worst no it's glass or plastic or nothing what's better man a two liter or or a smaller thing i like my drinks flat honestly oh that's disgusting sometimes i'll open up a can and i'll put it in the fridge and come back later you have it but you open it off gas a little bit that was i mean if that's what we got i mean you know this is making the episode right yeah totally this is ridiculous

i can't even take it this needs to be a poll we need to bring back the poll for one episode because this is ridiculous nobody prefers a two liter i can't believe it yeah i like it uh my wife is a hard time i like it uh without the bubbles that's that's awful i mean i don't know why you lose the carbonation in it like you're not you're supposed to put the cap back on when you're done i don't care if you torque that thing down to like nine million newton meters it's gonna lose all its carbonation i don't know i haven't had that problem maybe maybe you drink them too fast well that i was about to ask like are you like leaving your your two liter around for historic purposes and coming

back like 10 years later and like i'll talk about two days carbonation i'll talk about two days two days dude two days and the thing is flat no you might as well just shook it up and then just like left it out the sun i can't get behind this this is no that's just not how it works all right pull back we need the poll for this this particular episode all right this is episode 220 217 and yeah it's whatever i got it right the first time you know what i said subscribe to us on stuff and hi i'm michael what are we talking about today i'm joe's acting yes i can't believe we're skipping all this is killing me guys don't like it at least subscribe to us so i said subscribe i did say that one. Did you? Okay.

All right. Whatever. Okay. So, yeah, I'm Alan. All right. Hey, so this episode. Do you want to tell us where we can be found on the World Wide Web? Use keyword coding blocks. www.codingblocks.net.

There we go. All right. Man, that's so painful. All right. So, yeah, what are we talking about? Yeah. Aside from flat sodas. Flat sodas. Oh, man. All right. Yeah. What are we talking about? Yeah. Aside from flat sodas. Flat sodas.

Oh, man. You kill me. Although, hey, anyways. All right. Moving on. So we're going to continue with Open Telemetry. This time we're actually going to get into some of their implementation and how you would actually set it up and use it. Right.

Like some of the features. So first, though, we'd like to get into the news. And Outlawlaw you want to read us off the reviews wait no there's no news just get into it we're talking about so sad there's no reviews no reviews we gotta beg better something we might have to dedicate like 15 minutes to begging this episode i think the key is is that like uh you know if you've already left a review thank you yes why did you only leave a review when you could leave multiple and if you haven't left any please you know it's yeah it's the one ask so all right uh i'm trying to think was there any other news it seems like there was

but i oh you guys saw jimmy buffett passed yesterday that's super sad so everybody played cheeseburger in paradise or margaritaville or something you know he's wasted away in margaritaville for sure yes yes um that was super sad uh yeah i can't think of anything else all right so i guess i guess we'll go ahead and start in yep all right so we talked about open telemetry last episode, but we mostly talked about the concepts that you need to understand to even be able to get into and really know what you're trying to buy into there. So this time, we're going to talk about the pieces that actually make it work. And specifically for this episode, so as I went through looking at the open telemetry things there's there's quite a bit of information and there's there's several sections on different

types of signals so this episode we're going to focus on one type of signal which is called a trace so just to back up for one moment for anyone who hasn't listened to the previous episode i mean first of all you need to rethink your life choices what are you doing go back that's right also but also uh we're this is coming from open telemetry.io yes right yeah so just wanted to like so we've already clarified that it is in the um what was it the incubating stage or something and that means it's perfectly fine to put in all your production apps, I think is what we said.

That's something that means something to Jay-Z and to the rest of us. We're just like, whatever. Whatever. Yeah, just add it. Yeah, that's fine. Yeah. I have this node module, this NPM module. Surely I can use it.

Right? Nobody would ever turn that off. All right. So the first thing we're going to talk about is a tracer provider. And it's exactly, if you're from the Microsoft world, you know exactly what this is. This is basically a factory for tracers. So in most applications, it's initialized one time and it's, it's life cycle is the same as the life cycle for the entire application. This is also sometimes known as a singleton, like a global singleton that's,

that set up. So yeah, I mean, it's, it's what it is. It's what you use. You know, I know there's no relationship here, but for some reason, like I just had this epiphany that do you remember, i want to say it was maybe the devops handbook that that talked about um the tracer bullets oh yeah and using that as like a way you know like how you would the idea being that like you would uh correct course as as you're going you don't like try to like make that perfect aimed sniper shot instead you just like spread it out there and you know just line it up yeah and adjust as you go and i got the thing i was like well they called this tracer and that made me think of the the tracer

bullet episode and then i was like oh i wonder if there's some kind of thing but then like no no this is just tracing well i think one thing they even said in the book is i didn't know this but like i guess it's not common to have like it's not like they're all tracer bullets i guess something it's like every 10th bullet every 100th bullet so i mean that's like literally what we're talking about it's great yeah so you can see where your where your shots are going over some period of time um so yeah they also say this is typically the first step in setting up open telemetry is you're going to set up this tracer provider, right? And they also say in the same setup, it includes a resource and an exporter initialization. And we'll talk about some of that in passing as we go through this.

So a tracer, it's created by a trace provider and create spans with more information about what's happening within the request. Now, I will tell you, if you're following along in the documentation for this, it's I forget if you go to the docs and then go to signals on the left, you'll see tracers like the very first one there. I feel like they did things sort of backwards here because they go into like these big blobs of like JSON and stuff that show you all the implementations, but then they talk about everything down below it and it feels like it would have been better if they had put that stuff at the end. It is super weird. It's like, this is how you do it. Yeah.

I mean, when I was taking notes, I actually had that up at the top and then, and as I went back through, I was like, man, this isn't going to make any amount of sense. Nobody's going to really know what we're talking about with this. so we have some stuff at the end we'll have a link to that

page because once you hear this episode you'll be able to go up there and look at it and be like oh okay i get it but also i mean kind of kudos to them right because like they understand how we developers work and they're like i don't have time to read your documentation show me how to use this thing i'll figure it out ah crap how is this working wait how do i do yeah fine i'll read the documentation let me scroll down you're probably onto something there so yeah it felt backwards but you're probably right it's probably the right way for us um all right so the trace exporter this is what sends the traces to a consumer. So if you think about it, you're going to be creating these traces. Well, this thing's going to export those things to something that's going to be listening, right? So you can think of the

trace exporter sort of as a producer of these traces. Um, you can set it and here's what makes a lot of sense. When you think about it, you can set it up to be just a standard out, right? Like if you just, if you're doing it for debugging purposes, when you're running it locally, you can have it print to the console. That's fine. Uh, you can also have it be set to send to an open telemetry collector. So think of something. Um, I I'm imagining something sort of like a Prometheus, right? Where a lot of metrics go type thing. That's a similar type thing.

Or they say they have all kinds of other open source libraries and tools out there that can be consumers of these exporters. Did we say yet, like maybe last episode, like who already supports this? Like Prometheus, I think was one that came up last episode, like who already supports this? Like Prometheus, I think was one that came up last episode, maybe like on a data dog. Uh, I was thinking like a,

a Google, uh, log Explorer. Like, does that support this? Like, cause I'm thinking of like Kubernetes. Cause they do have specifically a Kubernetes section in the documentation.

So yes. So they have have i could have read ahead they do have uh a collector documentation for it all right yeah so check this out they on their site open telemetry.io slash doc slash collector um in their opening paragraph here they say the collector offers a vendor agnostic implementation of how to receive process and export telemetry data so they they have some docs on this and they do have some things up here they have like an hotel collector is one of them they have jaeger prometheus listed fluent and bit or fluent bit i see it's also in there yeah i was telling him like i kind of want it to just pass on to it's like my normal logging framework you know like let me configure that stuff how i want it well that's why i was asking the question they're like you know because go let's rewind the clock right and it's the year 2016 or 15 or whenever we did the the 12 factor app discussion right and one of the 12 factor app

one of the 12 factors of a 12 factor app was to just standard out everything right standard out standard error all your logging and then that way you could just rely on something else to scrape that right or you know we're not necessarily scrape but you know what i mean and so that's why i was asking the question about kubernetes because that's kind of a common thing in in the kubernetes world is that like your output's going to standard out so what jay-z said i think like just having it go to a standard logging i don't think that's good enough because i my guess is the job of the collector is to correlate things, right? And we'll get into that stuff here shortly. And I think that's the big distinction between what open telemetry is doing

versus your log for J or your, or your log back implementations or whatever, right? Is those are truly just messages you put out there with a timestamp, whereas the whole point of using open telemetry is it helps you tie all those things together over time, right? Yeah, and I was thinking, like, you know, it's common for larger organizations to do some sort of structured logging so that all the logging from all the various services is using, like, common fields or common names and common levels for, you know, severity and stuff like that. And so you want all those names to line up.

So I don't really want to configure my app to, like, try and kind of, like, make sure things are named in the same format you know like for both open tunnel train for logging so i just kind of wanted to you know sum up past through i did look and they do uh say they have support for auto instrumenting with most popular logging libraries so if you want to kind of have your transaction id dumped out in the logs that you're already dumping out then it'll go ahead and do that it'll kind of feed into it which is pretty cool i guess that the hole that jay-z and i are both falling into uh and i'm taking you with me i'm taking you down with me jay-z is that like we're we're automatically thinking of like logging and so that's why we're thinking of like i want it to go to my other my standard logging destination

source whatever that might be and but this is more than just logging. Cause there's like a whole metrics capability that, uh, open telemetry is trying to provide, which is where you're going. That's why you need that collector be able to like piece all these things together. Otherwise how are you going to do metrics on it? Well, and even past metrics, right. Just being able to create the overall trace, right. Because that's where you're going to start tying spans together, which we'll talk about here in a minute. So, so I guess the next thing is one of the big keys of open telemetry is they have this context propagation and they say, this is the heart of distributed tracing as it takes multiple spans and correlates them. And we will talk about that right now. So you have a context. Go on.

So context, the context is what contains the information that allows these spans to be correlated. So if you have, this is just an example, right? You have service a calling service B service a is going to have a trace ID, right? And it will also have its own span ID. So the trace ID, if you think about an overall request, you know, let's say that you did something from a user interface, you clicked it, that's going to generate a trace ID. That thing's going to be constant throughout all these things. Service a gets hit. It's going to generate its own span ID, right? Then it's going to call service B. Service B is going to reuse that same trace ID so that those two can be correlated as part of the same trace. Service B will also have its own unique span ID because

it is like its own new unit of work, right? If we're talking about like developing terms, but then it's going to have a parent ID that's going to point back to the span ID from service A. So now the second span, which is like a sub request from the first service, it can be tied back up. Now, here's an interesting thing just to keep in mind. I think

we talk about a little bit later as well. Let's say that you made the request from the UI and it went up and it hit some generic service, right? That generic service calls service A and then that generic service also calls service B. Those are more like, what are they called? Sibling, sibling type things. Yeah, so there's no hierarchy there, right? So service A is not going to have a parent span ID that it's going to link to, and service B won't have a parent span ID.

So they'll just show up as children of the main process, right? Wouldn't the generic thing be the parent then that in that scenario? It could be, I mean, if we went that route, it could be the parent. And then those two children service calls would just show up underneath it. It wouldn't be a hierarchical thing, right? So that parent, that parent thing is what links it up and says, Hey, this was a sub request under this other thing. I think this is also, like we talked about it similarly, referencing the dev tools in Chrome.

And even in the documentation, they talk about the observability waterfall diagram that's kind of similar to what, their example looks very similar to what you would see in the Chrome dev tools. And there are some of those calls that you can tell are like peers of one another because they're on the same level. Yep. They sit beside each other. So the only thing that's apparent of them is that

process that was just above it. Right. So, so if you have that parent ID in there, then it makes that new span, a child of that other span. So this is why the open telemetry collector, whether it's one from open telemetry.io or some other third party thing is important is because it's stitching these things back together for you. Right. Um. So propagation, this is what moves that same context between the services and different processes that happen. So behind the scenes, it's actually serializing and deserializing the context objects. And it's providing the information that needs to be carried from one process to the next process. Right. So that you're not having to do it every time. I mean,

I'm sure we've all done stuff like this where we'll have some code that we've written in and we want it to piggyback off some other stuff and you have to keep passing it along. Right. This is doing that work for you. Have you ever done like a pre on your logs? Like you put out like the same three pieces of information and you want them in the same order and the same.

So you like copy and paste all your logs like yeah you're you're basically recreating like a terrible version of this yeah exactly that's that's 100 well i was going to say it kind of seems like the idea of like like another session right except you you you there's the session that you're logged in as right and. And you're just looking at this webpage and then nothing happens. But then there's like the quote session of you, like clicking on something. And then that initiates a whole series of actions. Right.

And like, now you're just bundling all that up into like this one quote session of like, well, that's what happened here. You know, kind of thing. Yeah.

It's, it's super helpful to have that view. And this is what open telemetry is creating for us. Um, so it said that the serialization and deserialization that's usually handled by the instrumentation libraries that come with the software, right? So this is where you kind of have to be careful. So when you look at open telemetry.io, they support a bunch of different languages. However, there's certain bits that are only like fully fleshed out in like maybe JavaScript or C sharp or whatever. Maybe it's not as perfectly done and go. I'm making that up. I

don't know that that's the case, but, but not everything is 100% done in every one of them. So they say, Hey, if it's not handled by the library, they also have API propagation APIs that you can call yourself if it's not fully hooked up. So you still have a path to go, even if your particular library doesn't have all the functionality it needs. Man, I was going to say like, um, where they fail in their documentation is that I just want, just give me a picture so that I can show this to my boss and be like, hey, this

would be cool. We should do this. Because at the end of the day, the higher up the ladder you go, the more succinct it needs to be. And visualizations definitely go a long way. Something about like, you know,

drawings are worth a bunch of words. I don't, I'm workshopping it. I'll get back to you when I figure it out. But, um, and then, and then I don't know if you've gone down the demo path and I'm like,

haven't installed it yet. I want to, that's the problem is, is like, Oh, go install your own thing in your own Docker or Kubernetes. And you're like, yeah, I just thought it yet. I want to. That's the problem is, is like, oh, go install your own thing in your own Docker or Kubernetes. And you're like, no, I just want to see like a working thing. Show me some pictures.

Yeah. But if you keep going down into the architecture of that demo path, you'll see some drawings and you're like, oh, I'm sorry. I asked because it's a little too busy because, because the very first drawing starts off with a bunch of like okay here's boxes for all the services and lines are going everywhere and you're just like uh who clicked what hey can we take a moment here real quick because i i love that little tangent it is really hard to do architectural diagrams that make sense. Isn't it? I mean,

like seriously, it's a skill. It's an archer. Yeah, for sure. You can take days trying to make something. I mean, there was one that I was looking at the other day with,

I think some of my teammates and, and somebody was like, well, why, why is the arrow pointing to that? They're like, Oh no, it's not it's

actually behind it it's like going through it it's like oh oh okay well that wasn't obvious can you make that obvious there's like 1200 lines on the page it's like yeah i guess i can figure it out yeah every it's hard every time i i've ever like sit down like oh i'm gonna work on this drawing give me an hour or you know 30 minutes or whatever like I always grossly underestimate it and then I'm like you know the end of the day and I'm like I'm almost done yeah I hate when I'm working

on one of those diagrams and like I'm you know just adjusting the lines and all you know this one's a little it's a little off center and you do you like move something like wait a second I this whole thing is like off center like everything is wrong like this basically this what I thought this is light and then nothing's lined up yeah i mean i know i know just about every time i've done one like i'll reach out to outlaw or jay-z and be like hey what do you guys think about this right and then you know it's like i'll work on it for an hour on my

own and then i'll get some outside eyes to look at it because I think that I've fallen down into the zone of I can no longer see what I need to see type thing you know the problem is though is this it's so beneficial to like explain to other people like this is what we're building this is what we're doing this is the new direction that we're taking or this is how this thing works and like everybody can immediately go back and reference it. And especially when like, as a team, you're all trying to like get on board with something new, you know,

it's super helpful. If you can just like, just give me a drawing of like how this works. So by the way, under demo, there is a demo screenshots section that that's exactly what I was hoping to see of like, Oh, cool. More stuff. So yeah, I'll throw some links out here. Yeah. a demo screenshots section that that's exactly what i was hoping to see of like oh cool more

stuff so yeah i'll throw some links out here yeah we'll have those in the resources um all right so so here's here's another thing that is pretty this is actually a very big selling point of open telemetry so they they have all this this stuff that we're going to get into some of the details on that. We talked about the serialized and deserialized. Well, there's a ton of different formats that open telemetry supports, but the default actually comes from the W3C, the web consortium trace context. So they are, they are actually using a well-defined published standard for the web, for the type of, of, um, formats that they're doing for this data. And we'll have a link to that there, because I'm sure that if you ever want to take a nap,

the W3C is where you can go to start reading. mean seriously um all right so with that i think outlaw you need to do better on the bag sir wait you're calling me out like i'm the problem jay-z jay-z let's do it what you got all right so you know we i guess we're out of five star reviews so if you have any if you have any stars left in you we'll take whatever you got whatever you got but feel free to round up you know but we'll take it take your one stars around we did try to make it easy for you though so if you go to cookingbox.net slash review and you know uh you can actually just start clicking the stars and just you know keep clicking them until until you're out of them.

Hopefully you make it to five, but we'll take it. That'd be pretty amazing. And if it's under three, maybe just don't. So I don't know. There's a spectrum. I know it's tough. This is why I don't do the beg. See?

Well, we'll see how it goes, though. So if you like this beg, let us know by leaving a review. Try to make it easy. Go to cookingbox.net slash review and let Alan and Outlaw know just how much you like this one. This is why we stopped asking Jay-Z to do the bag. I totally forgot. The crazy thing is I wasn't saying that as in I wasn't going to do it or that somebody else had to do it. But it worked out so well.

Yeah, right? Yeah. So yeah. Just open your toe. It's amazing. So we should probably change the intro to like, you're listening to your favorite two-star review show. That's awesome.

All right. Yeah, there's some cool screenshots here, by the way, that I put some links out there for you guys. They're showing the Jaeger UI to show that. But this is all coming from that demo screenshots portion of the documentation where they're trying to show how things are working. And definitely some um you you were making the reference to the peer uh you know processes in there i mean it's it seems like it's apparent here if i'm following this right but maybe i'm not

yeah that first screenshot looked good and so does that third one wait have you gone to the the dag uh that's what i'm looking at i believe the load generator with the front end and all that stuff yeah yeah yeah i'm assuming that's supposed to be the directed acyclic graph yeah yeah ha ha ha look who got the math stuff wait it's not whatever uh okay well so that's my trivia for the night. So how about you guys want to play some Jeopardy? Let's do it. This is Coder Parody.

Coder G. Coder G? It only gets sued. How would Coder G? I don't know. It's pretty close. Coder D? Code.

Sounds like it. Coperty. I don't know. It's pretty close. Coder D. Code. Sounds like it. Coperty. Okay. Whatever. All right. What episode is this? 217. Alan goes first according to Tutko's Trademark Rules of Engagement.

So your choices for categories, sir, are you're going to have fun with this first one star trek the next generation because alan is so good at pop culture yeah man answer in the form of an abbreviation the patriot act with author itty so like authority but like break apart yeah yeah that's not gonna work for me going green or pretty little lies pretty little lies uh the second category whatever that one was answer in the form of an abbreviation yeah let's do that one i want to go 300 300 level yeah three yep it was once largely a military abbreviation helping coordinate operations now it means when i think i'll make it to the kegger what in the world do you know this joe uh no i i mean asap is the only thing i can even think of

i mean yeah i got nothing so you can go with his answer i will give you i i too was stumbled on this one i was thinking stat would have been my like i need that stat i remember there was like an old movie that like made fun of that where like once the boss learned to use that everything was that it's like how you doing stat yeah great all right well i'm sorry to inform you guys you both got a zero on that one big fat goose egg eta eta oh geez okay yeah that uh That's good. Yep. Yep. All right. So, Joe, your turn.

Categories are turning into the 90s, the U.S. Constitution, world capital attractions. No. Memoirs. High finance. No. Figures of speech. No. The first first one turning into the 90s yes which level uh three sounds just like it yeah medium confidence three out of five all right Medium confidence. Do we have five? All right. Listen to all y'all. It's a sabotage performed by this rap trio at the 1994 VMAs and blowing the roof off of it.

Listen to all y'all. Who's the Beastie Boys? Beastie Boys. Doggone it. Beastie Boys. That's right. Beastie Boys. And Joe is on the board.

Man. Alan. Can I get it turned into the 90s? Yeah. Well, I tell you what. I'll give you these categories instead. Maybe you'll like these better. You ready?

Let's see. Medieval architecture. I almost couldn't get that one out. That was a straight face. Classic TV. Directional geography in America. I could almost I almost couldn't get that one out of the straight face classic tv directional geography in America I swear I did not like pick these these were there it's already there out of this world and this one is pick a quote card c-a-r-d will appear in each response dude give me the last one for one or well the last one is the chase and it's uh brad rutler of the chase it's his show and that's his category so i

don't know that you're gonna want that one but i don't want that one the next to last for one pick a card for one yeah Yeah. The card for one. I would have totally picked classic TV. I don't know about you, Joe. I would have, that's, that's the way I would have gone.

I mean, what blue acres? Like, or what is that the name of it? I can't remember. Well, before our time, no way.

We're going to know that one. Thanks, Mr. I do. I would do all those. So yeah, the Nickelodeon, right?

Leave the beaver. Yeah. The card card for all right here's this wrapper i don't know how we're going to do this one is this a visual oh here's this wrapper where wait yeah because i think there's a link there oh man how's that gonna work okay i can give you a link no way that link isn't there dang it uh i don't know that you're gonna get link isn't there dang it uh i don't know that you're gonna get this i'll say it and if you don't know it because you don't have the visual because it's a visual one then we'll move on here's this rapper wearing what could be described as bodak yellow so no oh no there's no way i could yeah i gotta say all right let's do two two okay to the u.s government it's officially known as a permanent resident card.

And it's got a card in the answer. Green card. Bing, bing, bing, bing, bing. Yeah, I was like, man. We have a tie game. Look at this. All right. We go to final Jeff.

Final blocker. D who is the, who's the Bodakula? Oh, uh, it was Cardi B. I don't know. We got net.

Nope. I thought there was a chance because that was the only reason why I said it. Cause I was like, well, it's rapper and it's card has to be in the name. Cardi B. I don't know. She looks like, okay. rapper and it's card has to be in the name cardi b i don't know what she looks like okay okay she's right behind you okay so here's your final if here's the final

question um wait how are we going to do this how did we do the final we're going to ding in one of us will ding in oh is that how we did it last time yeah that's how we're doing it this time. Okay. Well, I mean, you know, cause I think that it was in your favor to go first. No,

I guess it wasn't. Cause this would be the fourth round. Yeah. Yeah. 19th century people. So put on your thinking caps. Cause here we go. Wager at all. In 1863,

Walt Whitman wrote that this politician has a face like a Hoosier Michelangelo, so awful ugly it becomes beautiful. 1863 walt whitman wrote that this politician has a face like a hoosier michelangelo so awful ugly it becomes beautiful abraham lincoln and that is the correct answer you didn't buzz that's why i was gonna ding but i have to ding you because you didn't answer in the form of a question. It was Lincoln. Yeah, baby. I'm on a one game winning streak. Easy for you to say.

The drawings. The drawings. So, okay. I was just joking about going to the drawings but oh okay it does it does quite visually show you though what the spans look like if you if you go to that those screenshots to see and like like you couldn't already do this in chrome dev tools but it's you know now you're seeing what it shows how it looks in Jaeger. Excellent. All right. So now we're going to get into the like more of the nitty gritty type bits of the information that go in. So spans, we've mentioned it quite a bit, I think, even between the last show and this show.

So we talked about the traces and all that. But spans are kind of what make up the meat that goes into these traces. So they represent, like I said earlier, a unit of work or an operation. They are the building blocks of the traces. Without them, you basically have nothing. But here is the important bit of information that comes into a span. You have the name of it. This is a friendly name that you give it, right? Like a shopping cart or ad item or whatever. You name it what you want it to be in their example it's the uh path to that particular api call like whatever that web

you know so like in their example it's slash v1 slash sys slash health yep and it it can be that or it can be whatever you want to put in there. Like it's truly up to the developer to, to provide what they want that thing to show. Um, the parent span ID. Now, if it's a root span, like this is the very first span in the entire trace, there is no parent ID, right? So that indicates that this is the top of the, of the chain. Um, the span context, this is going to have some more information in it. We'll get into that here in a second. It's got a collection of attributes. It's got span events, span links, span status. And the last little bit here is that I mentioned it earlier. They can be nested, right? So you can have spans that are nested under spans. And if you have that, then basically you have a child operation. Wouldn't the other span's parent ID be the previous span in the case of nested spans?

I don't know. It might be that you can actually put the span inside the span in your code. I don't know. I'm not sure how that'd work. I think when you have the parent span ID, it automatically links it together for you and builds that tree out i don't know if it's saying here that you can actually embed another span inside the span that you're working on you know what i mean it doesn't it doesn't

look like you would it looks like from looking at that sample that the parents if it was a span within a span it looks like it would just be the parent ID would be the other span. It'd be linked, okay. Because looking at, you called out the span context, for example, right? And so it's kind of weird because some of those other things that you called out,

there's specifically a name value pair or a property in this JSON for that thing, right? Like the span ID or the parent ID. But in the case of the context, there's not. At least in the example that they gave, there is no span context object. Okay, okay. So here then, let's get into the span context.

Well, what we talked about just a second ago, remember, was was the span so the span itself can have the parent span id oh now when we're talking about the context itself these things are completely immutable like once this thing is set it cannot be changed so it contains the trace id the unique span id for that span they also talk about trace flags i'm not exactly sure where this comes from i'm guessing that the collector or something kind of builds this but it says it's a binary encoding containing information about the trace it has the trace state which is a list of key value pairs that can be they can be be vendor specific, right? So like whatever that vendor decides to put into the trace state, they can. And this data inside the span context sits along the distributed context and the baggage, which we're not talking about those in this episode. I think Jay-Z brought up baggage in the last episode, and we'll be talking about

that here in the near future. So I did want to close the loop though it it does i was i was right span it says spans can be nested as implied by the presence of a parent span id okay so that's perfect that's what we were talking about earlier and like that span id is part of the span context so that's why i was saying like there is no like one object or property in the JSON for it. Like it is it's a collection of things. OK. So, by the way, I did look to see like what those trace flags like, what kind of stuff there might be in there. And it looks like right now the only thing that supports is a flag basically saying whether or not it's sampled.

So whether, you know, it's one of whatever or not, but they're leaving room there for other additions there. They might have in the future. Okay. Did we talk about sampling the last episode? I don't think we did. Maybe. I don't think so. Well, I think we probably did.

We've probably covered, maybe we covered sampling as part of like the Google SRE book or something. Maybe. I mean, I don't know when we'll cover it in this, but, but open telemetry does support sampling so that you can do it randomly. You can do it based off conditions. You can do it based off all kinds of things, right? So, so that, so that if you have like a million events coming through in a 10 minute period, you don't necessarily have to be adding trace logs for all of them, right? Because

it can actually be a little overwhelming. They do have something like that. I don't remember if it was for traces or if it was for metrics, but like I said, we'll come back to that here in the future. Well, there's a blurb under the sampling portion of the documentation that says the idea behind sampling is to control the spans you send to your observability backend resulting in lower ingest cost. Okay, cost okay so rather than sending like if you were thinking about google for example right like they're not going to want every click from every single user around the world being ingested into this like uh open telemetry system so they can see what's going on so they might say like you know what just give me

every like ten thousandth click right on this particular thing, right? They're just getting a sampling of it. Or if this particular criteria was met or whatever, right? So that they can get the interesting thing. So yeah. Excellent. All right. So the attributes that are part of the span. So these are key value pairs that are used to carry information about the operation it's tracking, right? So it can basically be whatever you want it to be. They did use an example in there. So you might have the, if you were in a shopping cart, you might have the user ID, the item ID that was added to a cart and the cart ID, right? Those are all reasonable things that you would add to the span.

I think we should recommend about not putting the user ID in there because that seems like it would be a disclosure of information you might not want. It's funny. When I saw that, I thought something similar like, ooh, is this PII type stuff? Yeah, I don't know. It's tough. Well, for the record, I don't think they included that in their documentation did they yeah it was in there oh yeah but you know i don't know sometimes you have it in your log so i'd assume if it's in your logs it's probably safe to go in here i mean you'd have to be careful about it you know know your user data and whether or not it's important enough to keep that stuff out

of it i'd call it out. Yeah, probably. They do have some rules. I mean, so to, to outlaws point, you could probably keep that out and you could probably, if you needed to, at some later point, you could look up that card ID, you know, programmatically and figure out, you know, who it was tied to or something like that. You know, there might be other ways to do it where you're not burying that stuff in logs. Oh, your keys cannot be null that makes sense right like why would you have a null key doesn't really make sense to have it in there so this is kind of interesting json even allow that i never even thought about that

like why would i don't know well remember this isn't necessarily json right like these are serialized objects in whatever framework you're using, whether it's C-sharp or... I guess because their sample is in JSON, I just assumed that it was JSON across all the things. Yeah, they show us JSON. It does not. Yeah, because that wouldn't make sense. Yeah, it doesn't.

I mean, it doesn't make sense. Equal this value. Yeah. Just a colon out in the middle. Just hang in there. NOLA's five today, right? Yeah, it doesn't make sense. Equal this value. Yeah. No one out in the middle. Just hang in there. Null is five today, right? Yeah. I don't, it doesn't make sense.

Values though. They must be non null strings. If you're going to have a string, a Boolean, a floating point value, an integer, or an array of any one of those types. Right? So you can have an array of strings, but they can't be null strings. So it is interesting that they call these out specifically and then here's the browser by the way does it really yeah if you had a dictionary or or um a collection with a null as a key yeah and i could do like you know my collection uh brackets null and get my value back that's fun i'll do that

yeah javascript for the win. Of course. Of course, JavaScript. Just like, eh, whatever. You probably meant to do it, and I'm not going to judge you for doing it. Right, right. So here's one thing that's really cool, and I like this, and I think I even had a link. I do have a link to it.

So they have semantic attributes for well-known attributes that you should just use, right? So these are standardized across systems. So I had a couple examples here. So like under general attributes is what they called them, which I thought was interesting. There's a server.address, a server.port. For databases, there is a db.system, a db.connection string, which is probably also not a great thing to put in there, but they do have a whole set of these things. And we'll have a link in the show notes to where if you just decide to use this stuff, you'll leverage those things. Because if you do, then any other software that you're integrating with it, whether it be Jaeger or something else, right?

Like they may have special reports and views set up for these types of things, right? Like that's the beauty of using a standardized set for these. I would be curious to see if like their connection string. I mean, I don't know. Did they have a sample? I think they just named it it i don't think it was in there well it does say it is recommended to remove embedded credentials okay yeah that makes sense this is this is uh like just generally speaking right yes whenever some entity, be it a person, company, whatever, puts out like, hey, here's some free code or whatever, or here's an example of how to use this thing.

But it's like they give you an insecure thing by default or some kind of bug. It's going to be hard. it's going to be hard people are gonna be hard pressed like like you'd have to go and like dig through the documentation or to know like oh this might be a bad idea right you could you could easily see where like uh you know a junior developer like you know first job right might think oh hey it i can log the connection string right sure that sounds that sounds like a great idea. And it might not dawn on them at the time that, like, oh, yeah, that could also include credentials. I kind of wish that they didn't include that as part of their, like,

standard attributes. And instead it would be like instead of even, like, risking that, like, they could possibly recommend that you do that, I think that they should take the secure by default kind of posture and instead say like, okay, you can log the database host that you're connecting to, right? And maybe it's like some kind of specific protocol or whatever if you had to use one. But don't just have this blanket connection string where you can potentially risk putting information in there that you really don't want to. Well, to add to that, the next one, if you're on this page for the database connection level attributes, the semantic conventions. The next one's db.user. Yeah. I would also argue that you probably shouldn't even have that in there because if somebody, a bad actor gets into the system and they find these logs, then they already have half of the information, right? Like they now know, okay, well, here's the server host. Here's the username. All I got to do is try and do some sort of fuzz or whatever to try and,

you know, some sort of dictionary attack to try and figure out how to get in there. Right. So yeah, I mean, I guess this comes with more senior level experience with this type of thing where you're just like, Ooh, yeah, I don't think, I don't think I want that in there. Yeah. I really think those two, I think they should remove those from their recommendations of like here's some like common ones but you know even even the port even the port is something that you look at and you're like oh yeah maybe not i mean i i can get behind some of these though because at some point you're like well you know the whole thing is logging like you're you're trying to log like what's going on right right and and not only log it but like

you know the telemetry like there's metrics behind it all that kind of stuff right so so at some point you have to just say like okay well if you've gotten into here and you can read this then you you might already have too much access but like you know the ability to like scan ports and whatnot then you know because you're already telling them that the DB system is identifying what the type of database is. So like SQL server or Postgres. So you're like, oh, well, Postgres is typically on 5432. Right.

You know, right. So some of those I'm okay with, but like the, the, the, the fact that like, you know, it's just, it's not even the first sentence., the fact that like, you know, it's just, it's not even the first sentence. It's the second sentence. It's recommended. Not even right. Right.

It's not even like you should, or it's not even in bold. It's just, it's recommended. You should include the connection string. Right. It bothered me when like people, things get put out there into the world.

They're like, you know, insecure kind of recommendations. And you got to dig for the details. It's like, Oh yeah rant over it's it's difficult a one other thing to call out here though is it's not just these top level attributes that they have these conventions for so like db. system was when i called out well they also have a whole list of db system um you know constants i guess we call them or whatever that they that you should use right so if you're using a microsoft sql server it should be mssql uh if you're using apache hive then it

should be hive whatever so so it's not just the top level key they might also have a value that's supposed to be in there um for these logging things and again who knows what they're going to be used for by other tools or whatever but but just be aware that they're out there. Yeah. Cause I mean like the problem that, you know, not to harp on this too much, but like the problem that I have with this, like, even if you have like good actors in your company, right. That are going to, that would be reading some of the output from this thing. Right. You know, one of those people could become a bad actor as soon as they feel, you know, slighted.

Right. You know, and then you have the inside insider kind of threat type of situation. Right. I mean, they happens all the time where like an employee leaks data, right. That they shouldn't have.

So that's why I'm like, I really wish they hadn't said that one. So if you're using this, don't, don't do that one. I will say though, I mean, they hadn't said that one. So if you're using this, don't do that one. I will say, though, I mean, we haven't gotten this far. I don't know that I've even seen anything in the docs yet. I think that the information that you'd be seeing in a lot of this output is probably something that you'd want locked down a little bit.

You know, you wouldn't want necessarily anybody to be able to get to these very yeah very detailed logs right that are tying everything together you probably want to lock that down to a certain group of people that are you know troubleshooting production or whatever the case may be but even if okay even if you were troubleshooting production and it's your own app that doesn't mean that i want you to know the credentials oh to get into that database never be in there yeah they should never be in there. Yeah, they should never be there. Right, right. And the connection string would have that. And that's what I'm trying to make is the point is like, it's your application.

But you know what I'm saying? Like in a large enterprise, it might be your application. That doesn't mean that you know the credentials. It doesn't even mean you get to touch it when it makes its way to production. It's all done through automation. So, you know, I definitely don't want to leak, you know, Oh yeah. And by the way, here's the credentials that you cannot use. And I can go and trust you. Hey, now in fairness, you should not have your credentials in your

connection string either. Right? Like almost every database system on the planet gives you a more secure way to connect. So, you know, maybe that's a smell that you should look at too, just calling it out. But, but his point is 100% valid, right? Like when you're logging stuff like this, you want to be careful about what's actually making it into the logs. All right. So the next section, again, these are all parts of the span, right? So the next one up is span events. So this is a structured log message or annotation on a span that is usually for a meaningful point in time in the span's duration. So this is important to call out. So a span has a beginning and end, right? Let's say it was some sort of shopping cart request, right? Like it started one second ago and it ended a second later. That's the span. The span event is more like if you have a log that has a timestamp on it, right? So they gave examples of this to sort of help separate it in your mind. And these are

both to deal with web browsers. So tracking a page load, this is a perfect example of a span in action because it has a start time when you started to load it and then an end time. That's when it finished. Right. So it could have been a second later or whatever. Denoting when a page becomes interactive. So if you've worked with any of the latest JavaScript in the world, right, your page load, but you can't click anything. Right. Like, I don't know. There's a million websites that come to mind, like the pages there, but you can't actually type anything or do anything yet because all the JavaScript isn't ready. That is a singular point in time where it's like, you know, we have enough stamp it. It's ready to go. And a lot of times this is a metric that will be used by people who make websites to say,

OK, my page load took five seconds, but the user could actually start interacting with it half a second in, right? Like that's fine to them. So good example of the distinction between the two. Span links. So this again also falls within a span. This one was a little bit less clear to me, at least reading the documentation. I think I got it after it was all done. And maybe this will translate well to you.

So this allows you to associate a span with one or more other spans, indicating a causal relationship. Now, the reason why this wasn't super clear to me is because I thought the parent thing was it right. But I'm guessing this is more like, Hey, there's other spans that are doing things that are sort of related to this. And you can only have one parent, right? There's only one thing that sort of spawned you as the next line in the span. Whereas this allows you to link to other spans and saying, Hey, you know, these were all part of some of a similar operation. And they gave an example is you have a system that cues actions based off other actions in an asynchronous manner. Okay. You don't know when the cute action will start. So you create a span link that can correlate these spans over time when the asynchronous event occurs. So in my mind, something like you added something to a Kafka queue, right?

There's something, there's a consumer out there, maybe many consumers out there that are listening for those events in that Kafka queue. And at some point, that's going to cause some other process to kick off, right? That's when the span link is created so that you can say, Hey, you know, this, this thing was because of that. So, okay. It seems to be an asynchronous thing, more or less. You know, I'm glad you, you said the causal, causal relationship because at first, like, as I was like trying to follow along and along and skim over that real quickly,

I saw it as casual. It's kind of a relaxed relationship. And I was just like, okay. So these are optional, but like I said, they can be a good way to actually associate things. And I guess the asynchronous, now that I've repeated it after I've taken the notes, it makes more sense to me. So service A calling service B, service A is the parent, service B is the child. That makes perfect sense.

However, if I just put something in a queue, I don't know what's going to act on it, right? I have no clue what the next actor is because I didn't call that next thing to run. I stuck it in a queue. Something decides to process that message in the queue. Now you have that causal relationship. The consumer is what knows that this thing had a span on it. And you can say, Hey, I'm going to link to that, right? So that makes more sense to me. It really helps to think about this one in the frame of mind of like a large system like if you're still thinking like a browser web page kind of example right this this is more difficult to like wrap your head around in my mind but using the q and the kafka example that you gave you're like well i don't know which broker or which system

which you know like a flink, like you might have a flink, uh, you know, task with like a lot of paralyzation. I don't know which one of those task managers is going to like take that particular job on. Right. And so now you want to create this new trace with the span link to say like, Oh, I'm starting because of this other thing that happened. Yep. And, and, uh, make it, I guess, even a little bit more complex, but to demonstrate why this is powerful, the span link is powerful, is you insert something into a Kafka topic. The whole point of Kafka, we've talked about this a little bit in the past,

is it's sort of like your backbone for triggering things with any type of system that wants to integrate with it. Right? So you might put a message in your Kafka topic and it might be that Jay-Z's program is going to do something with that message. Outlaws program is going to do something with that message. And my program is going to do something with that message. So now you can actually have three span links back to that one thing and it'll show that hey there was multiple things interacting with the same message that got thrown into a stack so it's it's pretty it's a pretty powerful thing now that i look back on it well you don't realize though is

that my process is basically like the wario of processes here and it specifically looks for your two processes and does a kill on them it snipes them that's good that's beautiful all right yeah i guess though yeah like it's possible that the queue gets paused for days even you know so it's kind of weird to think about keeping that span open because you've got a child out there still right and you can't right like you couldn't keep that thing open forever and that's that's one of the things they call out is well after the thing has finished producing right that span has closed quote unquote these links can still be created all right so the next thing that we have up were you about to say something about it nope yay no nay all right i guess you're maybe if you were to think about this in terms of an

order or the e-commerce kind of example right like the mere fact that you shipped something like that whole process like the spans and traces that it could create like maybe you could link those back to the initial it could be a span link back to the initial i'm trying to like make this tangible i actually like that a whole lot i mean if you i believe years and years ago i believe that the way that Amazon operates its order stuff is it does go into a queue, right? Oh yeah. Basically so that they can do reprocessing, right?

If something fails, when it tries to hit that thing in the queue, it can go back to it. Um, so it's, it's a perfect example. You place the order, that thing has its lifespan is going to end in a few seconds, right? Like it's done when it gets picked up in the queue, whether it's today, tomorrow, whenever it can create a link back to that original order thing, which is,

I think beautiful. Or, or maybe you're a visual and you remember the fifth element and you knocked the glass off the table and the little robot comes and vacuums it up. Oh God, I do not remember that. I thought I remembered that movie, but now I don't know. It was the whole thing where, who's that actor, Gary Ullman, I believe, was?

Bruce Willis? No, no, no, that played the antagonist in the movie. And he was going on this whole speech about like uh you know these things to have their whole reason for being is to like you know that little robot's whole reason for being was to clean it up and so by him creating the mess he was giving that robot purpose okay his memory or i guess it wasn't really it was actually like a little animal for anybody that hasn't been with us for a long time on this on this journey of 10 years of episodes outlaw has the most incredible memory for movies and shows that he watched you know 20 years ago because he's only 21

i've i've never in my entire life i don't know or 36 you know whatever right yeah yeah i mean it's been a while it's unreal like he remembers full-on gi joe episodes from his childhood like how how does that happen i don't know you know okay tangent alert tangent alert so here in the atlanta area one of the big events going on is definitely dragon con like that's absolutely what did you do that's what you were going to think when i said that right so there was something big going on in atlanta yeah and and so we're at the parade this morning watching watching the the parade and everything and like as you know like different things that come by. I'm doing quotes from the movie. There was one there was a whole

Coming to America group going through their parade and everything. I rattled off some quote and the guy next to me was like, what? That's great. Coming to America. Do you remember the baby, he's got his own money

i mean baby he's got his own money no you don't remember that scene when did you last watch that movie see this is this is the key here to understanding i honestly i couldn't tell you but that's what i'm saying he watched it last night it's when mcdowell realizes that uh who who's mcdowell he he was the the dad that had the the restaurant that was like not mcdonald's right they didn't have the golden arches because that would be that would be copyright um it's ridiculous i'm telling you right now but oh yeah i mean that like everything you said there is completely foreign to me yeah when he realizes when he realizes that uh akim is actually a prince and he like sees one of the the uh paper money that you know from his country that actually has his print prince akim's face on it that's when he's like baby he's got his own money there's

the whole scene like i don't yeah near the end of the movie, fast forward, you'll get there. He remembers where in the movie it is. Yeah, it's amazing. I know that there's a movie. I can't even tell you the plot. I feel like I've seen it more than once even. I have too, right? It's Eddie Murphy.

You don't know the plot? Come on. I feel like I'm being egged on here. This is a joke. No, it's not a joke. I can't I'm being egged on here. This is a joke. No, it's not a joke. I can't tell you the plot of shows I watched last night. Yeah, I agree with that. All right, I'm starting a new show.

Hey, for what it's worth, Hard Knocks, I love Aaron Rodgers. He might be one of my favorite people on the planet now. So, yeah, you should watch the latest season of that, even though you guys don't like sports ball so anybody else out there who actually cares at all about football should watch the latest season of hard knocks on hbo all right back to it back to it sorry okay i remembered that because i watched it last night it'll be gone in a week so you know here we go span status so this one's pretty interesting again going back to the span this is a subsection of attributes on that it's attached to the span and it can be one of three values so this is pretty

easy it's unset this is usually what you want to do you don't set the span value you just leave it be it can be error which in the case that in your code in the section that's going to be adding to the span, if you actually caught an exception, you would set the span status to error to let it know that there was a problem, right? Like that's on you in the exception code to do that. And then there's OK. back end that processes this, and I assume this is going to be the open telemetry collector, whatever that thing is, it should set this for you as the final status, assuming there was not an error, right? So let's say that you had 10 spans in your, in your request, all of them had unset statuses at the very end. It's going to say, Hey, the status was okay. Right. My assumption is if you had those same 10, um, statuses and one of them was set to an exception, then my guess or error, then my guess is then that span status at the

very end will be set to error. Right. So just know that this is sort of wrapping it up and putting a bow on it for you at the very end to let you know if this thing succeeded or failed for some reason. Which I think is another benefit of this, right? Now we have Spankind. This one's for all Spankind. This one's kind of interesting. One small step.

That's right. Do you remember that quote? No, I'm just kidding. I do remember that one. So I actually like this. Do you remember that quote? No, I'm just kidding. I do remember that one. So this one's useful for the context of how things are happening as they go through a span. So when a span is created, it is created with one of the following types. It's either a client, and a client represents a synchronous outgoing remote call, right? So if your web

browser is making a call to the web server that is your client call now this it says it represents a synchronous outgoing remote call it doesn't mean that it's not asynchronous right so and this is it sucks that they kind of had this overloaded tech or method or talk in there. It just means it's not being queued for later processing, right? So think about a web, your web browser, you make an async call to your web server. If that thing is doing some action right now, like reading your name out of a database, doesn't matter that the call itself was asynchronous. It's the fact that you made a call that was waiting on a response from the server to do something right then, right? So that's a client. The server, if the spam kind is server,

then that represents an incoming synchronous call, such as an HTTP request, right? Standard type stuff that we talked about a million times. Internal. These are operations do not cross process boundaries, things like instrumenting the function, right? So you have this big function in there and you're setting timers to find out how long things are processing. That's all internal. Doesn't require any external stuff. A producer, spankind, they represent the creation of a job that may be asynchronously processed later. Things like messages sent to queues or handling of events. So that was like what we were talking about. If you were throwing a message into Kafka for a topic, then you are a producer. A consumer. This is something that represents the processing

of a job that was created by a producer and potentially starts well after the producer span ended. So this is like what Outlaw was talking about a little while ago with the shopping cart example, right? Like, hey, you placed your order that went into a queue. Whatever picks up that record in the queue to process later is a consumer. And what they said is these types provide a hint to the backend spam processor as to how these things are supposed to be assembled. And when it says that, like that's a very generic, vague statement. My assumption is it's going to sort of show things in order, right? So they said right here, based on the open telemetry specification, the parent span of a server span is usually a remote client span, right? That makes sense. Somebody did something in the browser, it hit

the server. So that client span was one, the server was two. The parent of a consumer span is always, always, always, always a producer span. So my assumption is it's going to take these and stitch them together in an order that will make sense for you when you're, when you're trying to view the output. You know, the way I interpret this whole thing is basically like, think about like, if I wanted to create a, a database system, right. That you could query a relational database specifically, right.

That you could query it. And I need to get data in for it. Right. And in order to do that though, like I'm going to have like all these other systems that are going to be, you know, dumping data into it. So I have to give you like some kind of a structured format that I want the data in.

And that's what this is. All of us talk about spans and the tracer and all that. That is, the span is the structured content of the data. And I'm going to then use span IDs and parent IDs and whatnot to be able to do those relational lookups to say like, okay, here's everything with this parent ID or, you know, and you know, maybe I like do some recursive CTE to like get all the, you know, nested ones, you know, that's what this,

that's what this sounds like to me, especially when you talk about the, how did you refer to it a moment ago? The span – not consumer. The span processor or whatever I think is how you said it. I think so. A second ago. Because you're basically – imagine if I gave you this – let's keep it in json format for the time being if you cap it in um json format and you're just dumping all of these records out there right like you know some other process is

then going to go and uh you know assemble all that together into this quote database that i was talking about and in this case like it could be Jaeger or whatever, right? Like that's, you know, and there's from their screenshots, you know, kind of thing. That's the way I'm interpreting this in my mind. I completely agree. I mean, if we go back to the span links where we talked about things could be asynchronously thrown into a queue and then picked up later, like I would imagine if it was Jaeger or whatever, if you went to that UI right after an order was placed, you're going to see that the order was

placed. If you were to come back tomorrow and look at the same thing, you'd probably see that the order was placed and that it was picked up by something, right? So, so that, that spam processor is going to continually stitch things together as it gets the information so that your view of these events over time might actually grow, right? Because there's going to be more things that are adding values that that spam processor is going to put together for you. The only problem that I have with this whole thing though, is that open telemetry has its own baggage which i'm just kidding if you look in the documentation there's like literally another of another section

called baggage which i'm assuming would just be a way to like add uh you know you're dumping your own information specific to whatever that thing is but i just kind of found it funny like yeah of course we've got our own baggage so uh yeah so we'll have plenty of links in the resources we like section for uh this episode and open telemetry and all the goodies here um including the screenshots that i was talking about are going to be in there and with that we head into alan's favorite portion of the show. It's the tip of the week. So I had no, no, I said, your lips are moving.

Yes, my bad. Um, I can't hear what you're saying. So, uh, I went from having zero tips to three tips and, two of them are totally stolen.

Uh, the first one, uh, by the way, uh, these stolen from our comment section on the website, uh, coding box on it.

Uh, so the first one is from, uh, I'm going to butcher this. i'm sorry uh alexander andre yes key according to google uh and andre said or sorry alexander says if you're interested in tldr.sh which i was i think it outlaws tip of the week last week yeah there is a faster from the slack channel yeah so we're we're gonna do one more evolution on it so uh there's a faster implementation uh in rust that uh is an open source project we'll have a link there in the show notes uh but it's pretty cool they call it tldr you know like tldr tldr uh it's just written

in rust and um i was surprised to see that they've got a comparison chart of various libraries that all implement tldr i guess it's kind of a popular thing to to write and so let me see here uh if i if you click this link and look at the chart i think the um the speeds are pretty interesting oops uh so i kind of blew out your second tip there yeah i messed up the notes sorry uh but um there's a speed comparison of just like various languages and the fastest one is zig uh however uh it doesn't support all the features it's missing a rather large one for user configurations uh so that one was kind of funny but it's mean uh time for 50 runs of uh you know this test data is uh 9.1 milliseconds i'm going to compare that to the slowest one listening here

which is the node client javascript which is 400 milliseconds which is the one that was like by default that was the most stable implementation from the uh the project that I talked about last time. I'm assuming that's the one that they're referring to because at least from the TLDR SH documentation, they referred to using the Node implementation as being the most stable. Yeah, it looks like the bottom three are all official implementations, so the one for C, Python, and Node.

And the C implementation is 38 milliseconds. It was just kind of crazy to see just how far off it was from Zig. But TLD was the second fastest one, but it does actually have feature parity. And its mean time was 13.2 seconds. So I was surprised to see that so many people have kind of taken this on and written their own implementations of it. I thought it was really cool. but can we talk about the real elephant in the room though sure what the heck is zig right yeah you know i've seen it around it's

like one of those kind of like cool languages that people do stuff with like hacker news like if you do your startup in zig like you're gonna hit the fourth page you know it's wikipedia says it's an imperative general purpose statically typed compiled system programming language designed by andrew kelly but yeah i'd never heard of this one i feel like anytime there's a language that's like somewhat popular i haven't heard of it's based on lisp so let's see no i mean it's intended to be a replacement for c with the goals of being even smaller and simpler to program while also offering modern feature

new optimizations and a variety of safety mechanisms while not as demanding of runtime safety as seen in others. Other languages, that is. Yeah, it looks nice. I would have a hard time doing it. I feel like Rust has at least earned its wings at this point.

Like, it lasted more than a year or two. Yeah. There's a lot. Yeah. A lot, too. Like, not even just the language, like the ecosystem, like onboarding, like organizations need to use this thing so I can actually use it in the future. Yeah. Yeah.

I don't know. I've always had a hard time with this. I mean, we've talked about this with JavaScript frameworks in the past. Like, I remember when Aurelia looked really nice, but was like and there's like there's like 10 people using it like i can't nope i mean i'm looking sorry go ahead that's one thing i was gonna say it's not enough to be a little better than your competition you got to be a lot to get people to switch a lot a lot lot yeah i mean i'm looking at the the language page for it looks it looks interesting i'm not going to say that it doesn't but in my mind i was kind

of thinking like okay you know you could totally bring this in to a repository like in a large you know like let's say you're in a mono repo kind of culture like a large enterprise you could totally bring this in and you know like have a docker kind of builder that like bakes in all the their dependencies for you but you know then like you have to come along to support this that you know alan introduced it and didn't tell anybody. And then Jay, he's like, Oh, Hey, there's this thing. Wait,

what am I looking at? Yeah. I had a small change to make and now I'm learning a new language. Thanks. Right. Yeah. And this thing is three years old. Yeah.

That's, that's like, I hope this takes off. I mean, it sounds like it's got some pretty cool, uh, you know, ideas behind it.

I just don't... I'm surprised that there's already an implementation for TLDR in this language that just came out. Well, actually, so I take it back. It's older than three years old. It looks like it became official in 2020. If you go back and look at the release tags on the on the github its first release tag 0.1.0 was october 2017 so it's been around a little bit but it looks like it started years

yeah it gained some traction in 2020 became it it became official but yeah i don't know i mean and i don't want to shoot it down i mean it may be absolutely amazing right this is the type of thing where like i would strongly recommend in your personal projects you know that you're working on like experiment like that play with it that experimenting with other languages and whatnot in in tool sets in your free time is like the whole thing that you should be targeting with your free time. In my opinion, don't like just introduce a brand new language into your work environment without like it being a conversation. Don't do that.

Like that should be a decision. People would be angry. How sad would it be though? If like 50 years from now, a hundred years from now, people are still using the same languages because otherwise, you know, we'll be here 50 years from now.

Yeah, that's true. I mean, that's why I'm saying, like, I hope that this thing, I hope that things like this succeed. You know, like, I'm wishing it well. I'm not trying to say don't use it at all. Oh, yeah, for sure. Yep. Yeah, I hear you.

Hey, a second tip. Oh, go ahead. Yeah, no, no, I was going to say your second tip is pretty sweet yep this one is also from the clean walks comment section from last episode so Mark Crawley wrote in and said hey did you know that you can just do

control shift escape and it's going to pop open that task manager it's great on Windows and it's super nice because usually when you're going to the task manager it's because you know you got a problem. Things are going slow. Things aren't working right. I hate when I do control-alt-delete, and I've got to wait 30 seconds for the lock screen and then 30 seconds for task manager.

It's like, by the way, I just wish I hit the power button. So it's nice to be able to just do this and go right where I want. Go ahead. That is a super old shortcut, though. Is it? Way, way, way back. Decades old Windows shortcut. I did not know this.

Hey, and for what it's worth, I always do Windows R and then just run task MGR. Yeah, that's good. Yeah, but the whole point, though, is that what if the system's locked up to where you can't even get to? Yeah. Because the Windows R is going to bring up can't even get to, yeah, because, because the windows are, is going to bring up a run dialogue that is part of explorer. And if explorer is the thing that's, that's choking,

uh, control shift escape is going to bypass that. Cause it's going straight into the event loop. So it's just going to be like, Oh, spawn up the task manager. Yeah. I like it.

Yep. Hey, I got one more frame. Uh, so this one is about a game. Now this is a, this is a niche Yep. Hey, I got one more for you. So this one is about a game. Now, this is a niche tip. You know, if you're out there and you hear this tip, if it's for you, you're going to know it. But there's a new video game out called Sea of Stars just came out a couple days ago.

And it's a neo- game. Basically it's, it's kind of like a new game in the style of a golden era of super Nintendo Famicom, uh, action RPGs, like specifically Chrono Trigger and Secret of Mana. This is like a, kind of a genre of game that has kind of, you know,

the world has kind of moved on by, you know, move on from, and there's not a whole lot of stuff being made kind of in the style. And so it's really cool to see, uh, something that kind of came out that like it took very direct inspiration from and those are like those two games i mentioned are like super popular there's people out there that you know love it and uh you know 30 years

later still we'll call those games uh some of the favorite games so it's kind of cool and i'm a big fan of anything that doesn't have like grindy battle passes or weird mobile ads or whatever. It's not going to be super hard. It's just a cool game. If you appreciate that era of games, you like the music. They actually got the composer who did a lot of the music from Secret of Mana. He was the director and did all the music for Chrono Trigger and a bunch of Final Fantasy games and other stuff. So it's just kind of cool. So if you

hear the word Chrono Trigger and the hairs on your arm stick up, whatever, then you should check out this game. Also, for fun, it's pixel art. You know, whatever. I looked up the minimum requirements and it's GeForce 520. I forgot what the other one was.

The Radeon. But I looked and that's literally the cheapest video card you can buy. You can get it for $50 and just toss them in. It's crazy. So basically, you just need to have some sort of video card to be able to play it, and you're going to be fine. It's basically like a Super Nintendo game or a Super Famicom game. It's that – I always forget what that perspective is called.

The Diablo. Isometric. Isometric. Okay. Yeah. You know what this reminds me of, Jay-Z? So you mentioned two of the popular SNES games from back in the day.

Did you guys ever have a Sega and have Phantasy Star? Oh, so good, yeah. Sega Genesis? Yeah, we had Sega Genesis. Was it the Genesis? No, this was the Sega Master System. I think Phantasy Star was on. Yeah, those are great games.

Dude, this is what it reminds me of. So going back to that, this was my favorite type of game when i was a kid yeah so yeah i may actually buy this for the switch maybe get my kids to play it yeah excellent that's been cool i like it all right so i've got two uh the first one i can't even really tell you a ton of information about it because I haven't used it. But there was a conversation that came up recently about fuzzing applications. And if you don't know what that is, it's basically where you just start trying to throw garbage at functions or methods or APIs or something. Right. And see what happens. Right. See if you can blow it up,

see if it's still function, see if it crashes in a spectacularly bad way, whatever that's fuzzing in general. Um, Jay-Z, I think you've done a little bit more research. Is there anything else you'd add to that? No, I haven't, I haven't looked into it. Um, but I mean, it's a good idea, but, uh, fuzzing is a, is an interesting topic. Like there's a lot of, uh, kindzing is a is an interesting topic like there's a lot of uh kind of gotchas and kind of caveats and kind of things like one thing that's kind of funny we're

talking about recently is that um the difference between a uh fuzzing test like not working you know not like not even running at all and it passing is like kind of negligible like because you know ideally you're uh you know the fuzzing test is going to come back and tell you that you know nothing nothing bad happened and so it's kind of funny you can misconfigure it type the url wrong test you know nothing and it'll come back and be like yeah it's good it's fine yeah no it's it's painful right for that for that reason like you really have to know what you're doing when you're, when you're setting some of these things up for that reason. Well, the reason I brought this up is if you are somebody that, that is trying to test things in a cluster and it's not clear to me based off this tool from

Google, it's called cluster fuzz, which I love the name. I think it's probably one of the reasons why I had to put it out here. It's not clear to me if this is a Kubernetes only thing. I don't think it is. I think you can set this up in any kind of server cluster because they said that you can use like 300 VMs or something. 30,000. 30,000 VMs. Okay.

So, again, I'm not exactly sure how this whole thing is configurable and all that kind of thing, but it looks like you can turn this loose in a cluster of VMs or maybe even a Kubernetes cluster or whatever and have it fuzz applications all over the place. So I'd have to do quite a bit more reading. So that's why it's a tip of the week for anybody that's looking into fuzzing right now. this might be a useful tool in your tool belt i can give you nothing else beyond that it does seem to be very google cloud uh i hate to say specific but i mean like there's definitely a bunch of places in the documentation where they were referring to google cloud Project and Firebase. Build buckets. I haven't seen anything yet specific to Kubernetes, though. Well, they have pictures.

So if you go to the architecture link right off that top link that I have. So the very first page that you go to, right underneath cluster fuzz at the top, there's an architecture link. And if you look at that, those look very Kubernetes specific. Some of those things there. Yeah. The icons. Yeah,

sure. So I don't, I don't know, but that might be a VM and not a Kubernetes thing. And that's why I'm like, man, I don't know all the Google icons well enough to be able to decipher this. Oh,

you don't remember how this game works, right? The one with AWS. Well, that that's the compute icon underneath fuzzing. Okay, so that might be just VMs. It might be Kubernetes or VMs. So at any rate, yeah, this looks like for anybody that's needing to do some large-scale and scalable-type fuzz testing,

this may be something that you want to take a look at. Okay, so it is Google-specific. Cluster Fuzz runs on the Google Cloud platform and depends on a number of services. Compute Engine, not strictly necessary. Fuzzing bots can run anywhere, so why did you put it as a requirement that it depends on? App Engine, Cloud Storage, Cloud Datastore,

Cloud PubSub, BigQuery, and Stackdriver logging and monitoring. No mention of Kubernetes. Yeah, yeah, it's interesting. So, you know, if you have a chance to mess with this or if it's something that sounds interesting, you know, take a look at it. And then this one was awesome. So this came across on X or Twitter or Twitter X

or something that came across. The other day I saw this C sharp now has a foundational certification that you can get for free. So, um, we've had people email us many times like, Hey, should I do this? Should I do this? Should I try this? Whatever. Hey, you can go get a foundational C sharp certification where it looks like Microsoft is partnered with free code camp and they, I forget, I think it was something like 30 hours worth of, of material, 35 hours, C sharp training course hosted on Microsoft learn. So you can go up there for free, do a 35 hour training.

So weeks where the work roughly and go take a test-hour training, so a week's worth of work roughly, and go take a test, and you can actually get a certificate saying that you completed the course and you understand the foundational bits. I'm pretty good with Photoshop, so they put a sample of what the certificate looks like. So if you want one, just hit me up. I can fill in name here. Outlaw for the win. Going out there. Fiverr.com slash outlaw.

That's right. Fiverr, Doug. Fiverr. I didn't even think about that. Jay-Z always looking for the angle. That's right. So I thought this was super cool, right? Like, I mean, there's tons of courses out there like Udemy.

I mean, Jay-Z and I have talked about Udemy in the past. I think, you know, Outlaws has gone in there too. There's all kinds of stuff out there. It's really good, but it's really nice to see that Microsoft is really providing for the developer community. Hey, you want to learn how to program? We'll show you how to do it on our stuff for free.

Just go take the course. It's up to you, right? Love it. So those are my two. And we'll have links for both of those things in here in the show notes. All right. So with that, subscribe to us.

No, I'm just kidding. It's my turn. Okay. So one, I thought was kind of a he, he kind of funny one. So one of our peers pointed out a site called there's an AI for that dot com.

And it immediately reminded me of like, do you remember the old iPhone commercials? Like there's an app for that. Well, oh, yeah, there's an AI for that. So, hey, you're having trouble with coming up with the name for your next website. There's an AI domain name generator. What? Like some of these are just crazy. Yeah.

You know, you're having trouble with your music taste. Well, there's one that like will critique music taste assessment and critique. So, you know, maybe if you were having problems, you know, with your, yourself worth, then that one can like, you know, make you feel even worse about yourself. Hey, real quick. So that reminded me of another one. I have to throw in here. Do you want me to do it before or after your next one? Okay. Do it after. But I was going to say like, I don't know how this one can be even legal. Cause there's a legal AI.

That seems scary, but you know, maybe you use it. Well, there's an AI music generator. So yeah, the point is, is like, if you go to, there's an AI for that, there's a chance that you might be able to find an AI for something that you, you need, you want to you want a professional headshot for your online profile there's an ai headshots for that so i don't know like how it could make this look any prettier than it already does but apparently

i guess it does i don't know so go check it out there's an AI for that and then I'll do my next one after Alan with this quick word from our sponsors yeah okay so because of this one it reminded me of another one that we probably heard on this on this the same thing that he's he got this one from there is a thing called chatbot arena and I'm going to put this uh right here so what this is is they've tried to rank how well various chat bots do at answering questions right so you got chat gpt you have llama you have i forget what the other ones are there's a bunch of them there's claude gpt vicuna there's tons of them. There's Claude, GPT, Vicuna. There's tons of them. At any rate, if you go to the link that I put there,

you can ask a question in that main tab. So the battle, if you ask a question, it'll spit out a model A and a model B, and then you can choose which one you think did a better job of answering the question, right? And then when you do that behind the scenes, it's going to store that and basically say, okay, well, when he asked this question, then we saw that this one did better. So it was kind of a blind AB test, right? But then if you look at the very top, if you go to leaderboard, you can see where it scored everything. And so you have these MMLU scores, which is like the fourth column over and the second column, also the arena ELO rating. These are all described up at the top of the page, but you can see how well people have scored these things in these blind AB saying that, you know, I think this one did better than this. And I, I forget, I want to say that there was

some sort of disclaimer that chat GPT is sort of like the creator of this thing or whatever. But behind the scenes, it's supposed to be based off these blind AB tests. And you can see GPT-4 is at the very top of the list. And it's pretty good. But then this Claude 1, Claude instant one, Claude two, those are right on its heels. So,

I mean, chat GPT isn't the only player in town, right? I think is what it boils down to. But, um, yeah, I don't know.

It's fascinating. Did I, I, I chose, I, so I put a different link out there cause yours included a query string parameter. Uh, and. Oh yeah. So I put a different link out there because yours included a query string parameter and mine would never load based on that.

So I wasn't sure like, is this the link you meant? Hold on. Let me see. Yeah. I think when I clicked on leaderboard, it added it to the, Oh, that's weird. Okay. No, yours is better. Just replace mine with yours. Okay.

Okay. Yeah. So, so yes, I thought that was super cool. Like when you said the AI thing, I was like, Oh yeah, that was, that was something really neat. So, so definitely check that out. It's pretty, pretty cool. Okay. So then, uh, there's an AI for that was my he, he won, you know, funny, funny, but a serious one though. Um, and you know know there were some others on our on our team that thought that this was a super helpful one so i thought i would share this with everybody else

but if you've ever like in your kubernetes type environment you're like wait which version of the of my docker image are you running like what's the tag for that and you would think that it should just be available in the description like when you describe the pod, you could do a kubectl describe and think that it would be there. Unless I'm blind and missing it, I never see it there, and that seems like such an obvious place for it. But if you do a kubectl get deployments dash o wide, so output in wide format, it'll show you what the in the uh image with a tag that is

deployed for each deployment super helpful if you're trying to debug something you're like wait a minute i thought this deployment had my stuff and does it or does it not and you know if you're if assuming that you're able to see, you know, based off the tag, you know, so your mileage may vary there depending on how you tag your Docker images. But, you know, if you ever wanted to, if you're in that situation like I was where I'm like, wait a minute, I thought that my code was deployed. It doesn't seem like it was.

What version of this Docker image is running? Then this is a quick and easy way to do that. Yeah, it's nice. Most excellent. You're muted again, Jay-Z. I feel like, can I mute him for him so I could be like, nope. So see, he's going to try to talk right now, and I so badly wish I could mute him because then that would be super hilarious.

Check it out. I actually did mute him. That's pretty good. Sorry about that. That's pretty good. Ha-ha. Hey, for what it's worth, we thought ha-ha very much. Yeah.

Oh, man. What were you going to say about it uh i don't know i just thought it was a good tip uh oh yeah oh i remember what's so with the uh like it's something like i somehow forget and remember all the time and like it goes in circles like maybe even like a couple monthly basis but um you can uh exec into deployments uh and i'll just pick a pod so uh i used to have scripts that would like get a pod name for me and just i would get the pod name and i'd shell into it instantly but you can actually just exec and you just have to do deploy slash deploy name that way you don't have to look up the hash at the end oh cool, that's pretty cool. Yeah. Oh, that's a pretty cool one. I didn't know that.

You should put that in there as a tip of the week. Yeah, I didn't know that either. Yeah, I was looking at a readme last week and the person who did that was like, oh, you know what, let me update this. It was a fairly new addition, I want to say like a couple, last couple of years to Kubernetes,

to KubeKuddle API. It was super handy. Yeah, I'll put an example in there. I got to tell you, though, like I can't imagine going back to a non-Kubernetes world. Oh, no. It's painful going there, right? It's painful when you're in it, too.

I'm not going to lie. But, you know, like the ability to have everything about your data center is now like in code. Yeah. Even though it is painful to like get through like all the networking and whatnot. I mean, it's so much more powerful.

So, so to tag on to Jay Z's tip, you know, you can also do a cube cuddle logs on a deployment and you can see the log output for all the pods in a deployment right so the shelling's pretty cool yeah it won't get hairy at all i mean fortunately they usually tag which pod it came from whatever the log so it may not be the most useful thing in the world unless you're just trying to see if it's actually operating like all of them are in a good state but yeah

so it's so cool the exact i mean maybe you don't know but like does it um look at like which one has the least cpu utilization maybe i don't know because i that would be a good idea because the reason why i asked that is like when i do shell into a pod i'll i'll like look to see like okay okay, which, which of this particular, you know, deployment type that I'm looking for, which pod is being utilized the least. And that's the one that I want to go dig in. I want to go use. Cause like, you know, if the other ones, cause I'm typically playing around in like Kafka related ones. So I don't want to like, you know, any brokers or connect instances that are being heavily hit by you know whatever the the stream

is that's happening at that time i'm like okay you know let them be i i just have i just want to poke around at some topics i love that outlaw is a good custodian of whatever he's using it's very thoughtful i don't do that i don't think i've ever looked like you're just like hot one right i would honestly i would shell into all of them and run my command on all of them just to see which one's faster. I don't care about them. I care about me.

Okay, well. Outlaw's like, oh, wait. He's at 30. He's at 28. We'll go with 28. I'm like, I don't even care what these guys are doing. I'm going to see which one i can get into you guys gotta go fast that's amazing well because like if you start if you like kafka if you use like a kafka cuddle to and you start consuming a topic and you don't like start

filtering out your results right and you just like get this endless stream of stuff coming out to standard out. And you're like, no, no, no control C control C control C control C. And it's like, uh,

I'm busy now. I don't stop. Yeah. Yeah. And you know, so that's why I'm like, okay, let me,

let me take a moment to like not, uh, abuse anyone that's already being heavily hit. All right. Well, if you want these tips and other crazy tips, you should probably subscribe to us if you haven't already. iTunes, Spotify, wherever you like to find your podcasts. We hope we're there. Be sure to leave us a review.

Three stars and up. You can find some helpful links at www.codingbox.net slash review. Also, if you've only left one review, hey, guess what? Now's the time. You have the opportunity to leave a second one. And why would you want to deny yourself that opportunity? Right? Like that?

Yeah, really? It would. Like you would feel good. And that's just going to put good karma out there for you so i highly recommend it i've already done it like three times while we were recording the only thing holding you back is you hey so while you're up there make sure you check out our show notes examples discussions and more hey and if you haven't already go join our Slack at codingblocks.net slash slack.

Yep, and follow us on xcodingblocks, which sounds like drugs, but I'm talking about the website. I can't get behind this name. X going to give it to you. Website too, codingblocks.net. You can leave comments.

We appreciate that. It was really great. Tielder. Alright, so thank you very much. Have yeah, we appreciate that. It was really great. Tielder. All right. So thank you very much. Have a good one. Peace.