Coding Blocks - Open Telemetry – Instrumentation and Metrics

Episode Date: October 30, 2023

Join us as we get back into the world of Open Telemetry. Piggy-backing on previous conversations from the Google SRE series as well as DevOps, metrics and instrumentation is a key component of understanding what’s happening with your systems. Follow along as Michael, Allen and Joe dive into Open Telemetry’s Metric offerings. Reviews Huge thanks […]

so what are we talking about then in this episode oh yeah so uh this episode we're going back to our roots from a couple episodes ago of open telemetry so we're going to talk about this that's our roots from a couple episodes ago right like we could change roots right so yeah we're going to talk about metrics this time instead of traces and spans and all that that was wait that was the intro yeah that's the intro i didn't get to say my name yeah i'm alan underwood that's not what it says here are you i'm embracing change guys i don't know if you notice this how about how about you start us off jay-z all right just reading the notes here, it says, Hi, I'm Ray Romano. Oh, that's right.

And I'm Bill Clinton. And I'm neither. That's how you can remember. The problem is, the only phrases that come to mind from Bill Clinton are probably completely inappropriate for this podcast. So I can't even say what yeah i did not maybe one time oh man and if you don't know don't look it up yeah there was one about him not inhaling too that was kind of funny

that's right did you smoke pot i did not inhale okay good job dude oh man unbelievable all right so so yeah i don't know if we're actually cutting that beginning part or not so yeah we're going to be talking about metrics and open to limit why would you think i would cut that what that was That was perfect. Oh, it might've been amazing. I don't know. Like I said, I'm embracing. I am Alan slash bill. So let's do this thing. Hey,

first up, we got it. We got some reviews that outlaws going to read to us. Oh man, these are too easy. Oh yeah. So, well,

okay. Uh, from iTunes, we have bill B one Oh one and from Spotify. Yeah. Now we're doing Spotify reviews, uh, Donnie and Clayton. So thank you both for taking the time to, uh, leave your reviews. Really appreciate that.

How long did it take you to locate the Spotify? Man, I, you know, maybe like now that I've, now that I've navigated through it, I'll be a little bit quicker next time, but like not you, not as a, as someone who doesn't use Spotify and I'm the one, I'm the one you met me. I'm, I'm the one person. I'm the unicorn that doesn't use Spotify. Yeah. It took me a minute to navigate through it only to realize like oh i'm not even in the right thing there's a whole other thing of like uh you know spotify for podcasters and that's where i had to go and find it and then then i found it so yeah yeah good times all right so i guess with that do we have any news are you speaking anywhere jay-z anything going on no were you supposed to speak somewhere tonight? Oh, I'll be right back.

Hey, I'm actually thinking about dusting off the speaking things and maybe doing something again. I don't know. I kind of got the itch again recently. Well, if you are, I believe the call for speakers for Orlando Code Camp in, I think it's going to be February this year, has opened up. That would have been nice news to share, huh?

Hmm. There you go. See? There's news. CodeCamp 24. We'll have a link. All right. Excellent.

I may actually do that, but that means that I could get stuff done here the next month. That might be tough. So it won't happen. So let's talk about metrics. Yeah, let's talk about metrics. Things that we could actually measure. Things that will happen tonight about metrics. Yeah. Let's talk about metrics. Things, things that we could actually, it will happen tonight, man. Unreal. All right.

So first, all right. So we've, we've sort of talked about, or I wouldn't even say sort of, we talked about metrics a lot when we were doing like the DevOps handbook and all that kind of stuff. Right. That Google SRE. Yeah. Google SRE was a huge one. So I think these are worth talking about again,

just for anybody that's missed it. But then also we are talking about this in terms of how it's being done in open telemetry. So going back a couple of episodes, open telemetry was something that Jay-Z brought up that is actually a really good topic because it's sort of like you're all in one way of being able to see what's going on in your application. And open telemetry is sort of the culmination of, of groups and companies coming together to try and give you one package that will sort of do it all for you, right? In, in a free and open source type way. So that said, we've talked about like spans and logging and contextual type stuff. And so now we want to talk about the other part, which is measurements. And that's what a metric is, right? It's nothing more than a measurement

of a service that's happening at runtime. And then a metric event is basically that metric with a time stamp i mean it's it's that simple right so if i don't know you had uh 12 millisecond latency at 1201 am that's that's your metric event and what they call out on this is and it's the same thing we've talked about in the past. These are your indicators of your availability and performance. Is your service running? How well is it running, right? Like if you're getting zero throughput and you're used to seeing, you know, a thousand

requests a second, then you probably know there's an issue. And then these also not just showing you your system, they can also tell you sort of how it's impacting your end users, right? And, and your business systems at large. So I think the more important thing than just having these is what you do with them, right? And this is something that the three of us have focused quite a bit of time on both on the podcast, as well as in our professional life is what do you do with these metrics right you're going to create alerts off of them and then you're probably also going to have some sort of system triggering events whether it's you need to restart or you

need to scale out or whatever right like there could be all kinds of things that actually happen when you see some sort of metric go above or below some threshold yeah we talked about service level objectives remember the the google book they're talking about how they aim to have like you know the however many nines of uh service availability and if they get below that that's when it becomes emergency because if you're doing enough requests then uh some of them are going to fail some number of them. So, yeah, just kind of harkening back to that, I guess. And, you know, talking about the, you know, how many requests and all that kind of stuff, something that we talked about when we were going through the SRE handbook and all that was don't report on every single thing. And it's funny, you read that, you go through it, but you kind of learn it the hard way too, right?

When you go in and you set up your first alerts, you're setting them up and you have this threshold. Then all of a sudden you're like, oh man, this thing should not alert on every single 500. It should not alert on every time this happens. And it's amazing how quickly you realize that doing that kind of thing will really shoot yourself in the foot because then you're just chasing things all the time. Yeah. That was also, there was also like a cost factor that they had equated into like, what does it cost to get to that next nine of reliability?

So it wasn't just a blanket, you know, we want 18 nines of reliability, like, you know, it's Google, right? So they, they took an engineering, you know, kind of approachines of reliability like you know it's google right so they they took an engineering you know kind of approach to it and we're like okay well what does it mean to get there to that next nine hyper expensive right for every additional nine you add on all right so so there was also another call out too though that i was going to mention you uh when you were going

through the metric things. And you're like, well, is your site even up and running? And there was that story I've shared before, I believe. I think I shared it where it was like, you know, long ago, we had a script that was like, hey, is the web server up and running? And it was like, okay, you know, port 80 is responding. So the web server is running but in actuality like yeah okay port 80 was responding to traffic but the site itself was down like i'll never forget that like really yeah so so i, you know, like that, that's why, that's why even with metrics you have to be careful of like, well,

what is it that you're measuring? And like, and, and is it actually what you wanted to know? And maybe the one thing by itself doesn't tell the complete story. Yep. Right. And so that's where like, uh, all of these metrics and panels and alerts that you were talking about, like, you know, sometimes it can be beneficial to correlate these things together to paint a better picture of like what's happening. For sure. But you could also, and we've seen this too, you can also go too far in the other

direction, right? Like you'd have 80 metrics on a page and you're like, what does it mean? I don't know. I would bet that anyone listening that has used Grafana has probably been in an environment where they went too far to one extreme. And have a dashboard that just had way too many panels. And they're like, okay, I'm just going to spin up another dashboard. These are the five panels i care about and then that then becomes like the thing that takes over and everybody's looking at it and then eventually it gets blown out like you know there's this constant like evolution you know it's like the desktop folder you know like put all the stuff that's on desktop in the folder and i've seen it's like yeah yeah it is evolution is the

perfect description of what happens with that stuff so let's talk about metrics and how they work in open telemetry so this is some implementation information here so first they have this meter provider and and the way they describe it it's basically a singleton and it has the lifespan of whatever the application lifecycle is. So what this thing does is it actually creates meters. So it makes sense, right? Meter provider makes meters. They say that this is the first step in actually metering and open telemetry is having this meter provider set up. And in some languages, I'm not sure which ones, they just call it out in their page. Some of the languages, the application sets it up for

you. So I guess if you include the open telemetry libraries or whatever, it's going to go ahead and wire it in and you don't even have to think about it, right? At some point you'll just go use it and life will be dandy. I don't know which language, but you know, Hey, all right. And then a meter, this thing that's created by the meter provider, this creates different types of metric instruments, quote unquote. These are the things that actually capture the measurements of the service. And we'll talk about these metric instruments here in just a second. But first we're going to talk about some of the other things. And this goes back to similar things that they talked about with their tracing elements. So one is there's this metric exporter. And if you think about it, anytime that you generate a

metric in your application, there needs to be a way to ship that data out to some sort of consumer. And that's what this metric exporter is. And then a consumer can be all kinds of things, right? They called out, it could be standard output, right? So just your logging that's coming across your screen as things happen, or it could be an open telemetry collector or tons of open source things. So just like they had, I think on the, on the span or not the span, the, the tracing stuff, they had exporters for that. They have a list of, of collectors or exporters that they have available in this registry. And it's worth clicking that link and taking a look at it.

There are tons of these things, like tons of them, just about anything you can think of. They've got in here that you can export these metrics to so just to make sure i got all right we had some terms there so uh metrics i got it um the meter provider is responsible for creating and configuring the meters the meter i don't quite understand what that is but it's basically responsible for creating instruments and instruments are what actually reports the measurements. Well, the meter is the instrument and that's why I said we'll get to it in a second. But just just so that it'll clarify things right now, a meter is like a counter or a gauge or any of those type of things that we've used in any kind of metrics gathering before.

That's what they call them. Well, to like make another analogy here though right like the the exporter here for example um because i'm going back to like you know prometheus and grafana kind of world right i guess i've said that earlier and now i can't get it out of my head but um you know prometheus could scrape a a set of HTTP endpoints, right, to collect metrics, or your application can send them to Prometheus. And in this case, with the exporter, we're talking an instance where you might have some kind of custom metric in your application that you would send to Prometheus. And that's what, in this case, the exporter is kind of similar to that. Correct.

You could do that to where you shipped it. Or like you mentioned, you could actually, your exporter might be an endpoint in your application, right? Like I think we've done some Java apps to where if you use the Prometheus client or whatever, it'll expose those metrics on a certain port so that Prometheus could scrape it. So that could be your export as well. Well, I want to be clear that this is only the one direction.

This isn't both like what I was describing that could be done with Prometheus, right, where Prometheus could scrape. Because here it's just you sending the metric. So it's the case of your app sending it directly to Prometheus, not Prometheus scraping it from an endpoint, right? Well, if your endpoint is what you're exporting it to, I guess is what I was saying. So you could, like it says, you could export it to standard out. You could export it to, um, you could push it to Prometheus, like you're saying, or you could have it exposed or pushed to an endpoint in your own service. Right. So I guess that's what I'm

saying. Like there's all kinds of places where you could make that stuff land. And if you want a Prometheus to scrape it, it would be scraping it from your export to your endpoint. I guess. I just don't, maybe we're getting hung up on the prometheus analogy that i was trying to make and so uh it's it's fine but i i guess the the the thing here is there are so many different things like here's an example i mean i'm just going to read off a handful of these they have the open telemetry collector thatector. That's awesome. Alibaba Cloud, AWS S3 Exporter. They have Google PubSub Exporter. They have InfluxDB.

I mean, they've got so many Parquet File Exporters. So there's tons of things that you could set up in your app to basically just shove these things out so that any type of consumer you want to set up will have access to that data. Oh, wow. And like, they actually have these things labeled as either collectors or exporters. So pretty amazing. And like,

like every language that you would want service, all kinds. Well, I take it back. I'm really disappointed. I don't see visual basic in here but yeah i know but uh you know go rust you know modern new languages cotland scala well scala i don't think counts as new right or does that count as modern? I think so. Probably modern ish. I don't know.

It's been out too long now. It's dead to me. Yeah. Um, yeah. And there's, it, it's well over a hundred gotta be. Oh, it's ridiculous. How many of they have? I mean, it's, it's pretty impressive what they've gotten.

I'm assuming if you want to make your own, they probably have the, the interface set up so that, you know, you can do it and then throw it in this registry and then somebody can go grab it. Well, those links on that page are to GitHub. So you could easily see how these things are made. Yeah. If you want to go create your own. Yeah. It's pretty awesome.

I mean, they have an Elasticsearch exporter. It's beautiful. Oh, wow. The source for this. I don't know if we talked about this before. It's all in. I mean, they have an Elasticsearch exporter. It's beautiful. Oh, wow. The source for this. I don't know if we talked about this before. It's all in Go. Is it? I don't know that I've ever looked at it.

I mean, I'm at least looking. Like, I randomly picked, like, the TPC receiver. I'm sorry. I meant to say TCP. I don't know what I said. TPC? Oh, yeah. A lot of these are in Go.

But, yeah, the TCP receiver, I just randomly click on it, and it was in Go. And then, oh, okay, no, I see that's still the same stuff. But, yeah, no, it's all Go. The Prometheus ones, Go, yeah, it looks like a lot of these are in Go. Yeah, if it wasn't Rust, I think everyone might just be using Go. There was a minute there when everything coming out was in Go, like Kubernetes and Docker and everything. Then Rust came out, and people started being like, well like well i don't know now it's kind of split

whoa check this out the very top link i don't know if you guys saw this the open telemetry collector builder a cl a cli tool that generates open telemetry collector binaries based on a manifest so it looks like they've got something that'll help you even get rolling on this thing yeah i don't know man i'm hearing builders i'm hearing providers you show this in java or c-sharp nowadays right yeah yeah all right so so now back to the the question that you asked a second ago jay-z was like the metric or the meter and the metric instruments. So these are the things that actually capture the measurements. Um, and they have a kind, a name, a unit, and a description. Now the name and the kind are required unit and description. You can provide as you want or don't

have to, um, the name unit and description are chosen by the developer or like they did in the tracing they have a whole list of semantic conventions that you can follow right so that if you're doing things like cpus or requests or things like that you probably want to follow a standard so that you know everybody's sort of familiar with what you have. If it's not one of those, then you can name it however you want, but here's the gist of it. So the, these instruments are either a counter and this one's simple, a counter in any type of metric package that I'm aware of, a counter is always a number that increases. It'll never go back down. Right? So they equated it to an odometer and a car. If it, if it were to actually work forever, it's always going to keep going up. What year car, right?

The kind that you could eventually roll over, right? Yeah, exactly. Either here, I was saying you've been, you know, you have an overflow and you've ended up back at zero at zero. That's right. But that means you hit a million miles and at that point you deserve to have zero again so the next one and this was interesting there's two asynchronous ones that they mentioned there's the asynchronous counter they say that this is the same thing as a counter but there's one big difference between it and the counter is that there's a count per export and i think what they're saying is every time you do an export, it sort of resets. And that's interesting. I'm guessing the reason they do it is because maybe you have something that can't accumulate based off the code or

however it's set up that it's not going to do it. And so you have to rely on the consumer to aggregate that data is more or less what it sounded like in their docs. Now, I haven't used it, so I can only tell you what I read there. I hope to use it at some point, but that's what it sounded like. They have an up down counter, that's what it sounded like they have an up down counter which is what it sounds like so it's basically a counter that can also go down so instead of only always incrementing it can also go back down they said that a good use of this is if you were like looking at the number of items in a queue so you know it's 100 now now it's 75 you know you

process whatever now i was i was thinking like the most obvious one to me would be like you know it's 100 now now it's 75 you know you process whatever now i was i was thinking like the most obvious one to me would be like you know current connections okay i like that one because that one would go that one could fluctuate over time right well so here's my question so i'm going to jump ahead well actually first we'll do the asynchronous up down counter this one again they say that it's, it's collected once per export. So again, I guess the, the number resets and then you're relying on the consumer to aggregate the data. But so now that we got those out of the way, why, how is an up down counter different? Oh, I know how it is. I know how it is. Okay. So the gauge, I was going to say a gauge. So a gauge is a value at any time,

right? And they, they mentioned like a fuel gauge in your car. I actually liked the idea of a speedometer better because it moves up and down frequently, right? As you accelerate, decelerate, mine just stays on one far end max. It doesn't even have a number but i think i think the difference is this a gauge is a point in time so so if you're going 10 miles an hour now and then a second later you're going 15 there's no incrementing and decrementing i think that's the difference between the up down counter as i'm guessing the up down counter is you increasing, like added one to the queue, took two off the queue, added three to the queue. I think that's the asynchronous counter and asynchronous up-down counters that you're describing. Because they make the point of the continuous changes.

So the counter and up-down counter, you're going to see every incremental change. One, two, three, four, five, six, seven, eight, nine. yeah yeah you're going to see like every incremental change yes one two three four five six seven eight nine you're literally getting all that streaming in versus with the asynchronous counterparts to those you might go from in your example 10 to 15 17 23 19 asynchronous doesn't make sense to me even there because you're only going to get you're going to get like the way i the way they they wrote it here the way i'm interpreting it is you're not getting the full stream of all of the changes so you're just going to say like what is it now and you're and whatever the count is at that point you're going to get the updated value for it. Yeah. I need to see it in action.

I wish that I'd actually had the time to do that before we did this. Cause again, see it because these are again sending, you know, because I, I mean, this is my interpretation. It's been wrong for export,

but again, because the exporter is sending the data to a consumer. consumer, how often do you want me to send it? Do you want me to send you each incremental change? 1, 2, 3, 4, 5, 6, 7? Or 1, 6, 11, 9, 10? Well, I think there's a difference between the gauge and the counter, though. And that's what I'm getting at is the gauge is like, hey, if it's 15 now, send 15. If it's 100, send 100.

Well, because the gauge was the next one we hadn't gotten to yet. Right. So a gauge measure is actually increasing and decreasing. I think it's an additive or subtractive type thing, and that's the difference between that and a gauge. So did you say this already? A gauge measures a current value at the time it is read. An example would be the fuel gauge in a vehicle. The gauges are asynchronous.

So that's where you got the fuel gauge. I didn't realize that you were talking about the next instrument kind when you did that, so I apologize. Yeah, that's why I was like, initially I was confused, but I think the difference is the up-down counters counters are additive and subtractive and the gauge is just, Hey, whatever the value is in the value. That's it.

But now I'm confused. It's like what the asynchronous counterparts are. No, that's what I'm saying. I wish I could see it in action. Cause I think the asynchronous from, from the way that they make it sound in the docs is it's once per export. So when you call export,

it sounds like it resets the value and then and then you go from there that's what it sounds like but i haven't seen it in action so i don't know but it okay so looking at that or we're not trying to like harp on this one thing but this is this is the episode so bear with me right so but it says asynchronous up down counter could be used if you don't have access to the continuous changes but only to the aggregated value example current q size yeah we talked about this with databases a little bit remember where it's like we said you could do like interesting kind of locks with like increment or decrement where you're basically just saying

plus plus or minus minus but you don't actually know what the value is you're just saying like i got a new one or i got rid of one uh and so like locking and stuff is easier but if it's something else where you're setting like the number 15 then you have to worry about like you know if you're counting or something in particular and like there's race conditions and things can be mixed up, they read the value, try to add one and then put it back of another one modified. It gets crazy. So if you do know the value, cause you're exporting that number out as the metric.

That's what I'm saying. I would, I'd have to see this thing in action because based off what they wrote, I don't understand exactly exactly it makes it sound like they're not making their implementation any easier yeah right yeah yeah so we're gonna need we're gonna need to test this thing out now the last one and this one is probably the most interesting because i've played with this stuff is a histogram. So this is a client side aggregation of values that is useful for things. And they give an example of like request latencies. And I can

actually give a real life example of where I've used a histogram. And basically what you're doing in a histogram is you're sort of creating buckets. And then you're trying to count the number of things that happen in those buckets. So for example, in, in a real life scenario, we had files of different sizes, right? So I wanted to see, Hey, how long does it take to process a file that is between zero and 1k between 1k and 5k between 5k and you know a meg or whatever right so you create these buckets and then you also measure the latencies and so it allows you to put counters basically in each one of those cells so that's what a histogram allows you to do is sort of chart these these values as you see them and yeah that's

similar to like the wikipedia definition for it what refers to it as the number of observations that fall within each bin it's a visual representation of the distribution of the data showing the number of observations that fall within each bin so i thought it was interesting their choice of bin and your choice of bucket you know it'll mess your mind up when you first go to create one if you've if you've never done a histogram and you start trying to lay this stuff out i don't know what open telemetry's implementation of this is but like for instance i want to say the the prometheus one specifically you have to define the buckets up front. You can't be like, hey, just figure it out for me.

Like create some smart buckets. You could really easily think about that. Like, you know, if you had to make data, if you wanted to, every one of your coworkers on your team, everyone who works on your office floor, right? Uh, if you wanted to say everyone whose first name, if you want to take account of everybody's first name starts with a, a B, a C, whatever, right? You're going to have this like

bar chart where like, you know, one of those letters is going to be taller than the other, probably something in like an M kind of in the middle of the alphabet or something, or S or T or something like that, right. Are going to be taller than the other words, right? Those, those different, each one of those little bars is going to be the history collectively are the histogram. That's the distribution of the data as it, as it relates to, uh, what's the distribution of, you know, first, uh, of, of my coworkers, first names across the alphabet. And your predefined buckets are your letters, right? And that's, that's kind of the thing with histograms is you sort of have to know what your data sort of looks like. So you can create these buckets to make a meaningful histogram. Or you just guess and start somewhere and then

iterate. And then there's that especially when because i say that because especially when it comes to like time type related things because you don't know they gave the example of like how many requests take fewer than one second you know and and i think it was the sre book too they talked about doing stuff like that in terms of like what you would alert on and what you wouldn't, you know, you like, you would,

you would set some, like you would time box some requests type of thing. And that's where your alerts be. So you might not know, like they were also using like percentiles, if I recall to like, you know, um,

I think it was like, as long as the most requests are happening, then it's fine. It's probably okay. Type of situation. So, you know, your, your mod is

going to vary but the point is in the beginning you might not know what that time frame should be and you could get too aggressive especially if you were going to like make an alert on it yep oh so we finished all of the the metric? No, just the first part of it. Oh, okay. So then I guess then you're wanting me to tell everybody how they can be cool like Bill and Donnie then. Or at least in our good favor. How's that?

You know, they're our best friends for life. You can head to codingblocks.net slash review and you find some heading blah blah links there and uh we would greatly appreciate it and i'll say words that make sense later all right well uh you we could take a small break then for mental Blocks, our newly patent-pended game show that is going to sound a lot like Jeopardy, but it's really not. For trademark reasons, it's totally different. It's 100% different.

This is Mental Blocks, whose intro sounds nothing like Jeopardy. That's right. All right, so 221 221 alan you are first according to techos uh trademark rules of engagement um i'm on a win streak i'm on a win streak just so we know it's one right no no i've won like two or three two or three in a row now yeah it's more than one it's more than one. It's more than one. It's plural.

We have such low standards here. Like the bar is like literally like just on the floor. And if, if it got over, they were like, yep, that's good enough. That was right. All right.

Well, uh, mental blocks. So your choices are a Tory party, the 50 States, other famous volunteers, letter perfect, plenty of fish, or I love my dad. The 50 states. All right. One,

two, three, four, or five to that other show uses money. We don't. Okay. That's the defining distinction. Uh, two created to serve the former Northwest territory.

Northwestern University is in Evanston in this state. Massachusetts. No, it's Northwest. You already answered. Yeah, Massachusetts. All right. I got a guess. Is it Illinois?

It is Illinois. How did you get that as a guest that that sounds that's not a guess i wasn't sure there was uh like there's this book i read once and like the person was talking about how cold it was in illinois and northwestern i couldn't remember if it was northwestern or not that's ridiculous i guess it was. Okay. Okay. You know, allowed to read. I really, I do. I, I was excited to see the first question for the other famous volunteers.

I did kind of hope that you were going to pick that one though, because it's the, the answer is a superhero on film playing Marvel's black widow. This actress spent eight years as a global ambassador for oxfam and i just thought like it's marvel so like obviously like why would we would all know that one did you know it just didn't know uh is scarlet or something scarletlett Johansson? Yeah. Was it her? Yeah. She played Black Widow. Do we? I'm so bad at this.

Do you ever feel like you're with other people, but yet you're still alone? Is it just me? Okay. That's kind of how I feel right now. All right. So Jay-Z's taking the win back. So here we go. Next round.

Power back. Your choices are the movies, triangles, colleges and universities, everyday Italian, the 20th century. Oh, what do you mean? Or gloomy authors. Everyday Italian. The 20th century. Oh, what you mean? Or gloomy authors. Let's go triangles. How much is equally? There can't be more than three questions, right? Yeah, there's five.

Yeah, I saw all these. What did you see? Acute. No, I'm just kidding. Seriously, one, two, three, four, or five? I'm feeling pretty good. Let's go four. Four.

Four. You've got to find out there's 12 types of triangles, sir. And we're sure four is like we're we're sick okay yeah i'm gonna win good real good yep on this holiday generally in march jews traditionally eat triangular pastries called i'm gonna mess this up um that's not what it's called uh i for all of our jewish listeners i do apologize i'm not even close to that how do you spell that that means maybe i can like no you know google to tell me like how to pronounce this thing.

What's the question? Is there a holiday in March when they eat some triangle food? Hamatashen. I was close. Hamatashen. Okay. At least according to what it says, the pronunciation key for that is. You're asking what this holiday is?

On this holiday, generally in March, Jews traditionally eat triangular pastries called hamatashen. Man, if you asked me what the pastries were called, I could have told you. You didn't know that. Get out of here. Apple turnover. Yeah. Jeez, when's the new year? What's the new year?

What's it called? I'm going to call it Jewish. What's Jewish New Year? new year what's called i'm gonna i'm gonna call it uh jewish what's jewish new year alan for the steel um oh man i was so off i I got nothing, man. Nothing. Okay. Parham. Okay. Yeah, I wasn't going to be anywhere close to that. All right.

And I had to look that one up. I looked that one up, too, to make sure I was. And even then, I don't even know. I could have still messed it up, so I apologize. All right. Triangles are hard. This is your chance to tie the board, Alan. Here we go.

Let's do this. Oh, by the way, how did you not pick the movies i don't know jay-z doesn't like movies any of them suck his powerful performance as the joker in the dark knight won him a posthumous supporting actor oscar well i know that one come. I don't know that one. I actually don't know that one. That was the number three question. I only know that because I'm on Reddit. Reddit loves Keith Bludger. Wait, it was Tom Cruise. Okay.

So, Alan, your choices are. Here we go. We've got chemistry, explosive music, misinformation where the miss is in quotes. Each response will start with M-I-S. They came to America, peninsulas, or oh, Canada. Misinformation for one.

I was so glad you didn't pick the Canada one because I'm like, like oh we're gonna like really upset our friends to the north hey all right joe you're gonna hate yourself no at christmas couples may steal a kiss beneath a sprig of this mist Mistletoe. That is the correct answer. Hey, look at me coming back. I tied it up. We gotta have a tie breaker here. Yeah, here we go. Final round. Category is Corporate Mascots. Born

on an island in a sea of milk, this pitchman was jokingly disavowed by the U.S. Navy by saying he is not in personal records. Sorry, I meant to say personnel records. I forget. You beeped. You're fine. He said bye-bye.

I tried. I don't know man nothing is that your final answer yeah yeah i got nothing you want to call a friend because this game we can do that i thought you called my friend outlaw uh captain crunch oh come on oh ridiculous i should have known what do we do here we got like do i just randomly pick another one just one more just pick one more let's see what we got popeye's on the spinach spinach c uh i'm trying to think like what would be a good one let me let me think here i'm taking too long oh gosh it's getting worse there's gonna be a metallic question in that explosive music explosive music category.

Yeah, you should just pick the second one out of explosive music and let's see what happens. Oh, is there a Metallica question in there? Got to be. All right. Oh, that's not fair. You guys know all the Metallica songs. I know like three of them. We're going to.

Okay. So your name is your buzzer so just say your name and whoever says their name first gets to answer first you ready there we go the answer is i'd catch a grenade for you he's saying in a number one song alan bruno. That is the correct answer. Alan for the win, continuing his streak. Look at that. Wow. Good. So you earned it.

You earned it. I did. Yeah. Because apparently you're way too excited to know anything about Bruno Mars. I know a song. Hey, that triangle one, though. Come on. The triangle one.

That was not about triangles no i mean they ate triangles this is mental blocks do you expect the questions to be like right i was like i know it's going to be sapinski i know it i don't even know who that is it's a triangle made out of the triangles pythagorean man that's what it was gonna be that's what it should have been all right all right so back into this okay i mean you were you expecting like a hypotenuse kind of question yeah totally so katoa i know triangles apparently all right so getting back into this thing with the metrics man uh i'm not recording really no i just wanted to throw you off okay go on i man i was like man i'm going to bed guys it was fun we'll see you in two more weeks uh golly wow i think it's sweating aged a year

so hair fell out but it didn't got any left oh man okay all right so aggregations this is pretty simple right like yeah we kind of know what aggregations are but it's a large number of measurements that are combined into this is the interesting part either exact or estimated statistics we've talked about this in the past like even elastic search right like when when it returns counts and stuff they're not necessarily accurate which is sort of interesting i mean it depends on how you have it set up and all that. So same thing can happen with aggregations. Now they did mention this OTLP, which they just kind of threw out in the documentation. Like, you know what the heck it was. They have their own open telemetry protocol and that's what OTLP stands for. And it does transport aggregated metrics. Let's see. Oh, also the open telemetry API provides default

aggregations for each instrument type. So we talked about the gauges, the counters, all that kind of stuff. They have default aggregation set up for those, but you can actually change those if you want. You can override them using views, which we'll talk about in a second. You think we're going to run out of like, maybe we already have like the whole, there's so many acronyms in software development, right? That, you know, maybe back in the seventies, I don't know, maybe it was like two, three letters, but you know, this one's now four, you know, pretty soon it's going to be like all six letter acronyms like because i when you saw when i saw otlp and

even as i try to say it out loud i instead want to say olap right and that was immediately how i read it and i'm like what are you talking about like that's nothing new oh did they like repurpose olap for something else is the oltTP for like the transactional protocol, whatever. Yeah, OLTP, online transaction protocol or whatever. But yeah, that's the thing. Like I was reading it and I was like, what are they talking about? I had to go Google it, which brought me back to their site, to their open telemetry protocol.

Yeah, I agree, man. The acronyms are out of control. All right, so when we were talking about request tracing or tracing in general, the whole purpose of that was to sort of wrap this context around a request, right? So you could see what happened in the context of the entire request. Metrics are different because on the other side of things, they're there to provide you some aggregate statistical information, right? Like how many requests happened in a second or how much memory or how much CPU or whatever, right? So it's a different purpose, even though combined with the other stuff can be really powerful.

They had a bunch of examples here. I'll roll through a few of them real quick. Like an example of metrics, all of these are total number of bytes read by a service per protocol, right? So HTTP, HTTPS, FTP, whatever other ones you can think of, SSH, total number of bytes read and bytes per request, duration of a system call, request sizes for just for trending purposes, right? Like, hey, these requests were one byte now, but then they jumped up to NY. CPU or memory usage during a particular process, average balance values of an account. I don't know if they were talking about banking, but whatever, it could be something. And current number of active requests. So those are

all really good examples of things that you might want to track metrics on. So now views, these views, these allow the developer to customize the output that's provided by the SDK. So one thing that's interesting, and this kind of goes back to the SRE thing that Outlaw hit on earlier was there might be some that you just don't even want to do anything with. Right. Like you want to minimize, you know, you don't care about the number of system requests to this particular thing or whatever. You can you can either choose to ignore it or process it. And then on the last part of customization, you can also change what type of aggregation is happening with any particular instrument. And you can also choose what attributes you want to make available on it.

And when they say attributes, I have to guess they're talking about things that are similar to labels in other metrics type packages we've done, right? Like, you know, you have number of requests per second, but maybe you want to trace it by each tenant or each customer, right? And so maybe you have that particular attribute on with the metric so that you can sort of slice that data that way. And then this is sort of like getting into the last little nitty gritty parts of OpenTelemetry. So they have their language support, which open telemetry. So they have their language support, which I thought was interesting. So they have a bunch of stable languages or languages that they have a stable library implementation and C plus plus C sharp go Java, JavaScript, PHP, and Python, uh, experimental. I thought this one was interesting.

They have Erlang elixir and Swift or Erlang slash elixir and swift and alpha is rust so it's still got the popularity and then maybe never is ruby it said not yet implemented and i'm like what does that even mean why would you even have it listed if you haven't done it yeah wouldn't it be apply implied that if it's not on there, that it's like, but I, I would guess though that the, the fact that they put it on there,

I would say that they have intentions to do it and they just haven't gotten to it. Questions about it a lot. Yeah. It's definitely weird. It's just stuff that feels like a dig or something. I mean, there's a few people out there that want it, but, you know, when we get to it, yeah, I don't know.

And then, so here's one thing that I thought was really cool. And I didn't go deep into these sections here, but they had this section for automatic instrumentation so basically what this means is with a little tiny bit of configuration in dot net java javascript php and python you can basically have this thing auto measure various things in your application oh so this would be like if you had a uh like a dot net app asp.net app or something like that and it's like oh i already know how to get certain metrics out of asp.net or yes you know whatever that blazer or whatever you're using you know to like as you're serving up requests i know how to talk to iis and get that kind of information and say like okay this is the number of you know site this is the size that you're sending here's

by protocol here it is the number of open requests and connections. Okay. That's the type of thing. Now, again, this goes back to, I haven't implemented this, so I don't know what kind of stuff it's going to spit out of it, but yes, that's my assumption too. Just what you said is, Hey, I know, I know what MVC does. I can look at these controllers and I can, you know, inspect the stuff coming out of it and I can create useful metrics. So the cool part is they say, with just like super minimum levels of configuration,

like they say the most or the one required piece is like the service name for the configuration. But then past that, you can also do data source specific config, exporter config, propagator, resource. So there's additional things you can do to give it even more information so that it can do more for you. But that's pretty cool if it's sort of just plug and play. Hey, put the library in, set a few configs and off and running, right? Yeah, you want some quick branding points. If you work in.NET, Java, JavaScript, PHP, Python,

drop it in and see what you get. Yeah, totally. And then, of course, they also mentioned they have manual instrumentation. And this is obviously going to be something where you can take things a step further. So an example I can give you from sort of my real life stuff is you're getting messages off

like a, a pub sub type queue. And when you start processing that message, you want to find out how long it took you. Like you might want to find out how long it took you waiting from the time that you got that message in the queue to when you actually start doing something with it. And then you might want to see how long it took for you to do whatever you're going to do once you started processing that message, right? So those are situations where like a custom or a manual metric or an instrument might add you some value because now you can track very specific things you care about, right? Like latency before processing and then actual processing time. And then, you know, whatever else in between that you have. So, I mean, that's, that's the kind of stuff that

probably most people are familiar with when they go and do any kind of manual developer instrumentation. So that covers a lot of what they have and what they wanted to talk about in terms of their own terminology they did have another section on on some more information deep dive i'll take a look through that and maybe it's something we'll come back and revisit but i think this is a pretty good overview and we've now we've got the uh the tracing we've got the metrics and then i think there's one more piece to it and i can't remember what it is now. You know off the top of your head either of you? Tracing, metrics, I'm going to Google it.

Yeah, I'm about to look. Tracing, metrics, oh, they have logs and baggage, which I think are sort of two different little things. At any rate, we'll come back to it, and we'll see if we can put a bow on this here in the near future these were all in the signals section though like traces and metrics logs baggage were all like the signals right so there's still like a lot related to open telemetry that you know outside of that to cover yeah we need we need to do it we need to see what some of these things actually do i'm actually excited to see it especially living in a distributed application world like some of this stuff would make life so much easier um i'm hoping guessing weird well we'll have some uh some resources we like here you can find some links and also uh we're gonna have some tip of the weeks there we go tip of the week or tip of the tips of the week tip tip of the tip of the

weeks tip of the weeks tip tip of the week well uh so i uh have been uh bringing everyone along with this journey i've been on into mechanical keyboards. I ordered a mechanical keyboard I was excited about. Yay, it came in. Yay. Oh, it's so loud, though. Oh. But then, you know, I got into it.

I started researching keycaps. And then I watched like a million hours of YouTube, like keyboard YouTube, which is a very large YouTube, by the way. And I kind of got uh excited about it and i watched like video after video after video trying to find like the keycaps that sounded uh you know basically i wanted as quiet as possible but didn't lose that cool feel you know and i wanted like the quietness and so i was watching all these videos and like oh this one sounds quieter than that one and then you google and it'll be like just like all these different opinions and like oh, this one says they're garbage, this one says

they sound wonderful, this one says they're silent and I finally just got tired and kind of ordered one that I saw on Amazon so I didn't have to deal with like these kind of weird companies and possibly being bobo and like you gotta be careful too because like some switches they're not lubed from the factory so sometimes

you have to get this little switch opener. Have you seen these? Man. So you buy the switches and you take each one apart. Like this is the 87 key keyboard. I'm not doing that 87 times. And you get a little brush and you lube up the spring and you put on the O ring and uh, uh,

so I was like, I'm getting some lubed ones. I know they're going to fit like the same brand. I had kale whites. And so, uh, i ended up going with kale deep sea switches and uh i'll tell you how i'll come in a sec so let me just know like even kale has like two or three different kinds of like silent switches and you watch the videos and it doesn't you know it's hard to really capture how how something that's supposed to be silent sounds you know and really compare it you know it's hard to really capture how how something that's supposed to be silent sounds you know and really compare it you know like if you're looking at something visually

you can kind of hold up a banana and say like this is how big it is compared to a banana when it sounds it's like i you know here's me snapping and now here's me pressing the key i don't know i can snap quiet i can snap can snap loud. I don't know. You know? Is it more silencer than the other thing? Is that the word? It was so stressful. And, you know, we know about mics and, you know, the frequencies you pick up and stuff. So I'm like, I don't even know if it's YouTube compression.

You know, like it just brings up the silence. It's like, oh, I was just freaking out, man. And I just ordered these Kale Deepsea switches. And they got them shipped. And I put them in. And they're wonderful. And these were not the quietest switches on YouTube. But here's the thing that impressed me.

They are so much quieter than the Chiclet keyboard I had been using. It didn't even matter like that's that was the bar i was kind of aiming for i didn't realize that all these videos i was watching all the stuff i was like you know kind of like agonizing over like a single decibel you know and it turns out that they were all way quieter than i was ever used to before and so it's way more than sufficient it's fantastic so i want to, if you've been thinking about putting in some silent switches, just do it. Any of them. You're going to love it.

It's great. I'm going to have a link here to the ones that I got. And yeah, can you hear? I'm going to see it right now. Oh, really? It's amazing. Yeah. Oh, that's really good.

Hey, so how long did it take you? Like, there's only 12 keys on your keyboard. How long did it take you to actually do it? Yeah, I did a lot of watching TV. So I don i don't really know and like yeah i didn't you know i was doing like on my lap you know so it wasn't exactly scientific uh you know like lab setup like people i have in the youtube videos where like you know it's a clean room they're wearing a bio suit you know you got the little brush there and stuff so i would say i probably did the whole thing within an hour okay that's not bad really you don't have a keyboard with a ton of keys on it right

yeah 87 and i i did damage some getting them out because i like i bought like the world's worst puller key puller you went cheap on the puller did you yeah i did i got one that was double ended you know i think i was like oh this is great and it's just like one of them and that was terrible it's because it's like too long like if i had to do it again i would buy shorter pullers that like they have they almost look like a ring like they're real short and that way you can get good leverage because this uh this one i kept like bending i had to bend it back into shape and then it wouldn't grab the thingy right and so that was my biggest gripe with it was like this five dollar puller i got that's you should include that as a link of do not buy right just don't recommend all right i'll get that i really i really wish like

this is where we failed because can you imagine like how great this would have been if we had joe on camera as he was trying to do this, like live stream or like record it and just like, you know, for the chuckles later of him, especially with this tool that he's trying to like pull the, I see the tool that you're using. Like one side of it is to pull the key cap off and then the other one is to

pull the key switch out. Yeah. So it was convenient. You know, I like the idea of it being able to flip you know but uh yeah it sucked that's amazing you know maybe that the idea sound but uh don't get the picture one i got because it was like basically working with the paper clip well maybe it wasn't meant for the entire keyboard it was just like for one or two yeah

dang it you got a cat laying on the keyboard right now you can't hear it because it's so quiet but it's uh messing up with me trying to paste a stupid link oh that's so funny man oh all right well so mine i hearken back to a tip that Jay-Z actually gave a while back, but it's so good that I had to share it again. So he had mentioned a plug-in, again, a ways back, called Rainbow CSV for Visual Studio Code. And when he showed it to me, and this is the only reason I bring it up again, when he showed it to me, he's like, Oh yeah, look,

it colors the columns differently and all this kind of stuff, which is really sweet. Like it, it makes it to where when you're looking at the, at a CSV that you open in visual studio code, it makes it really easy to, to visually see your data. What he didn't tell me,

or maybe he did. And I forgot it is you can query this data as well in visual studio code so they have this i didn't know that yeah so they have this language i think it's called rbql or something it doesn't even matter what it is it's basically like sql so you can open up a csv and then do your um command shift p or control shift P if you're on windows. And it, I think it's like RBQL you type in and it'll pop open a new, a new tab that'll have your data in a tabular format in like a table and you can type in a query. So like, let's say that you want to sort by a certain column. Like, so I had, I had a whole slew of data and I needed to sort by the count of something.

And you could actually do that. You can say order by, and then parse int that column. And when you hit return, it'll actually drop all that data in a new file ordered the way that you asked for it. Like there's just, and you could even project what you want, right? Like maybe you didn't want columns D and E. You can just say, Hey, select columns ABC and then, and then do your order by and you get everything. It's so good. So again, if you don't have have this if you ever once in your life have to open up a csv file in something and deal with it other than like excel or something i highly recommend this for visual studio code it's amazing episode 149 eh that's been a minute yeah that was actually my tip of the week on yours for vs code oh yeah i could have

sworn that was was jay-z's i used it briefly but uh yeah i didn't uh i didn't know you i i think it's real basic stuff with it so oh it's so good so good yeah i just i didn't i didn't realize that too uh or at least if we talked about it before i don don't remember it now, but I liked it just for nothing else, just to make it easier to read this stupid stuff. Yeah. Just wait until you do the RBQL thing, man. It's mind-blowing. It's so good.

But also, I mean, if you rely on RBQL for your CSV, maybe it's too much CSV. No, man. No, man. I don't think you understand. But also, too, though, like, okay, I'm not trying to knock anything against the plug-in for that. But honestly, like that type of use case that you're describing, like I would have just taken it into Excel or like something like that, you know,

a Google spreadsheet or whatever, you know, pick your, pick your thing of choice. I would have just imported the data there. And that's typically what I do. I don't know why I did it this way, but I was very happy that I did.

I got no, I have no, um, yeah, great reason. Like I said, like I love said, I love the plugin, but I liked it just for the readability. It makes it so much more readable

because CSV data isn't columnar, like visually easy to read, right? But it'll colorize each column. So if the third column is green, it's going to be green on every row. Even if those green lines are like, you know, scattershot across the thing or, or, you know, you could even have missing. That's why I liked it. But, uh, it's cool though, that they added that. I wonder if that's an, if the RBQL is something new, like maybe, cause that was in, we talked about that in 2020. So that was

three years ago or yeah. Um, I'm telling you, like, even, even though like what you said, I typically do open things up in Excel and work with them that way. I promise you, if you were to open this thing up in visual studio code, you'd be like, there's no reason I don't need to open up Excel. Now I can just live in visual studio code you'd be like there's no reason i don't need to open up excel now i can just live in visual studio code for 99 of the things that i do yeah yeah i guess i guess what okay so fair i guess i was just questioning i was kind of curious as to like what made you decide because i i wouldn't have even thought about like oh do i have a plug-in for this i would have just me and that's i would have immediately gone to Excel. And that's why I was curious like how you, what your thought process was.

It was like, Oh, let me do something else. I can't, I cannot for any reason, back up any thought process I have. I can't get much like the rest of our show. Yeah. Yeah. Well, uh, okay. So keeping along with that tradition then, so have you ever, uh, stashed a change in get, and then you're like, Oh, but I've since done like, you know, pulls and whatnot, like after that, but I want to go back to that stash, but that stash isn't going to be good on where I'm at now. Oh crap. What did I, what did I make that stash on? You know?

Oh yeah. Oh yeah. So, uh, name stash. I love stash now. There you can, uh,

there's three commands. I'm going to, um, I'll give you a link to the stack overflow answer. Cause I'm not going to steal the credit from, uh, the, the person who answered it,

but, um, there's, he But he provides three or she provides three different ways to see, like depending on what data you want for that, you can see where your stash came from. So assuming that this is stash zero, so it's going to be stash at, and then in curly braces, zero, you can either do a get show the stash ID and then the carrot to see like the,

the actual changes that were stashed, or you can do a get log minus one, one line. And that stash ID carrot again is going to be at the end of all of these. And it'll show you the, the, the hat, the short hash,

the, and then the commit message at the time of that, or you can do a get rev parse. And again, that stash ID and it'll show you the full hash of that. And so it saved my bacon because I had stashed some code away that was still a work in progress and then went back to doing other things. And that included like checking out the latest version of the dev branch and, you know, uh, you know, had,

had moved along. Right. And even in the branch that I had originally made the stash off of, I had done rebases in there too. And so I couldn't just like pop the stash back into the original branch and have it working. So this was like, by finding out what the original commit ID was that I was at when I made the stash,

I could do a get checkout of that commit ID was that I was at when I made the stash, I could do a get checkout of that commit ID, then apply the stash ID that I wanted and then see my changes. And I was able to create a new branch from there to continue on. So again, if you ever wanted to, you know, find out like, oh my gosh, where did this stash come from? And then also to, you know, find out like, oh my gosh, where did this stash come from? And then also too, you know, this one, uh, again, putting no thought processes, uh, you know, keeping consistent with our theme here, no thought processes behind the things we say.

But if you ever like, uh, you know, some, some people are gonna be like, well, duh, Michael, but a helpful tip, like for debugging your like, well, duh, Michael. But a helpful tip for debugging your Docker images, after you've built them, sometimes you want to get into it to see like, hey, does it actually have the latest changes of the code or the configuration or whatever it might be? Or did something take or not? Right? And for security reasons, you might have, it might be, or like, did something take or not? Right. And

for security reasons, you might have as part of your Docker file build instructions, you might be changing the user to something more restrictive. And also to, uh, you might have like a default command or entry point that's going to be in there. So like, if you were to simply do a Docker run of that image, then it might, you know, automatically take you into something that you can't, uh, you know, really poke around at the operating system. Like take a Mongo, a Mongo image, for example, the Mongo image by default, uh, will take you straight into a Mongo shell, right? If you do a Docker run on it. So you can use dash dash user and then as well as dash dash entry point, and you can override both of those things. So you can say

dash dash user equal root dash dash entry point equals SH. And now you're in just the plain operating system and you can poke around and see whatever you needed to see. And by the way, now, cause in your typical Docker run statement, right? You're going to say the image name and then some command that you might want to run,

but you don't have to do that other command now because you've already over ridden the default entry point to be SH. So it's just going to dump you straight into to a show we really should do the docker episode yeah for real another one okay so i i'm glad you said that glad you said that because in in our uh you know regular work life there have been things that have come up that have been presented to us, like about the inner workings of Docker. And I have been trying to find the equivalent of you. You recall the get from the bottom up article that we did.

Oh, right. I loved the approach that that author whose name eludes me right now. Um, but I, I loved the approach that that author took for that series. And I was like, I wonder if there's something out there that is similar, but for Docker, because we have done Docker episodes before. all that because if you're listening and you're like oh i know exactly what you want michael and here it is then uh you know please do let me know because i would love you know it but we we talked about we did the docker for developer series that was episode 80 we've done a little bit of docker

since then a little bit yeah so just like off the top of my head like thinking about things that are kind of tricky or weird or you might not know it's like talk about mounting caching the various ways of caching secrets uh entry points is a great one build kit build kit d the difference between those two things the file system moby the file system like how do you even start with it you know like so many questions that that have come up over the years that it's like yeah i would love to be able to have like here's here we've we did a deep dive on it right so i'm throwing i'm putting that out there you know feelers out there like if you know of one uh definitely hit me up and, yeah, we would,

I would appreciate it. Cool. So, uh, yeah, subscribe, whatever. Later.

Nah, man, you can at least go check out our Slack, coding blocks.net slash Slack. Yeah. And maybe on the Twitter verse. Hey, no spoilers in there.

The episode discussion channel. Maybe watch out for that one. Other than that, no spoilers. It's great. It is a fantastic place to go hang out. Wait, there's spoilers in the episodes channel? There was concern about it. Maybe not

concerned. Spoilers about the episodes? Someone mentioned that they had been avoiding joining the episode discussion channel because they were only on episode 160. I missed that conversation. Okay, I get it now. Yeah.

Thanks, by the way, for joining and hanging out. Come on in. come on in. Come on in. Yeah, so subscribe to us on iTunes, Spotify, and more using your favorite app. Oh, you're really doing it. You're doing it. Okay.

You're doing it. No, no, go ahead. I'm really not doing it. I'm really not doing it. You've got to finish it. I'm like at it. Yeah. Send your feedback and rants and

questions to X hey and you can get the show notes at codyblocks.net slash episode 221 one more than the last one 221 yeah n plus one all right

we good