Coding Blocks - What is OpenTelemetry?

Episode Date: August 21, 2023

In this episode, we’re talking all about OpenTelemetry. Also, Allen lays down some knowledge, Joe plays director and Outlaw stumps the chumps. See the full show notes at https://www.codingblocks.net/episode216 News What is OpenTelemetry? It’s all about Observability Reliability and Metrics Distributed Tracing To truly understand what distributed tracing is, there’s a few parts we have to […]

You're listening to Coding Box. Do I say the episode at the top of this show? I don't think I do. You're just, whatever. Hey, you know, can we just, let's do it. Let's, can we change the opening from now on? Like, does it have to be, do we have, you're listening to Coding Box, whatever. You probably figured that out already.

Yeah, I'm Michael. I'm Joe. Oh, that hurts. I'm Alan. I'm Michael. I'm Joe. Oh, that hurts. I'm Alan. We're going to have to break him, Joe. We're going to have to. My body temperature just went up like five degrees. The look of panic on his face was just, I wish everybody could have seen that.

Did you screenshot that, by the way? Maybe that should be the hero image for this this episode is the look of panic on alan's face when i didn't do whatever the normal intro is but you know what's funny about that out of the three of us i'm probably the most off the cuff guy like i'll go on vacation tomorrow like i don't need to plan for it but But this, we're 216 times of this. How are you going to change it today? Anyways, all right. Okay, so with that, this particular show, see how we skipped all the fun stuff at the top.

We're going to start to talk about open telemetry. And we'll get into what that is here in a minute. But first, we do have a little bit of podcast news so as always uh we're gonna let michael take a stab at this it's always me it's always me all right so uh yeah we want to say thank you to those that left us a review so from itunes we have uh lynn june i hope i'm pronouncing that right three, three nine, because the previous 338 were already taken. That's right. But thank you both for your,

your reviews. We really appreciated reading this. All right. And then next up. So I finally did what I mentioned. I was going to do like, I don't know, four episodes back or something where I was going to walk through and try and

get chat GPT to actually write an application for me by me, just sort of telling it what I wanted. And so I made that video. It's about 30 minutes long. Uh, my recommendation is drop it on 1.5 or 1.75 X, but I, to me, it's still super cool. Like I even went back and watched it afterwards and it's, it's still super cool. I even went back and watched it afterwards, and it's very neat. If you've been using ChatGPT to answer development questions and all that, that's super cool. I do it as well. But to see what it can actually do for you from scratch, like treating it like you're a project manager or a product manager and just telling it what you want is kind of a cool perspective. So go check that out. It's kind of fun. And how did you have to deal with all your

dependency management for that? Was it like handled for you or? We didn't get there. We didn't get there. And the credentials, like, did it not just ask about that? But yeah, no, I mean, seriously, though, like, it's still sort of mind-blowing to me what you can do with it, even though it may not be perfect, but it's still very cool. I'm waiting for somebody to submit an app to the App Store that was written entirely in ChatGPT. Oh, it's going to happen. If it hasn't already. No, it can't. No, you know it can't, and I'll tell you why.

Because if you ask it to make a UI for you, it'll tell you that it cannot, but it can give you some pointers on it, which is kind of interesting. So maybe in the next version, I don't know. I've heard previous, I've heard in other podcasts where people have described where they did get UIs out of it.

Interesting. Yeah. Cause I asked it, I was like, Hey, can you put a UI on it? And it. Interesting. Yeah, because I asked it. I was like, hey, can you put a UI on it? And it was like, no, I can't. I don't know how. Maybe the questions were asked differently. Probably.

I mean, I guess if you lead it directly, you might could get it to build you some forms and stuff, possibly. Dear ChatGBT, as a UI developer, how would one write this? Right? Yeah, you have to trick it. And you had to say, dear would one write this? Right. Yeah. You have to trick it. And you had to say, dare chat GPT before. Right.

Definitely. We have to be polite. This is modern society. Like, what are we doing? I do like it that it always replies with something like, certainly. To me, it reminds me. There's an old cartoon where it's certainly. That's not a cartoon.

That's the Three Stooges. The Three Sto three stooges that's it there it is yeah yeah it says it's important to a lot like it's important to remember right yeah it's it's pretty cool it's funny the one comment i got was like you know having listened to the podcast so long i'm surprised it took you so long to get into it um that's fair but it's also fair to say that I did the initial type thing of that like several months back and I just got around to recording doing it again. Right. So either which way, it's still fun, still cool to try. You know, as a follow-up though, as a similar follow-up, there was that, what was the Kafka one that we talked about a while back as a similar follow-up there was that um the the what was the kofka one that

we talked about a while back as the tip of the week that i think joe gave like ka dent or something no i gave it for micro g uh i tried it and i couldn't get it to work i was curious to see if you guys ever tried it but at least in my environment i could never get it to to work so i wasn't sure because i wasn't paying for it either so i wasn't sure if that was like why i wasn't able to get it to connect i haven't tried i was running a cluster in gke and trying to use it there so so i was like relying on port forwarding and things like that and And I'm like, well, I don't know if it's because of the licensing. I'm not running this locally. Yeah, possibly.

I don't know. You know what would be interesting? So I'm going to be giving a talk to some people here soon on telepresence. I wonder if you could possibly trick it with that. And we might even do a show on it. I know that I've mentioned it as the tip of the week, but I've gotten deeper into it,'s it's got some amazing stuff so it was a deck was the name a deck yeah yeah i only use kafka 2e now kafka 2e wait oh it was that that was one of the

2e's that you talked about last episode i don't remember if i haven't i haven't tried this one but there is one yes there is people and you can look at consumer groups you can yeah logs kafka brokers i'm going to uh drop this in the show notes here you need more two weeks in the world even though even though it doesn't matter if we put them in the show notes because apparently we just won't even read them. So, you know, because we didn't even do the opening. So here we go. Let's get into this. Well, we did the new opening. That's what it's like now. And that's the world.

Yeah. You just start, hit record, and go. I got to get with it. Yeah. All right. Come on, Boomer. So here we go. Boomer one.

Open telemetry. So actually Jay-Z brought this up a few, maybe a month ago now, maybe a little bit longer. First off, the website is open telemetry.io. And I want to point out a few things. So this is a quote unquote incubating project on the CNCF, which is the cloud native computing foundation. So incubating that in me triggered memories of Silicon Valley with the incubator. I was like, I don't really know that we should be using software that comes out of this incubator. You didn't trust the middle out compression algorithm from Pied Piper, right?

So it's, it's important to know that incubating means this. It actually says it on their site projects used in production by a small number of users with a good pool of contributors. So it's actually technically production ready code that just hasn't been like widely adopted yet i guess is probably the best way to put that so you shouldn't be left out to dry you know like if if something goes wrong with it you actually have a community to reach out to and there are contributors to that community that are probably fixing bugs and doing that kind of stuff right so that's that's really where this is at but wildly wildly uh widely wildly supported like uh various cloud providers various tools like um what you call it uh grafana just added support for it um pretty much anything that you're going to be using if it's another open source tool and it does anything sort of like telemetry ish it's

probably gonna have some sort of connector and be able to use it but yeah production ready no no it's being used in production yeah but you said you said production ready now was the part i was taking issue with is like i don't know that i would like it's being used in production in places so So I guess that's better than... I have shell scripts that are being used in production. Does that mean... So you should be happy with this, right? Well, let's look at other things that are cloud-native incubating projects

and see if any of these spark joy. And we'll have links to OpenTelemetry and the CNCF also. So that's where Joe's getting ready to go. So it's cncf.io is the cloud native computing foundation oh yeah well we're not using any of these there you go all right so but you should feel comfortable with some names hit me uh so there's gonna be stuff that you've heard of that we've talked about a little bit sometimes but nothing uh like i thought there were more uh thanos um falco i'm just like hitting the ones that like we may have touched on

before uh let's see psyllium prometheus dude prometheus is incubating that's absolutely being used yeah it's absolutely being used all over the place k Kubernetes is shown as, oh, no, this is graduated. I didn't scroll down to anybody. I'm sorry. Man, it was supposed to take me there. See, there you go. So I wouldn't necessarily call it production ready just because in production in some places, right? I think that's the point that they're trying to, I think that's what they're trying to clarify. You know, the point that they're trying to clarify, the point that they're trying to make.

It is used in production by a small number of users with a good pool of contributors. So, gRPC is considered incubating, which is hugely adopted all over the place. One place. No, gRPC is used by tons of stuff, right? Yeah. So, so at any rate, yeah, I mean, look, just like anything, you know, you have to look into it and see if it's going to fit whatever you need to do. But let's get into some of, I mean, that's fair because even if it is like quote production ready, you know, like you're still going to have to like support it. So, you know, SQL server has been around for how many years, but you know, you're still going to have to support it. Wait, you actually have to do things with it. I thought it just ran. Maybe not so much. All right. So let's get into what is actually open telemetry. It's a collection of APIs, SDKs, and tools that are used to instrument generate and collect and

export telemetry data. Okay. That's a, that's a whole bunch of keywords, right? A whole bunch of buzzwords. So I purposely didn't go through these because I wanted to play like the questionnaire guy. So when you say that, though, that makes it sound more like it's a standard of like, hey, here's the things. There might be some libraries out there for it. Right. Uh, for the, for the APIs and SDKs, but it's, it's not like, um, or maybe it is more like, uh, Oh, what was that one? Search the J, um, that we were using. Oh, yeah. Jaeger. Jaeger. It actually, I believe brought Jaeger into the project as one of the things, right? Jay-Z, you've probably looked into this more, but I want to say that it encompassed Jaeger.

Yeah, I don't remember the lineage because OpenTableLimitry kind of came out of a combination of a couple of projects. I'm trying to look it up now, like OpenCensus and something else. And so we were using Jaeger briefly, uh, but I don't think we were using open terminal limit tree at the time, but I'd be surprised if it didn't use it now. So it did. There's actually a page that says that it did have Jaeger.

It had a Jaeger agent collector and gesture in one. It says not to use it because it was experimental and discontinued. So my guess is they probably replaced it with something that was similar but more bespoke to what they were trying to do but yes we were using open tracing with it at the time yep and and really so to to your question outlaw yeah there's tons of libraries around this thing already so really what open tool image and we'll get into this as we get further along, it'll make a little bit more sense, but it's a collection of tools that come together that basically help you analyze your software's

performance and behavior, right? So I think it's only been a couple episodes back, we were all talking about the fact that, you know, you need logging, right? You need really good logging, but logging a lot of times doesn't connect the dots for you, right? Like you'll see a log and then you have to go trace down something somewhere else and you have to go find something somewhere else, right? And same thing with dashboards like Grafana dashboards or anything like that. Like it's telling you something, but now you got to go look through the logs and try and find stuff, right? So the whole goal of this is to make everything a little bit more um observable so so to to paint a picture as like why you want this or what this is doing for those that may not

have ever experienced this if you if you do any kind of like let's say you just had like a simple uh uh what i think you know that the old school three-tier architecture you know the web server the app server and then some kind of database right and you want to trace like something's happening and you're not sure what so you go looking through your apache logs and you're like okay well i see the request came in here and then you go digging through you might have your app tier logging out something too so you go digging through that kind of log and you're like, okay, I see that happened here. And then in your database here,

you often don't have anything. You're just like, you know, record was changed, you know, maybe you have some kind of a history on that table, but you know, maybe not. And so you're trying to piece together what happened. The idea here is that what if you lived in a world to where that request comes in from the user, they clicked on a button to initiate some action or whatever they did to initiate the action. And there was an identifier that followed that request through every part of the system from the web server to the app server to the database

server, et cetera, so that you could follow along every part of the journey that that request took to and from on its way back around trip, right? So that you could see like, oh, this is where things went sideways. And I'm just describing those three tiers like but it would if it was like more complicated a more complicated environment right especially in like a streaming data kind of world you know where uh something's coming through so that's the idea that we're trying to start when we talk about like uh open telemetry and uh uh jaeger what was the other one you mentioned jay-z the uh something tracing open tracing and open census both which are kind of uh deprecated right but but the point being is like all of those kind of things we're trying to solve this issue of like i want to be able to follow from a all the way

through z like the single request not have to like try to piece it together by okay well this server was configured in utc and this was the time stamp where that request happened and this other one is in eastern time zone and here's where it's request happened because you remember those days were awful yep yeah but uh so open telemetry is definitely uh definitely encompasses that but i think it's also a lot more it's a lot of other stuff like that was kind of like the open tracing side of it which it like brought in but then there was the open census side, which I don't really try to figure that out now. Well, here, let's let's go through some of it, because I think we'll end up touching on a lot of what it is.

So one of the key things here is open telemetry is available across multiple languages and frameworks. So if you just go to open telemetry, I.O you go to the maybe the getting started or something, at any rate, they'll have a whole list of languages down the side that they support with various libraries and frameworks and things. So it is documentation, documentation, documentation. Yeah. So there's a ton. So let's get into a few of the things about what what they're actually trying to accomplish. Right. So, again, it's all about observability. So let's get into a few of the things about what they're actually trying to accomplish, right? So again, it's all about observability. So what they call it is they want to be able to understand the system from, quote, the outside, right?

And what they mean by that is they don't want you to have to be able to understand the inner workings of a system to be able to see what's happening with the system, right? Like you should be able to look at some sort of view of it without knowing the Java code or the.NET code or whatever and see what's happening. So their whole goal is to be able to troubleshoot difficult problems and answer the why is this happening question without ever cracking open a source editor, right? Like that's the goal. So to answer these questions, this is really important. They say that the application must

be properly instrumented. And so like outlaw just mentioned, he saw the various different languages under the instrumentation. That's why. So it says to make sure in an application is properly instrumented that means that it must emit signals like metrics traces and logs and you know it's properly instrumented when you can completely troubleshoot an inch an issue with the instrumentation that you've put in place right so that's that is the big thing right there you're not having to jump around and go look here and then go back to the source code and see oh well it went through this method and then no you should be able to look in one place and see it um i found a list of vendors on upland total machines website

just kind of googling around and it's a huge list and this is just like companies that make tools specifically kind of in this area. So like Honeycomb, AWS, Azure, stuff like that. Also like Datadog, you know, it makes sense that Datadog is going to be able to process that data elastic. There's just a ton of them. GCP, pretty much any major cloud vendor or product has support for OpenTelemetry. And it says, so this page is pretty cool it says it's a non-exhaustive list of organizations offering solutions that consume open telemetry natively so that's pretty awesome yeah it makes sense if you're doing any sort of like telemetry which you know just kind of

basically speaking like the time i first heard telemetry about a product it was like publishing stats about what customers were using and what features they had turned on and how often you know they were using it and stuff like that um so it kind of makes sense if you're building any sort of product that you're going to be shipping to customers or you know obviously your own product too that it makes sense to support open telemetry because you can plug it into so many different things yeah and and i think the the thing that led jay-zZ down this path and really makes a lot of sense, especially if you're living in a world with microservices and stuff, is if you don't have some sort of standard way of doing this,

it's really tough to tie your requests together across different things, right? Like you could write a ton of code to try and make that happen. And this standardizes it. And that list that he just provided there gives you a really good reason to use this because now you don't have to work as hard to be able to visualize what you've got. So, yeah,

like if you've got a Grafana and Prometheus already set up and you've got a dot net app, there's probably some sort of dot net open telemetry library. That's going to publish a bunch of like a base set of stats that are useful to you that you can plug right in like today. okay so in my example before though like you called out well that's just focused on more on the tracing aspect of it you're saying this also encompasses like the observability metrics that we had previously talked about i think related to like the the sre book from google um for as an example right or

even the devops handbook right like it's it's also being able to see like to alert like oh hey you're you're hitting you know 90 cpu utilization on your server you might want to do something about that like that sort of thing so there is a section dealing with metrics uh i don't know how comprehensive that is compared to like a you know prometheus style thing um but you know you can get at least some base metrics but also um things like what service endpoints are being called um if you've if you've got a like a front end hooked up to it you could be like what web pages are being called what urls uh how many sessions you know stuff like that that's a little bit more um i'm sure you can get that same thing from metrics but you're probably going to be either relying on your framework to do some of that work or you'll

be doing some more custom work yourself so after the break we'll be getting into more of the specifics of the different types of things that it offers but along along those lines, I don't think it's, it's not going to do like what a Prometheus does, what it does more of. And Jay-Z and I have experienced this in the Java world is, uh, I think by default, I can't remember exactly what the Java library is for exposing metrics, but it's, they have some sort of standard base library. That's kind of a pain in the butt to use because like, if you want to add multiple labels to it, it's a really weird, funky way of doing it. Or you could use the Prometheus SDK that they provide and expose metrics that way. I think what OpenTelemetry does is just give you a library that you can use that is a way to say,

hey, here's a counter, here's a histogram, whatever. And then it's just a more standardized way of doing it instead of using a Prometheus client or the Java-based one or whatever, right? That's really what it's giving you, is just a way to expose the metrics from your application. It still needs something. Prometheus is a database. It know, it's got like an API that you can query. It's kind of a lot of other things, too. So OpenTelemetry is just a protocol. Yeah, it's just to be able to expose to it.

The way that they actually put it from OpenTelemetry is it's a way to emit signals. So it gives you a way to emit metrics from your application in a standard way a way to emit logs well that's what i was trying to like emphasize then because like you know the example that i gave was clearly just focused on tracing from a to z like the path that that that request took but i'm like looking through some of the um links in here oh wait that took me off to a Microsoft documentation. But it clearly seems to be more than just that. Like there's calls and APIs for talking about performance and everything. Yes. But it does get a little bit confusing, too,

because some of the documentation from OpenTelemetry took me over to Microsoft. So maybe that's where I'm making some confusion. Yeah, let's come back to it after the break. So first, we need the outlaw, I ain't too proud to beg moment here. Oh, maybe I am too proud to beg. Why? Have you ever known me to beg? Salt and pepper reference?

Okay. Well, yeah, of course I did. Of course I missed that. You kidding? So, uh, yeah, as I didn't say earlier,

um, if you haven't already left us a review, we would greatly appreciate it. You can find some helpful links at www.codingblocks.net slash review. And it's crucial that you use that www. And in fact, you should probably also include the s the little s in your https thing so yeah all right well with that this is jepper blocks coder db coder

jepper code that hurts i can't say it the evolution of this is amazing yeah oh man that hurt um okay where we at so this is episode 216 so according to taco's trademark rules of engagement jay-z you go first oh yeah here we go and your uh categories are whose line is it anyway? Got a bridge to sell you pitch. Perfect. Classic kids books. Yes. Chef or Q force.

And each response begins with Q. Wow. Jeez. Let's go kids books, I okay classic kids books one two three four or five level how sure are you four four oh i like it you read some kids books here we go yeah problem is this one goes off with another like proper noun, which is like my Achilles heel. Any capitalized name is over. In a book by Johanna Spirey, this title or this title orphan is sent to live with her grandfather in the Swiss Alps. Oh, wow. I don't know this at all.

So just give me one more second to think. I actually do, and I can't think of it. What is Little Orphan Annie? No, it's not Annie. For the steal, Alan. It's not even the title. It's not Pippi Longstockings. Doggone it.

Heidi. Heidi. Oh, wow. Oh, man. That's classic for sure. It is. So, Alan, it's your turn. Your category choices are cats through time biology holidays and observances adverbs that make your day that they all end in ly in L-Y. Variety or Hollywood Reporter.

Adverbs for 400, please. Adverbs that make your day. Well, you mean the fourth? Level four. A body part starts this word meaning cordially

in a more exuberant sense. It's the way you welcomed. It's the way we welcomed you to the stage. Of course, this is, you know, I didn't welcome you to the stage, but you know,

pretend that I had, uh, could you repeat that one more time there, Alex, Mike, thank you. Uh, although now it's like, uh, Miriam, Kim, Mike. Thank you. Although now it's like

Miriam? Kim? There I go, messing up my name again. Why did I even try? Why do I set myself up for this? So, a body part starts this word, meaning cordially, in a more exuberant

sense. It's the way we welcomed you to the stage. And this ends in an ly uh i i got nothing i don't know jay-z i don't even understand the question so the body part that ends in ly that you would use to welcome no the body part doesn't you do this with the body part like to welcome someone on to welcome somebody on the stage and clap is the only thing about it there's no oh why there yeah i got nothing man what is it heartily heartily welcome somebody to the stage man that was okay get out of here all right this. I don't know that we've ever gone this far without an answer. So here we go.

J.Z. Third round. Advanced criminal law. A little poetry. U.S. presidents. Of course I'm not breaking up with you. Music of today. Or the last one is, uh,

around Philly with Ryan long. Not that one. Jeez. Um, Oh man, not the music one of today. Uh, I forgot all the others.

So the second one. Do you want me to repeat them? Yeah. Advanced criminal law, a little poetry, US presidents. Of course, I'm not breaking up with you. Music of today or around Philly with Ryan Long. Let's go. Let's go presidents.

Oh man. Really? Okay. Let's go. Let's go Presidents. Oh, man. Really? Okay. Yeah. What's your one, two, three, four, or five? Hey, because this is the last round, you can't go lower than three. What?

No, there's no such rule. No, because he gets to go first. No, that's the rule. I've heard that. I've heard that. So let's go with four. Okay. I'm trying to skirt it a cnn headline said this man had taken 8200 round trip train rides to delaware cnn train rides delaware uh i i have no chance i can't even think of a president from Delaware right now or why someone would go to Delaware.

And it's certainly not in the age of CNN. So let's go with who is Taft. Okay. Let's do Bill Clinton. Man. It was Biden. Yes. Was it Biden?

That's his hometown. That's his home state, right? I thought, I know he was from Pennsylvania, I thought. I don't know. Wow. Or we failed. Maybe I'm thinking like where he currently lives. I thought he currently lived in Delaware.

We failed miserably all right so there's the final round then all right here we go the category is world landmarks built of more than 18 000 metal parts and two and a half million rivets it was the world's tallest man-made structure from 1889 to 1930 statue of liberty i have another guess okay what is it does it work eiffel tower joe wins man eiffel tower come on wow that is the closest one we've had so far we had to go we we've gone tiebreaker before but not because it was zero zero because we were terrible yeah that's amazing you guys suck right i mean whoa did i say that i thought i was muted meanwhile if i was given any of these questions i would just be a deer in headlights some of those hardly yeah that was tough i want to wonder what the breakup one was i i don't think i've gotten it that was the one that would have been most interesting i think yeah you're right dang

i tried of course not do you want to hear just a question from it like yeah yeah give us one so question number one of course not honey think of it as this feature at a play when you have 15 minutes to beat the line at the bar or bathroom. Intermission. Yeah. I would have gotten it. See, I could have won. I could have won. But now I'm on a zero game winning streak again.

Yeah. Wait, did you win last time? No, I didn't. That's why I said I'm still on a zero game. Well, you said again. Again. Yeah. Yeah. Continuing that zero game winning okay all right okay so back into it where we left off

is exactly where outlaw and jay-z were headed which is the reliability and metrics portion of open telemetry so telemetry the actual term itself means it's data emitted from a system about its behavior in the form of metrics. So numbers, traces or logs. And we'll get to what's special about a trace here in a second. Reliability. It's the system behaving the way or is the system behaving the way it's supposed to, meaning not just that the thing's running, but is it actually doing what it is expected to do? Right. You know,

like going way, way, way, way back into the nineties. I remember we had a, a server and we had a major outage and part of the, the takeaway from it. Like, you you know we didn't call it like a post-mortem kind of thing at the time but um you know we were trying to figure out like okay well

what went wrong and part of the part part of the problem was that there was a script that one of the team members had written to verify like the status of things right but all the script did was checked that the server responded on that port not that not that it was like what you expected their response to be from that port but just that yep i tried to reach that port and i got something back didn't matter what it was that came back yeah yeah and it's funny i mean we we say that but it's real easy when you're just trying to get things in place to do things like that. Right? Like it's real easy to not go after what's real. It's, you know, Hey, let me get the low hanging fruit here. Is it still alive?

Yeah. So that's why this one like hit a little home bit home, you know, close to home for me. Cause it was like, well, is it up and running up and running well yeah technically it was up and running it was it working no but it responded like you could go to it and you'd get a response it just wasn't what you wanted yeah yeah so super important the next bit here is metrics so when you talk about metrics it's nothing more than numeric aggregations over a period of time about your application or infrastructure. We have a few examples here, CPU utilization, error rates in your application, number of requests per second. Like there's tons and tons, you know, memory utilization, all kinds of things that you can do there. again it's just a number just space cough cough not not that we've seen that lately yeah um all right so sli we've we've talked about several of

these in previous episodes i think when we were doing the g uh the google um site reliable site reliability engineering stuff easy for you sir yeah right um soI, service level indicator, it's a measurement of a server's services behavior. This should be in the perspective of a user or customer, right? So even back when we were talking about this stuff, in terms of the Google thing, right, like Google Cloud Storage, their SLIs were, you know, how, from a a user's perspective is your cloud storage available, right? Are you getting errors back when you make the request and all that kind of stuff, right? So that's, that's your SLI. Um, or another one that they had as an example in their, in their docs for open telemetry was how fast a webpage loads, right? As a user that impacts you, you're actually going to see it and feel it. SLO is the service level objective. This is the means of communicating the reliability to an organization or another team, right? So imagine a customer wants to come up and say, hey, is my thing working? You might have a

dashboard that shows them, yeah, everything's responding within one second or something, you know, who knows? So the page returns back in 300 milliseconds, but your objective is 200 milliseconds or less. So you're exceeding your objective. Right. And then basically they're just saying that you accomplish getting SLO by attaching an SLI to a business value, right? So whatever you're trying to deliver to your customers. Now, this is where we start getting into the meat of what OpenTelemetry is actually aiming to do. And this is distributed tracing. This goes back to what Outlaw was describing at the beginning of this. So before we get into exactly what it is, we got to talk

about the various little pieces of it. So the first one is a log and they define it really well. I actually like what they did in their documentation because it broke things down to very basic things that we all talk about, but we maybe don't give it as concise a definition. So a log is nothing more than a timestamp message emitted by an application. And you've seen it, right? I think outlaw mentioned Apache logs earlier, right? If you go in there, you're going to have a time at the front of the, whatever the message is. And then you might have a few other pieces of information, like whether it was a post, a get whatever URL it went to, and then maybe a time, right? That's's a log there's no context to it it's

just this happened right this happened somewhere and is being spit out by the server now yeah what i mentioned too i didn't see it in notes uh so maybe i just missed it but there's the notion of baggage i don't know if you've come across that not we haven't gotten that far yet yeah okay so uh it's kind of cool to worth mentioning so when you're you know dealing with kind of uh logs or like just tracing stuff through your application open telemetry and other similar um things have the notion of baggage which like key value store so you can kind of associate things like a 10 id or user id or or you can kind of tag uh something in that that those set of tags your baggage is going to go

along with everything uh in the context of that particular trace so you can like log the user id with like uh it's going to be tagged on like each uh line that kind of rolls through there and you can choose whether or not those tags propagate uh onto the next service so if you're talking about like distributed tracing and stuff so that's kind of the um where some of the cool tracing stuff kind of gets involved but i just thought it was kind of i didn't see it under here but i just wanted to call it out because i think that's something that's kind of where some of the cool tracing stuff kind of gets involved. But I just thought it was kind of neat. I didn't see it under here, but I just wanted to call it out because I think that's something that's particularly cool. Yeah. So I think that's going to be when we start talking about the implementation of OpenTelemetry, which will probably be the next episode. But yeah, that is what makes OpenTelemetry so cool

is it has features like that so that you kind of get them if you just use their framework, right. Or their library. So one important thing to call out about a log versus a trace is the biggest difference transaction, right? So what Outlaw was talking about earlier, you know, a user makes a request to add an item to a shopping cart or something, right? That request gets sent to the server. Well, that is going to create the initial trace or log. And then everything that happens after that in a trace is going to be linked back to that original message. Right. So that is the difference between a log message is just kind of standalone out there in the middle of the forest versus a trace is going to show you exactly how you got to where you're going. Traces and open telemetry are heavily used. That's how you actually find your way through what's happening in the application, right? Or actually, I'm sorry, the logs are heavily used.

You've seen them everywhere, right? Everything has it. Even if you go into your cloud, if you're in Azure, AWS, or G Cloud, whatever, if you go up there, they have tons of logs. They don't necessarily tell you anything that you can do anything with. You might see that there was an error, but now you're going to have to go track it down. Right. So logs are used all over the place by everybody.

And but they're not always the most helpful thing. Now, this is where they introduce a new keyword that is that makes sense. So they say logs become much more useful when they become a part of a span or when they're correlated with the trace and a span. So they define span as a unit of work or operation, right? So if you think about that request that the user made to add something to the cart, that is a particular request, right? But the whole operation, and this is what's important. They sort of touch on it in their docs, but they don't go deep on it. A user working inside a shopping cart is sort of like a whole set of operation, right? Like there's all

kinds of things that happen in there and the final operation might be checking out, but you can correlate not just a single request, right? Like add something to a cart. You can correlate everything. So call it like a shopping cart session ID, right? So everything a user does while they're in a cart session, every request they make is going to have that session ID on it. So you can actually tie an entire string of requests together, right? Each request is going to have its path that goes down,

but you can actually tie all those together and see what happened for the entire operation. So the key point here is you can actually trace an operation across multiple user requests if you'd like to do that as well as being able to trace a single request through multiple paths in a system it doesn't have to just be system generated or system acting i guess is what i'm trying to get at here well it almost sounds like we're making the difference the distinction between like, Oh, a request ID. Like I did this one thing and I want to trace that one request through the system versus the session ID. Right. Where like here's the whole body of things you might've done over the course of some period of time. Right. Yeah. Yeah.

Which could involve multiple requests. Totally. And technically they're both basically doing the same thing, right? You're just doing it at a different level. One's like maybe a request to a server. The other one's like multiple requests from a user, um, in the same session or whatever. So, um, let's see. So yeah, a span tracks the operations that a request makes, meaning it helps paint a picture of what all happened during this span of that request or operation. By open telemetry standards, it contains a name, some time-related data, some structured log messages, and other metadata or attributes.

And so they had an example of some of the other metadata, and this is something like the server that was requested or the IP address that the request came from or the browser that it came from, whatever, right? Like there's all kinds of things that can be added as attributes that you want to track along with this. Now, this is similar to what Jay-Z was talking about with the baggage and that when you put this on here, it kind of follows the request, right? Like there's no reason for you in every step of your application to log out the same information over and over and over. You can put those attributes in there. And as that thing's traced throughout the application, those attributes come along for the ride. So like servers, IP or the user's IP address,

user's browser, things like that. Unless you're in GDPR and then you don't, you never track the IP address ever. Maybe, but no, yes, you're right. Yes, it all goes along for the ride with the span that's being done. All right, so a distributed trace, it's the same thing as a trace. It records a path taken for a user as it passes through various services

in a distributed multi-service architecture like micros services or service applications. So this is important to call out, right? If you have your own thing, you're doing your own micro services, you have a bunch of things running in Kubernetes per se or whatever, you can set this stuff up in there. You can also do this on serverless apps like aws lambdas or azure functions or what's what's the google one called i can't even remember is it google functions you know i don't i don't even know if they have it yeah they didn't you asked that question and i'm like you know i don't know it either but they call them cloud functions okay so so have it. Um, but you can do the same type thing across

them. So you can see what's happening from when something triggered to when it hit a function, whatever. So just know that can be tremendously helpful. Now they call it out and obviously they're going to, because there's, they're selling their product. I say selling it's open source. You're not paying anything for it. But tracing is essential for distributed systems because of the non-deterministic nature of the application or the fact that they're just really hard to reproduce locally. I can pretty confidently say all three of us have experienced that, right? Oh, yeah.

I mean, when did this thing hit that service and when did it get returned from it and then when did it hit this other service and oh by the way how long did it take in each part right like it called those three microservices which one of those ate up the most time like all those things in a traditional logging world where you're just logging things from those various services nightmare oh it's impossible you can't you cannot stitch everything together well because i'm thinking to kind of build your own where like you start having a transaction id and now you're starting to carry between services and stuff if you've ever started doing that like just know that you've kind of like a very tippy tippy i don't know sticking barely tipping your toe and tippy toe uh into the

open tracing uh or sorry open telemetry world well that's why i made the the call out earlier about like if you were looking through this log and you're trying to trace that you're trying to like piece things together by only the time stamp that was like your only key to it and you know oh my god the environments where like the the servers weren't set to be on the same time zone and they weren't synchronized or even if they were in the same time zone but they weren't synchronized synchronized yes oh man you're like trying to do the math in your head like or you know follow along in the math you'd have to put some spreadsheet together like okay so this server is in eastern but he's 15 minutes off and it would never be like a clean 15 it'd always be like okay he's he's 14 minutes and 37 seconds behind this other

one's system clock so i have to like translate all of these logs into that yeah you know what it also reminds me that too like speaking of tippy toes, have you ever, we should, here's a never have I ever, never have I ever put logging in my methods and or classes where it said start and end in my methods. Right.

Right. Like, and I remember too, like trying to do some of this back in the you know days of like where especially like c plus plus kind of days where in my constructor and destructors i might like you know put something in there like for for logging purposes to test it you know but i generally tried it as a general rule of thumb i hated doing anything in a constructor that might or even a destructor anything that might have the chance

of throwing a exception but i do recall times where like you know okay i'm a memo it's in here like let's just temporarily put it in and figure out like what's going on i want to trace this thing through right and that's i think that's the key is what he was just talking about and if you don't totally follow what he's saying where the clocks aren't synchronized, just imagine you have a server that's, you know, you have two servers. One of them, for whatever reason, is 30 seconds behind the other one on its clock.

And now you're trying to correlate the logs from one server to another, and you can't do it because everything's, like, pushed out of sync mentally for you. It's, I mean, it's insane. That's part of what this is trying to solve. Here's the real out of, out of sync mentally for you. It's, I mean, it's insane. That's part of what this is trying to solve. Here's the real world of it. Go, go set your phone and or watch to, you know, to where they don't synchronize anymore.

And then set your clock to where you're 14 minutes and 37 seconds behind the current time. And then see how long it takes before your significant other is constantly mad at you for always being late. Right? Yeah. I mean, it's impossible. It really is impossible. But so the whole point of this thing right here is tracing makes it easy to understand and troubleshoot these problems because they break down what happens as it's flowing through the system and then it correlates them, right? Like that's, that is the whole key to this. Instead of you as a person trying to go gather the logs from 10 different places and stitch them together, this does that. A trace is made of one or more spans. So the first one, like I mentioned earlier, it's actually called the root span. So imagine you made that request to add something to

a shopping cart. The very first end point that that thing hits, it's going to create this root span that is now going to be the parent of anything else that happens. Now there can be more spans that are added. So that will be the parent of any child spans that come in. And then those child spans could also spawn off more child spans. There's no limit to it, right? But it's just depending on how you start adding data to these things as they come through and trace through the system. Okay. So if I were to equate span to session, right, I could have a span that represents the entire session, but inside of that giant span, there'd be like, okay, well, here's the one span for the action where you authenticated.

Here's another span where you added something to your shopping cart. Here's another span where you initiated the checkout. And inside of each one of those three, there'd be like a whole series of other traces uh, tray traces that I did for that particular request. Totally. I mean, just imagine like the add to shopping cart, right? Like it, maybe it's calling a microservice to make sure that there's inventory. Uh, and then that's going to be a span, right? There may be, there's another one that goes and I don't know. I can't think of anything else right now. Whatever. Verify the credit card.

One problem I had a million years ago at another job where it's like updating the address would fail sometimes. We never knew what was going on. We go check it out. Go through checkout, change the address. It's fine. You go into my profile, change the address. It's fine. But somehow we're still, change the address, it's fine.

But somehow we're still getting these errors because no fields when you're changing address. What's going on? Well, if we had tracing at the time, we would have instantly seen like, hey, that's funny. There's not a span here for coming from the checkout. There's not one coming from the address. There's some third-party API that we kind of forgot about that customer service was using to change addresses. and it's missing some field that we added a month ago and it's not used that often because people don't usually you know call to change their addresses for a website but it happens just enough and customer service is having a hard time doing it and that's the kind of thing that could be a mystery for months that's

just instantly solved when you have information like this well that's that's amazing um so here's one of the things that's that's really cool about this is when you start creating these spans and they're attached, you can visualize it now with tools that tools like OpenTelemetry provide. I have a link here to the page where they actually show one, but it's basically called a waterfall. And if you've ever looked in Chrome debug tools or anything like that, you can actually see request waterfalls, right? And this is very similar to that, right? So it will actually show you the root span than any child spans.

And then if they branch off, right, do things in parallel, then you'll have multiple spans that are sitting beside each other in this, in this waterfall view. Yeah. So it looks as soon as I clicked on that link, I was like, Oh yeah, this looks exactly like what you would see in in the debugger. Yeah, the Chrome requesting is it's identical. So that's all that is the groundwork to sort of tell you exactly what the pieces are to allow you to actually do tracing. So in, in the next episode where we continue the open

telemetry, we'll actually talk about some of the specifics and what open telemetry actually provide you as an application developer. So like the things like what you were talking about with the, um, the baggage, right? That kind of stuff. We'll get into it. So you'll actually know why you want to use this and what it's going, how it's going to make your life way better. Especially if you live in a world where you have more than just a three-tier architecture, although it can be helpful there as well, because you can, you don't have to think of traces and spans just being things that are running across microservices. They can be things that run through 10 methods within your application as it's going down and doing different things. Right.

So you can get benefits no matter what type of app you're running. So yeah, with that, we'll have some, some links that we like some resources and stuff that will be in here. And I think now we head into Alan's favorite portion of the show it's the tip of the week i'm wondering hey do you have one outlaw are you kidding me of course i do i'm a professional sir i come to the show prepared to talk business and business is my game i like it i like it well i was kind of

hoping that a lot wouldn't because i i kind of wish you would read my tip oh really yeah are you up for it uh yeah sure i've got a little script here here you go okay so am i am i doing oh there is only the one part no there's multiple you got all the parts actually all of them oh well no somebody characters okay okay trust me yeah you got both characters there's a narrator and a user all right here we go ted tbt worked very hard on this by the way all right here we go hey there windows wizards tired of those unruly apps playing hopscotch in your task manager well hold on to your seats because we've got the magical trick you've been waiting for. As the camera zooms in on a hand dramatically holding a control key. Introducing the control key cha-cha. Just hold down that trusty control key while gazing at your task manager and watch those misbehaving apps freeze like deer in headlights. No more app jumping shenanigans.

It's time to dance to your tune. Cut to a satisfied user with a thumbs up. Thanks to the control key cha-cha, I've got the power to keep those apps in line. Is that legit? It's legit. So how many times have you hit uh control alt delete task manager and you go to like shut down amazing job by the way that was perfect that was exactly like i imagined no way i could pull that off uh so you open up the task manager and you go to like shut

down the process and you click and oh it because it moved you clicked on the wrong one and you're chasing after it and you go click and you know you finally change it down but it's a pain in the butt because those things are jumping around based on whatever you got it sorted by turns out if you hold the control key it just freezes i never knew man what you want yeah me either that's amazing like don't you feel silly now for how many times you like chase trying to shut down you know chrome or whatever it is like the crypto hacker you know you know what's so amazing about that is there's going to be somebody who's going to be like well you just sort it by alphabet alphabetize it that's not

true it doesn't work there either it still bounces around yeah because new things start up and whatever that's a problem yeah so yeah i used to try to sort by various things in order to slow it down uh but yeah ultimately stuff was still around so yeah control works as well as on twitter uh i don't know who it was that tweeted i just kind of saw it it had like a million retweets so uh yeah that's it and chat gpt wrote a commercial for me this this deserved a million retweets this is amazing yeah i'll never forget it i'm going back to windows now that's all it took that's right okay when convincing uh commercial read and you're like yes yeah all right so key cha-cha i actually had things that that happened to me or i stumbled across this week that i thought was interesting because it seems like i never actually oh yes have anything cool so um first just steal this one no just kidding so if

if intellij is being a dog for you if you happen to be programming in java or whatever i have a link here it's from 2021 but it's mostly still the same type stuff uh ways to improve your ide's performance and it's everything from, uh, increasing the Ram, which by the way, is buried in the most bizarre place in IntelliJ. You have to go to the help menu and modify your Ram settings there. Like that, that makes no sense to me. Um, but at any rate, they have all kinds of things in here. Um, disable unnecessary plugin, some, some common sense stuff, some maybe not so common sense um but at any rate take a look through that it's pretty good the next one this one this one completely caught

me off guard so do you guys do this like you're in a file you're doing code and if you see like an orange tick over on the right side where it's like giving you a warning. Like it just bothers me. Like I want those gone. And I moused over this one. I had a logging dot trace message. And the thing about Kotlin is it's amazing with string interpolation. So just think JavaScript where they use the backticks and you can do interpolation inside the string.

Kotlin has that as well, except you don't have to put those silly backtakes. You can just use regular quotes. But it was complaining at me that in my logging, I was using string interpolation. And you could go up there and say, quick action, fix it, and it would turn it into a string template. The same garbage that we've seen from C++ back in the day, or however far back it goes, where, you know, print, instead of putting name there, you'd have like a percent, right? And then a comma outside of it, and then the variable that it's going to replace. It's a similar type syntax, but they call that a string template. Well, I was about to go in and just disable the thing. So I went to the settings and I looked at it and they had a warning message there that said, when you do string interpolation, even if you don't have tracing enabled for your logging output,

it's going to evaluate that statement every time. I don't know exactly what it's doing in the compiler that makes that so, but by changing it to a string template to where you're not actually interpolating those variables inside the string, it'll completely skip it and won't cause any additional processing during the running of the application. So wait, say that again. Even if it was, even if it was never going to execute it, you said, right. So if you had your logging level at info, it was still going to process that trace message, even though it wasn't going to actually run it, it was going to process it.

Right. Um, whereas if you do the string template instead, it just wipes it out completely. I'm guessing. Um, I don't know. I don't know what kind of optimization I didn't go look at the byte code or anything, but it was interesting that that was the thing. And so that is why it was giving me the little yellow mark on the side. Yeah, I guess it makes sense that, like, if you have an argument as a string and you're doing some interpolation in there,

it's going to interpolate that string before it passes to the function. So even if the first line of that function is, hey, if tracing's not on, get out of here. It still doing that that interpolation and figure out those variables and stuff so that's interesting yeah so you know maybe not a big deal if your application is not doing a lot if you're in a streaming world every little bit of performance matters right so you you kind of want to optimize you know maybe you don't want to go to the craziest micro optimizations, but this one kind of makes sense to me. All right. And the last one. So we all three love.net. I haven't done a lot of it in of late, but one of the things that I

really liked about it was the using statements on closables, right? Or I think it was closables is what they were called. Disposables, disposables and.NET. So Java has something similar. They have closables. Well, in Kotlin, I was trying to use a.use statement, which is similar to the using in.NET around a closable and it wouldn't work. Like it just, it kept telling me there was an error. Well, I dug into the class that I was trying to do the use on. And it turns out Java has another notion called auto-closable. And what it's supposed to do is Java is smart enough, or I guess these types of classes are smart enough that if you use the try with resources pattern in Java, when you're done with your try, it'll auto close that object for you, right? So I guess it's a feature of Java 7 and up. Well, the reason why it wouldn't work for

me is by default. And if you didn't know this about Kotlin, it was news to me. It targets Java six. So it doesn't know about nor will it do anything for you for any auto closables because those were available starting in seven. actually need to use a use on something that's an auto-closable class, you have to include the Kotlin std lib dash JD key seven or eight in order for that feature to be picked up. And then you could do a dot use on your auto-closable objects and it'll work just fine. So, uh, yeah, three, three little buried, interesting things that I ran across this past week. And I'll,

I'll get a link in there for that. Weird. Okay. I mean, cause you want it to just work, right? I mean, now I'm like thinking as you're describing,

I'm like, Oh gosh, do I have to go double check all my code where I was using a use to make sure that like, no, I have that throw an error. JD code where I was using a use to make sure that like, no, I have that throw an error. JD. Oh,

well, okay. Yeah. I thought you were saying that it just didn't know it through, like it was showing in, in the IDE that my code was broken and I'm like, no, it's not.

I'm sorry. Yeah. Yeah. Okay. Well, all right. Um, so in the spirit of the man page conversation that we had before,

I don't know if you guys saw this, but Tom and our Slack channel shared TLDR dot SH. So there is a project out there called TLDR pages where it's a community driven effort to simplify this so rather than going into like being super verbose with the pages like the man pages tend to be it's just examples of like oh so you can you could do like tldr tar for example is one of the things that they show on their on their page and and it will just say like, okay, here's an example where you create an archive and write it to a file. And so you see the, you know, they even call out like, um, you know how like you'd have the parameters like C,

for example. Right. And so it very specifically like brackets in the sentence, like create an archive, like C is bracketed to let you know, like, that's why you're pressing that parameter. C is to create, right? Things like that. So, and there's a whole slew of commands. Now it is, like I said, it is community driven. So like I was trying to get on some, cause I was like, Oh, I wonder if like Kafka cuddle is in there. Right. Like you know, but so not all of them are in there. Obviously the only downside I hope this, this effort takes off. Cause I do super like this. Uh,

just, you know, cut to the chase. Just give me the examples. The downside is that the current stable version that they recommend to use requires NPM. So the process to install this, like you have to install 8 billion dependencies,

uh, before you even get to use it. But once you do, it is super cool to use. So the first time it can be a little bit, or if you hit it with a command that it doesn't know, it'll be a little bit long because it'll be like, oh, let me go update my cache. And then you'll wait a bit for it. So that's where I'm hoping like, you know, we get something that maybe isn't NPM driven, that's just part of the operating system, kind of like how the man pages were. But I love the concept and the idea.

So that is my tip of the week. Thank you, Tom, for that. Now, Jay-Z has done this before to us. And I say he's done this to us because I said that on purpose. So you know how, Jay-Z, you like to share tips of the week that would be like, hey, here's something to code by. Here's music to code by. And there'll be some 27-hour- long YouTube channel of like non, you know, nonstop music and everything. Well, I just learned that the one Mr. Kirk Hammett, whose name you might be familiar with, released a solo album called Portals. And it's an instrumental album that it's only four tracks but each track is

like you know six seven minutes in the spirit of metallica they're long but um it's super cool and and and you know you don't think of it as like you know know, oh, well, it's going to be just another Metallica because it's really not. So I was reading some of the backstory on this and he is a huge collector of like horror movie type memorabilia. Right. zombie guitar and the mummy guitar which are uh you know the the designs on them are specific to those those old movies from like the 30s or whatever you know something like that um and so he has he has a lot of you know he has this huge collection so he was doing he wanted to show piece it or showcase it in a in like a museum kind of showing his collection and he wanted music for the the showing but he thought you know what would be super cool was like what if we were to like create some music that is kind of like horror movie type uh soundtrack kind of stuff but i don't want to

be something you've already heard so he went off and created and he got with, um, like one of the, uh, I think the producer or the conductor was like the, you know, they, they've done several, they've done two different symphony of Metallica albums. Right. So the most recent one, he worked with, uh, one of the guys from that effort to, um, uh, help create this thing. So, so my point being is being is like there's there's more instruments in it than it's going to sound more soundtrack ish than metallica ish but yet you definitely hear kirk hammett's uh well-known like you know style of of bluesy wah kind of soloing so it's super cool and i thought well

okay i mean in the spirit of things right because if we're going to talk about that then you know one would also appreciate in the similar type of setting um john petrucci who you might be aware of. His album, this one's a little bit older, but Terminal Velocity, I thought kind of falls in that same type of idea of, let me just throw something on in the background so I can listen. I want to listen to something in the background that's really cool as

I go about my coding day. Right? Oh, that's awesome. So I thought you'd appreciate that. Very cool. Yeah. So that's it. That's the show. And we're done.

No. Oh, I got it. I got to say all this stuff, huh? You got to say, Hey, we didn't do it at the top.

We got to do it at the bottom. Nobody's going to make it here anyway. So we should be good. Yeah. That's why we're done. Right. No, we got to do it anyways. It's's a must so you've already subscribed to us

probably maybe on spotify or itunes so good on you you did you did good and if you if you didn't and you're hearing this shame on you like yeah we want to ring the bell if we're going to walk down the street shame shame all right but if you have if you have only left us a single review now is your chance to correct that uh that mistake in your life you can head to www.codingblocks.net slash review where you can find some helpful links there yeah hey and while you're up there at codingblocks.net make sure you check out our show notes. We have all kinds of good ones there. Examples,

discussions, and more. Hey, and if you have not been to our Slack group, you should check out coding blocks.net slash Slack. And Hey, um, make sure to follow us on X.

I don't know. Is it x.com that I'm not even going there. It is. It is. Well, it'll redirect you to twitter.com Is it x.com? I'm not even going there. It is. It is x.com. Well, it'll redirect you to twitter.com. Oh, it did. Okay. But I have a feeling that it's not. I suspect that

we're going to see it eventually change the primary URL to x.com. I was a little nervous. I don't understand what's going on, but I'm not talking about the drug. I'm talking about the website, x.com. We've got an account there still I think at Coney Walks. I won yep give it to you and we're uh then that's where you can uh get sneak peeks on some of these tips like uh for example we control cha-cha key that's so amazing that might be my favorite

tip so far in 216 apps it's wait really dude i can't tell you how many times I've gotten really irritated trying to kill task. I just gave you like a horror genre of music. Like, I mean, it was good. It was good, but it wasn't,

it wasn't control. Cha-cha good. Yeah. And that, the ad read was just amazing. It was good. It was good. I oversold it.

Dang it. All right. Let me, hello, dear listener. Have you ever wanted to listen to horror music in the background as you work would you too like to be scared while you're doing your job check out the new album from kirk hammett portals no that's really good okay i was just making that

up on the fly yeah maybe i should do this for a little bit i'm just kidding oh you missed your window chad gp is pretty good at it too. Dang! Yeah, you could have. Dang.