Programming Throwdown - 174: Devops

Episode Date: June 10, 2024

Intro topic:  Social Media Auto Responder LLMNews/Links:Amazon releases Amazon Qhttps://press.aboutamazon.com/2024/4/aws-announces-general-availability-of-amazon-q-the-most-capable-generative-ai-powered-assistant-for-accelerating-software-development-and-leveraging-companies-internal-dataCheap RiscV “Super Cluster” from BitluniDIY 256-Core RISC-V super computerhttps://www.youtube.com/watch?v=-4d3PgEXhdYCH32V203Phi 3 Vision Releasedhttps://azure.microsoft.com/en-us/blog/new-models-added-to-the-phi-3-family-available-on-microsoft-azure/OllamaChatGPT 4ohttps://openai.com/index/hello-gpt-4o/Book of the ShowPatrick: MyFirstMillion Podcasthttps://www.mfmpod.com/Jason: A Path Towards Autonomous Machine Intelligencehttps://openreview.net/pdf?id=BZ5a1r-kVsfPatreon https://www.patreon.com/programmingthrowdown?ty=hTool of the ShowPatrick: Dave the Diverhttps://store.steampowered.com/app/1868140/DAVE_THE_DIVER/Jason: Turing Completehttps://store.steampowered.com/app/1444480/Turing_Complete/ Topic: DevOpsWhat is DevOpsDevOps vs SREWhy DevOps is importantEngineering time is expensiveOutages can hurt company metrics & reputationBuild & Testing InfrastructureBazel & Build/Test IdempotencyBuild/Test FarmsBuildBarnGithub ActionsJenkinsInfrastructure as codeTerraformBlue Green DeploymentContinuous Everything!Continuous IntegrationContinuous DeploymentHow to Measure DevOpsBuild TimesRelease cadenceBug tracking / round trip timesEngineer SurveysTime spent doing what you enjoyDevOps Horror Stories ★ Support this podcast on Patreon ★

programming throwdown episode 174 devops take it away jason hey Hey, everybody. So this episode, we're going to cover DevOps. For people who are new to the show, we're going to cover it somewhere in the middle. We'll talk about news and some other things first. That is the number one comment we get is, why did you take so long to talk about DevOps? You're going to learn a whole bunch of cool stuff today. If you're a brand new listener, welcome to the show. So I wanted to kick it off with a project that I built, which I'm pretty proud of. It's one of these things, you probably have this feeling, whether you're doing woodworking or stuff in the backyard or robotics or anything, you have a project and you say to yourself,

this is a coin toss, whether this is going to work or not and it's kind of exciting it's it's like um you know it's kind of like a rush because that that sort of randomness right um and then you know as it really is a coin toss half of them you know don't work and just decay into dust half the projects work really well this is what i feel like works pretty well um i created a linkedin uh social media bot um but it doesn't do what you think you know most of these bots go out and like harass people or maybe that's not the right word solicit people right mine is the opposite so mine is a defender it defends me against the other bots and so the way this works is it uses a large language model. So I'm using ChatGPT.

It has like in the cookies file, uses a headless browser with the right cookies set up. So it's logged into my LinkedIn account, goes on LinkedIn, accepts any connection requests, and then goes to my messages, looks at the unread messages, and I ask ChatGPT, is this a solicitation? Is this someone trying to sell me engineering services? And I tell ChatGPT, answer yes or no, followed by an explanation. And the next one is, is this somebody who is looking to work at my place of work? Um, and based on these answers, it sends automatic responses. And so I actually, in the. And I paste the chat sheet answer. And then I mute the thread and put it in the other category.

And this has worked amazing. So first off, after some testing, I kicked it off. I got banned three times from LinkedIn because not banned, yeah, not banned, but like throttled, you know, or like literally if, if from my desktop, I went to linkedin.com, I just got like a HTTP timeout or something. So like they throttled me, uh, completely, but, um, but eventually it went through my entire backlog of unread messages. And the other thing it does is if it doesn't catch this as a solicitation, it marks it as safe. And so I have a safe list.

And so when I was done, I looked at it because I had thousands of unread messages. And it was almost all spam, right? But then when I was done, I looked and there was all these people who were legitimate people who had interesting things to say who I now could see. It's like basically like I had a handful of needles, you know, like the haystack was burnt to the crisp and what I was left with is all the needles. And so some of the people I reached out to were surprised. One person was upset that it took me eight years to respond. But a lot of people is, is like,

I built a lot of connections with people who just, you know, serendipitously like, or just, you know, message me on LinkedIn and I just couldn't see it. So, so it was an amazing experience.

And so now what I did is created a cron job that runs on my Raspberry Pi, which runs the same process every hour. And for example, at 4 a.m. and at 9 a.m. today, it sent automatic messages and muted to people who are trying to sell me software services. So that's just today, this morning. So it's pretty awesome it's kind of weird but uh it's pretty cool i think the the idea is interesting i mean i guess the incentives in the system are a bit weird because linkedin doesn't really have an incentive

to stop the spam as long as if like it doesn't overwhelm and like mess up user retention so like some some people may like it's it's also a personal thing like some of the stuff you might consider spam other people might consider like if you're searching for a job some of those people reaching out may be like oh i want to reply to them like i'm i'm right right and so yeah i i mean the getting throttled it makes sense it's very hard to distinguish between someone doing like what you're doing and someone you know scraping a bunch of hacked linkedin accounts that you know trying to get everyone's personal data but yeah i mean i was definitely nervous when i got throttled that like i had

inadvertently got myself kicked off linkedin but but uh um yeah so i just to double down on that for friends i have friends at LinkedIn and you guys totally did the right thing by throttling me. I don't feel like that was the wrong... I'm glad that it didn't last forever. But oh yeah, you hit on something really important, which is there's this spam like, hey, I'm in Nigeria. I have a billion dollars. I just need you to send me your credit card. And then there's what you said, which is like, I think solicitations is the better word. And in fact, like even this chat GBT thing, the very first thing I ask is, is this a person looking to offer me a job? And if it is, then i mark them as safe so like clearly like there's

some solicitations i'm interested in and then others i'm not and and i don't expect like linkedin to mark those as spam right or have that nuance yeah so i guess like that's one of those i you know i guess you call it fine tuning or whatever like at the limit being able to describe your preferences in a maybe just a you know chain of consciousness style like way you're thinking about it and having it tuned to those preferences like i am interested in jobs that are legitimate but i'm not interested in job offers that are clearly just blasts and have none of my personal information in them like the person didn't bother to do research and you can kind of list off all the things you're interested in and

kind of build the yeah like you said like it's a you can do a lot of probably what you're doing with natural language filters or just like going in and doing basic stuff it just you wouldn't take the time it's not worth it right right like for me to build some nlp thing it would take me a long time versus this only took me maybe three four hours um and you know the the false uh alarm rate is not the best uh you know i actually um somebody texted me uh and i i had muted them on linkedin and they actually knew me from a meetup that i run here in austin and and uh they sent me a text so yeah there's definitely the false alarm rate is let's's say 5% or something, which is not great. But for an application like this,

I think it's fine. I'm not landing a space shuttle or anything like this. But it definitely, when you go and look at all the people messaging you and you auto messaging them and muting them, like every now and then I'll go and look at it. It makes me think of what a weird world we live in where someone has, you know,

a machine that messages me and my machine answers it. And, and it's all like natural language. Like, it looks like they messaged me, you know, manually. And it looks like I responded to them manually,

but it's all, it's all just fake. I think, I think that was on, Oh, I think it was Lex Friedman talking to Yann LeCun and, and, uh,

uh, LeCun was saying the same, something similar, which is we always think that, Oh, there's going to be some propaganda, uh, campaign that,

you know, somebody is going to use chat GPT to just be super convincing with fake news. He's like, what people miss is that if they can do that, you can too. And so a lot of our email and correspondence at some point in the future, likely it was astonishing that we effectively ran open ports, uh, and like, let everyone just send everything they want to us. Um, and instead we'll have secretaries that are trained on our behalf to like filter out that stuff you know and you'll just have your bots talking to their bots and some stuff we'll get through but most of it you know will be be sort of caught by intelligent filters on our side and so yeah they'll develop together yeah i mean scott galloway from the nyu business school had this

talk that i watched where he was talking about, um, Amazon having this unfair advantage, because if you search for just the word batteries on amazon.com, you got their batteries first and how that's, uh, you know, just a huge advantage for them. I don't think he was going down the, you know,

it's not fair route. I think he was more just saying what a tremendous advantage it is for them. And so you can imagine a future where when you search for something on Amazon, there's some browser plugin that re-ranks it that has your interests. And so now your ranking kind of fights with the server's ranking. Yeah, it's an arms race. I mean, it sort of sounds appealing to engage with social media in a way where it's like, I want something to go through and curate stuff on my behalf

that works for me and incentivized by me rather than the social media companies trying to get engagement or push just absolute crap to me or try to get me to stay on longer to find the nuggets, right? Like they're incentivized. Even if they 100% know the things I'm interested in, they're incentivized to spread them out at like a specific repetition

interval so that i spend more time see more ads and so they keep showing me other stuff that's maybe just close enough that i don't click away or whatever and so to have somebody go through and filter all that out it ruins all the advertising models of course but uh it does sound an interesting way to consume media yeah yeah totally um so yeah folks out there uh if you want uh connect with me on linkedin i will accept everybody and then you can say if you well i will literally accept well i won't accept it but my ai will accept it convince jace's ai bot and he'll share with you yeah if you if you message me on linkedin about your software development company in Eastern Europe and how they can do software cheaper than I can

or whatever it is, you will get blocked and it will be pretty funny. All right, on to... Oh, one last thing about that. The code is also totally available on GitHub. Just look me up on GitHub,ub aren't them a jason gauchies find my most recent repository um you can get all the code and run it yourself um all right on to news news uh my first news so when i was building this linkedin autoresponder i used amazon q which is uh you know one of these co-pilot type things patrick have you ever used github co-pilot or amazon q or any of these no i haven't i mean at my work it's it's

sort of a band and i uh as i've admitted many times on here they don't do as much coding as a hobby as i should so it's also banned in my work So this was my first opportunity to use it. You know, banned isn't the right word. It's more just like... No, yeah, exactly. It hasn't been considered, maybe. Oh, no. For us, it's like, yeah, it's...

You don't want to upload your code to somewhere, right? In order to get comments on it. Oh, I don't know if this one even does that, though. Does it do that or does it just run? Most of them do. Or at least have clauses that they could upload portions of yours. you know that's something i didn't even think about i just assumed that it was running on my desktop but you're probably right actually um okay at any rate so i tried i tried this out um it was awesome so i'll give you some examples like

really concrete examples of things i found really impressive. So and these are all going to be tiny. So this is part of it is, you know, the whole like build me an entire website chat GPT. I've been extremely skeptical of that for a whole bunch of different reasons. The biggest one is that projects are many modules that you want to have like some thematic consistency. Right. So I kind of went in pretty skeptical thinking this is not really going to help but one thing that it did was you

know i had a variable called finished and uh uh you know i had while while not finished and so then later on inside that while loop um i put uh um i put i think i put finished equals if i get this right oh no sorry i had while true and i had a variable called finished and then i typed if finished and then it auto completed you know colon new line break again like not not like a big deal but like it kind of figured out like okay the variable is called finished it's inside a wild true loop so if it's finished you want to just punch out of the loop right um there was another thing where um oh yeah so i was using this library called playwright to do the web automation and um when i went to run a function, it figured out that I should set this parameter, there was like a wait time parameter. And it looked further up and figured out, oh,

I'd set this wait time parameter before. And so I'm going to want to set it again. And so right when I put the open parentheses, it was like wait time equals two seconds or a constant whatever it was so it was impressive um i actually was really really surprised i i was very skeptical i was expecting to uninstall it right away um there were a couple of times where it wanted to auto generate like 30 lines of garbage like literally just garbage characters so it's not perfect by any means um but it was uh i would say it's a net productivity gain you know i was really surprised and you feed it your like context of your entire project in advance or it's doing this from like common available code that it knows about i have no idea have no idea. I mean, basically, I'll tell you what I did as a user.

I installed the extension. It makes you create an AWS builder account, which is separate from any other Amazon or AWS accounts you have. So I had to create a new account. And that was it. Under the hood, it could have uploaded my entire project to the internet. I really have no... I mean, it was an open source project anyways, so I'm not going to lose. But yeah, I really don't know what it was doing under the hood, but it was really clever.

I look forward to this. I mean, I think this would be one of those things where it used to be, I remember, that was a long time ago, Google used to sell search appliances. Oh, yeah, that's right. And for the same thing, you would put them in your infrastructure internal on your network at a company and it would index all of your internal websites and webpages. So you go to like an internal Google. Now a lot of that has been integrated in other tooling and other solutions. And so we don't really think about that anymore. But I think something similar will happen here where you'll have on premise boxes initially, because it's the only way to really guarantee I guess it's it's sort of safe for AWS instances, maybe. And you'll upload company, you know, your repositories and have this stuff crunch on it. Yeah, yeah, that makes a ton of sense. I know there's a bunch of open source versions of this

that you can run locally or run in your own cloud. So I know the technology is there. It's just a matter of time. Very cool. Mine's on a pretty opposite end of the spectrum, I guess. And that is, I've seen a series of these. So I just want to highlight, I picked one to focus on, but there's actually two, which is there's a series of very cheap RIS v or risk five i don't think i don't

remember yeah risk five okay that's what i was gonna say uh the risk five uh processors are starting to kind of roll out and also i saw like commodity prices so i think they're like 10 cents um roughly for one of these processors you have any i don't have any yet it's on my yes i have them in my cart on aliex, but they're kind of expensive there. So I need to find another place to buy them from. And yeah, I found them very expensive as well. I was looking at the Mars something and it was like 200 bucks.

I was like, I can get a Raspberry Pi 5 for that. So, okay. So maybe these people are out to help us. So a YouTuber by the name of BitLuni, B-I-T-L-U-N-I, if you've never seen him before, lots of interesting computer related projects there. He built a super cluster of 256 of these with some interesting discussions, if it's not something you've ever considered before. But if you want to have 256 processors talking to each other, how do you

handle that as like putting them on a bus? And what is the communications matrix look like? And how do you organize it and think about it? And then, you know, how do you tell if they're all working or not? And he's not the, you know, of course, other people have built, you know, very high core count, even production processors before but at a hobbyist level it's kind of nuts uh you would never have spent whatever it is you know five dollars for a processor and then 256 of them but at 10 cents each and pcb prices have come down to sort of make your own boards and so it's just interesting time to kind of consider consider some of these and the nice thing about these which i think

there's a pair of them that are pretty similar ch32v i think one is 003 and the other is 203 so ch32v203 these are the chips um but often what happens is they these chips are very very cheap and if you get help from the company and you need to put them in like your hair dryer to display you know to control the temperature output or something you could buy one of these but if you get help from the company and you need to put them in like your hair dryer to display you know to control the temperature output or something you could buy one of these but if you want to do hobbyist stuff with it getting an sdk getting documentation is pretty hard so you need a sort of critical mass of people to get you know good english translations and sdk and the very very popular um i guess example of this is if you hear people talking about, oh, hey, what?

I just want the S, no, not the STM32. What are the little Wi-Fi chips? You were using them, Jason. Oh, why is my? The Pico, Raspberry Pi Pico. Ah, no, now my brain failed me. Oh, this is a bad time. Oh, the ESP32.

Yeah, thank you. Thank you. I was going to say STM32, but ESP32. Those are two different things. The ESP32 is similar. It was like a very cheap commodity thing that was like going in light bulbs and it wasn't useful. And then hobbyists sort of realized, wait, this is actually, you know, we can get good documentation for it. And now there's a huge, you know, community of people developing things for ESP32 and other chips. And I don't know for the company if it made an impact or not. But for hobbyists, it's certainly cool to get access to these really

cheap things from this Espressif, I believe is that company. So now we're starting to see the same thing, debug tools, compiler tool chains for these RISC-V processors. And they're ridiculously cheap, in part because they don't have to pay a licensing fee for the risk five architecture itself and so you can get them even cheaper not that everyone always pays licensing fees but for sure now like they they definitely can't and then um in the show notes i don't have a great way to do it but if you search a powering a nixie tube from a risk 5 uh chip you'll find there is a link here to a youtuber who i hadn't run across before um but they built a um like a boost buck boost controller from a microprocessor so actually taking usb sort of five volt power and like going

i think it was 200 volts to power a nixie tube which is a little like vacuum tube except it displays numerals instead of like oh wow a transistor i think it's oh i did a cnl ohr look in the show notes uh or search powering a nixie tube from usb with a 10 cent risk 5 processor and so they i guess we're doing something at a not coordinating with the other person bit loony but they both end up having something they have a bunch of cool stuff on their channel as well. So worth checking out. So adding this definitely to my list of like, things that would be cool to play with. And as Jason has told us on the show before, you know, there is a risk of letting out the magic smoke. So spending, you know, 10 cents on a processor makes letting out the magic smoke much less painful yeah yeah totally um

yeah i feel like just personally at the level where like i have to put the processor somewhere and put the ram in another spot and the wi-fi chip in another spot and hook them all together that's like one level below where i'm interested like like the uh that's why i was looking at the this mars thing. I can't remember the exact one, but it was basically a Raspberry Pi where you have the Wi-Fi, you have the Bluetooth, you have an Ethernet port, HDMI port, all that stuff,

but it's RISC-V. And for some reason, those are still really expensive, but I think it might be just that they're not mass-producing them yet. Yeah, I think they'll come. I think the dev boards, I mean, I'm seeing dev boards in the sub $5 range for some of these CH32s. I don't know what the proper abbreviation is for these, but they don't have Wi-Fi and

stuff like you're saying just yet. Got it. Yeah, that makes sense. All right. My next news story is Fi3 Vision release. So a bit of background here. Microsoft has been pushing, well, a lot of things,

but among them, these SML, or no, wait, SLM, Small Language Model. And so the idea is, you know, this is a model that I actually have one running right now on my phone. And it's pretty good, you know, this is a model that I actually have one running right now on my phone. And it's pretty good. You know, it runs in real time on my Samsung. It's not even really the latest model or anything. So small model, you know, it will give you some weird

answers for sure. You know, it definitely doesn't have the depth of these other models. But for general queries, it works pretty well. I had something recently. Oh, yeah. So there was some issue with Verizon. We ended up having to call Verizon, our cell phone company, and get it sorted out. But for whatever reason, my entire family lost internet, which was actually kind of terrifying. When you're on vacation we're on vacation and and and we just we still could make phone calls and texts and everything but we lost the actual

internet you know uh you know mobile data um and so i was you know downloading maps when i was on wi-fi and all those is a mess but but one of the things i did is i was asking this phi 3 because it was local on my phone just for some general queries and it was giving me answers. So it's amazing. And so now they released PHY3 Vision, which is an actual like multimodal model that you can run locally pretty easily. So people have done a lot of really interesting things with this. You can actually, um, I have a friend who took a photo of his driver's license and asked five, three vision, um, Hey, pull out the birthday and the driver's license number and the person's name

out of this photo. Um, you know, and it did all of that and stole his identity so it's uh i'm just kidding um but no he ran it locally and so you know he's totally safe he knew that it wouldn't uh hit the cloud or anything like that with his driver's license um you know he gave it a few other pdfs and asked it to uh you know extract information and he said it was amazing. He said everything came out flawless. So I think this is just, I think, really something to pay attention to. There are better models like Lava, which is a multimodal model. But these require GPUs with 16 gigs RAM or maybe more than that. and so they're heavier models um it's really interesting to see like for an average person with an average desktop laptop

or even a phone you know what are what are we capable of doing here um this is uh really taking that to the next level so you can check it out it's it's pretty easy to download. And there's something, I'll put it in the show notes called O-Lama, where it will basically hold your hand and you don't need to be even an engineer or a coder to use these models. It gives you a like chat GPT like web interface, but for open source models.

So this is a super low barrier of entry and i think there's a ton of potential here so if you're out there you know college high school just getting started um you know check this out i think there's some real powerful stuff you could build on this very nice i haven't i do have a couple apps that that will run open source i think hugging face there's an app i don't know if it's an official app or not, but lets you from your phone hit those. Those aren't local though, but yeah, it is kind of cool to try the various ones

and see various ways you do or don't, that they can kind of fall over. But the multimodal stuff is cool. To dovetail it right into my next one, talking about multimodal is OpenAI. I mean, news all the time these days, controversies and new releases, but just focusing on a new release, release chat GPT for O, which the O being Omni and sort of their take on this multi multimodal.

So doing being able to use pictures or voice, the demos are really cool. Sometimes a bit hard to recreate the demos uh if you try it yourself but in general pretty exciting to say like you know hey i'm sitting here with someone who speaks russian i only speak english like i'm going to speak english and i want you to translate it to russian they speak russian you know just describing you're like oh i could do that with the translator app it's like yeah but now you don't need to like you know for arbitrary configurations you know it can kind of understand but also some of the behind the scenes things that they're sort of saying that you know running this new version

even though it's improved it's just like orders of magnitude cheaper they're figuring out not really i mean you can kind of read the rumors about next version of you know chat gpt5 but it's kind of cool to see them get more and more efficient at running their current generation stuff sort of almost faster than you would expect. And it's just interesting to see how quickly things are moving

and how the pace goes for training and running a lot of these models. Yeah, definitely. I'm a really big fan. I think, you know, they started, OpenAI started by doing reinforcement learning, which is obviously near and dear to my heart. I don't think, I don't know if they do much of that anymore, but still, I'm or stealing data from the internet. They've stolen all of my code, but that's okay because I stole it from Stack Overflow. So I copy pasted it first.

So how can I hold them responsible? So I think, well, I mean, that's a whole rabbit hole around data. And if you scrape Reddit, are you like you know trouble or whatever but but either way i mean i think it's amazing yeah this this uh i think the 40 the chat gbt 40 stands for omni because of the multimodal part um and uh i think there's so many interesting ways you could take this i mean you know one thing that just came to mind is, you know, you could run, Oh, circling back to what some of you said earlier, uh, the app that lets you actually run the models on your phone is called MLC chat. And, uh, you can get that. I think it's on iOS and Android. Um, but, uh, but yeah, imagine like you, you take a picture of like, uh, even just a restaurant receipt and it keeps track of what things you've eaten at restaurants.

I think there's a whole bunch of cool stuff you could do with this. And we'll have to see where people take it. All right. Time for book of the show. My book of the show is not a book. Spoiler alert. My book of the show is not a book. Spoiler alert, my tool of the show is also not a tool. Book of the show is a podcast. This is a bit of a different podcast.

And I don't want to give like a ton of disclaimers. I've been enjoying listening to it. It's the My First Million podcast. This is like two business folks who are kind of like doing marketing, but they have interviews with people who have sort of done startups or sold companies. And they talk a little bit about details about, you know, kind of exact dollars they were earning and how marketing goes. They were talking to someone recently who had a blog about gardening and then started selling products and turn it into what I find it interesting is, it makes you think a lot about, you know, stuff that's pretty different, but adjacent to the

stuff we do. I think a lot of software engineers always have this question about entrepreneurial pursuits, and like what the various trade offs, these folks are out talking to, you know, people who have done massive things, people who've have done small things and also just their observations it's a very casual podcast um a bit long-winded they also don't necessarily get to the topic of the podcast until late into the podcast um but uh you know they they release twice a week and it's it's a very casual listen i've been enjoying it i don't know that i'll listen to for you know always and ongoing but it's been good to sort of uh listen to kind, you know, always and ongoing, but it's been good to sort of listen to kind of the sales aspect, marketing, the kind of parts of the business that I don't normally focus on, right? I'm always thinking about solving hard problems and machine learning models or whatever, right?

Like those things are much more up my avenue. This is adjacent and people doing often software businesses, but talking about motivations and things you know related to personnel and you know how to relate to people and anyways personally i've just been finding it a nice orthogonal set of views i guess to to what i might normally pay attention to yeah that's awesome um yeah i'll have to check this out jason freed that name sounds familiar is he the base camp person or is that somebody else yeah he was one of the ones uh him and dhh or whatever yeah okay there we go yeah very cool yeah i'm definitely going to check that out it looks like fun um all right so totally on the other end of the spectrum my book is the one of the geekiest books i think we've had on the

show it's this research paper from yan lacoon um that he wrote in 2022 called a path towards autonomous machine intelligence i have a link to the pdf um in in the show notes um It's not a, it's not a, called a, an approved paper. It wasn't submitted to any conferences. It's just an open paper that's written on open review. And you can kind of tell, you know, there's some languages,

you know, errors, grammatical errors, or some hand wavy stuff. There aren't really any, actually, I think there's literally no results section. But sometimes these are the best papers you know it's just like raw thoughts of of of folks and so this

this paper is really interesting it tries to tie together um uh you know these these uh really large latent models right so so large language models, and I think we're starting to use foundational models because they are multimodal, but you know, these foundational models are, you know, forward models, which means they look ahead. So it says, you know, given I just wrote this part of a paragraph, what's the next word I should write, or what's the next handful of words I should write. Um, and so you could extend that to really anything you could say well uh you know given that i made this move in chess what's the next move i should make or given that i turn the steering wheel this way in this in this you know uh car you know what should i do next with the car um so so there's sort of like a

theory of everything kind of thing going on here where, you know, the idea is you build this massive world model that, that tries to predict the future. And then you do a variety of different things like sampling and some gradient based, uh, search and different methods to search through potential futures so i mean maybe taking a step back so you have the the world you're in imagine you are it's a good problem we can use for this um like uh just let's take chess for example what about okay let's take chess where instead of you know you getting a chess board the way a computer would expect it, you're getting pictures of a chess board. And so you have to play chess from looking at pictures of a chess board. There's a lot of things that really don't matter, right? Like if the pieces start casting a shadow in the opposite direction because there's a new light source, that's not relevant

to the problem you're trying to solve, right? And so if you look at ChatGBT, ChatGBT is taking text and trying to produce more text. So it really has to understand everything about text. But in this case, you're not really taking in a picture of a chessboard and trying to create a picture of the chessboard with your move on it. You're trying to take in a chessboard and make the move. Right. And so there's a lot of confounding information there, like shadows, for example. So so the idea here is you you create like an embedding or like a low level projection of the image.

And then you try to predict the low level prediction of the image with your move in it. So imagine I say, here's a picture of a chessboard. And I want you to give me a picture of the chessboard where the pawn has been moved up a square. And so but instead of literally rendering that photo, which, you know, requires a lot of wasted time, it creates sort of this low dimensional presentation and then a low directional representation of the board with the pawn in the new spot. And then what you could do is you could physically move that pawn, take a picture, and then embed that picture and see if it worked. So did the embedding of the future match the embedding of the present

plus this function that's supposed to take it into the future? And one thing that you'll quickly notice is'll quickly notice is like you can hack this or you could you could um exploit this you could say oh well my embedding is just the zero vector and all actions just turn the zero vector into the zero vector and so so i embed to the zero vector i move the pawn now i have the zero vector i embed the future which is also the zero vector. I move the pawn. Now I have the zero vector. I embed the future, which is also the zero vector. And sure enough, the zero vector equals a zero vector. And it could seem like my embedding is amazing, but actually it's not doing anything useful, right? And the way they get around that, which I thought was super, super clever, was I'm going to see if I can explain this easily.

They try to maximize the information of the embedding. And the way they do it is, you know, if you have a picture, right? It's, and we've talked about this on the show in the past, like it's little RGB values for each pixel, right? And so it's like a ton of these like RGB values. Right. Um,

and then when you crush it down into this embedding, now it's like a small number of, of numbers or dimensions, but it's trying to like accomplish some goal. And the goal in his case is to do this forward thing we talked about but also he wants the variance of the parameters to be as high as possible so basically um trying to make fancy way of saying trying to make the numbers vary a lot from one picture of a chessboard to another. So if you take a bunch of pictures of chessboards and they all collapse to the same embedding, you could assume that it's not doing a good job.

Now, it's possible that nothing has changed, but it's also possible that you're not doing a good job of capturing information. So yeah, that's just part of the paper. I mean, this is a 50-page report. So there's a lot of content here. The thing I like about this is you don't really need to know anything beyond just algebra and some understanding of differential equations to grasp this. It doesn't refer to a million other papers that you then have to go read. So I think it's one of these things that folks who know Diffie Q, or even if you don't, you could read this paper

and get a lot out of it. It's a good kind of starter paper, but there's a lot of content too. And this is part of his thing. I mean, I think he's talked about this, but the contention that the current machine learning stuff, like the LLMs, can't really do planning. It can't sort of figure out how to do stuff.

It can describe the text of a plan, but it itself doesn't do planning because it doesn't sort of move through the space like you're talking about. And this is related to that. Yeah yeah that's right yep yep so you know he actually famously put on twitter um like uh if you're in college right now if you're getting a phd right now don't bother with llms because like the companies are all over it he's like do the thing after llms um which i think is absolutely correct.

Like this LinkedIn thing we talked about at the beginning of the show, like take an LLM and these things in favor of some of these other approaches that he talks about in the paper. Very nice. Yeah, if you have been paying attention to, I guess, X, is that recently there was shade being thrown between Elon Musk and Yk and and yann lecun about about being being scientists and writing papers and anyways feel free to go read that that thread if you want some drama yeah exactly all right it's time for tool of the show as uh foreshadowed spoiled mine is not a tool to show but it is a game um this game is available i think on actually various consoles um xbox playstation but also um i've been playing on my Steam Deck and that's Dave the Diver. It's just like one of those things if you tell someone you're playing a game where you're like a slightly out of shape spear diving fisherman who goes and collects fish and then runs a sushi restaurant by night.

It would just be like, what? But it actually works. It's kind of fun. And it's a, I guess you would call that, I'm not, I'm gonna misuse this from anyways. It's a bit of a rogue-like game in that you sort of have the cycle of going down and doing the fishing repeatedly.

You're not meant to sort of win it. You're just progressing. And you can do better or worse at going down and catching the fish. But then you go through the cycle of serving the sushi and earning money and upgrading your equipment and then you can go again and you can learn you know get equipment that helps you dive deeper and have better weapons and fend off you know things trying to eat you in the sea um and so the game sort of sort of progresses i've not beaten it presuming that there is a an end game i've not gotten there

yet um still and there is a story along with it that i won't i won't spoil um but it's just sort of like a fun never you don't have to really play for a long time the controls are kind of cleverly done for how to sort of line up your spear and and shoot your spear out if you're not into i guess fish cruelty definitely stay away i mean it does involve eating sushi so it has to be procured somehow um but uh you know definitely like a fun game well done like the art is kind of a pixel art style so uh it's it's kind of fun that way um but the gameplay just is cool and you know if you only have like five minutes to play you can go on one of the dives or whatever

you can play for longer but you don't have to like sit down and be like okay i'm putting 90 minutes into this um and so i've been enjoying that what happens when you die like how does he make it to the next day if a fish literally eats him so i i mean i think it just sort of hand waves over that part so if you're collecting up fish and there's like a limit to how many you can carry like a weight that you die like you lose everything you're carrying but i i i presume he has some sort i don't know that i remember exactly how they hand wave away the fact that you return to the boat and you can try again got it okay that's cool i'll have to check i've heard good things about that game so i'll have to check it out yeah i was very skeptical but it works very cool all right my my uh tool of the show is also a game, but very, very relevant.

It's called Turing Complete. And I learned a ton about low-level engineering, like electrical engineering, I guess, or computer engineering from this game. So literally, the first level they give you, and Patrick, you're going to know this, they basically give you one of the gates, and you have to build all the others i think nand no right okay yeah they give you a nand gate and they're like make a not gate they're like make an or gate make an and gate and

um and so you build all the gates from a nand gate and then as you build things um they end up on your toolbar so because it's very modular right so it's like okay now i don't need to make the not gate as soon as i build it i just have it now um and then you get all the way to where you like build registers and you build like an alu and then you end up like up creating your own assembly code. So you learn, in the beginning, they give you assembly code without arguments. So you literally have to be like, if you're moving memory from this register to that register, you need to have a named instruction for all possible pairs of registers. And then it gives you like assembly code with arguments where you could just have like a move operator or like multi-symbol,

I guess, assembly code. And then in the end, you end up like creating things that shoot down asteroids and solving mazes and doing all sorts of crazy stuff. You play a space invaders. You build an ai to play space invaders um and like you can actually then like drill down like you could build the ai to play space invaders and if you're inclined drill all the way down to like an individual nand gate in this computer you built um it is unbelievably satisfying it actually taught

me a lot because as people know um like computer engineering is not my background and so um i learned a lot by doing it um it gets really really hard um you know i was able to get through the whole game but but um i i think it's i would say the mid to end game is pretty much inaccessible for people without an engineering degree. I don't know if I'm just like taking my skills too highly there, but I feel like if I feel like a person is going to have a really hard time with the last few levels of this game. But that's OK, because you've experienced so much of the content already. And maybe it would encourage people to go and get like a four-year degree or something. But there's a lot of complexity, but a ton of fun. The UI is really well done, the way you place the circuits.

And you can color code the traces and everything. Highly recommend it. Definitely worth the... I don't think it's even that expensive definitely worth the 20 bucks i think we mentioned that one in passing before but this is like definitely a good review review of it i've not tried it so now now you make me like want to go do it but i know i'll also hate myself having done that bottoms up thing a few times in a few different contexts uh the pain and the suffering uh and knowing that like yes you can just look up how a flip-flop works if you already know what a flip-flop is um but you're

supposed to figure it out again from scratch yeah i i did my that was another thing is i did my best to not cheat at all um there was something um i think it was like an x or gate or something there one of these gates were like i felt like there's an incredibly simple solution but i had an incredibly convoluted solution because i didn't want to cheat all right on to the topic devops or developer operations uh patrick what is devops all right this is this is a fair question uh so words are hard uh but but you know devops i think is in the set of responsibilities that take you from compiling your code testing your code deploying your code the sort of life cycle of your code into an application from development and into production

and the processes that go around that. Yeah, that's right. I feel the same way. I've heard of site reliability engineer. What is that? Is that the same as DevOps? Are they different? So this is a fair question. I had to double check just to make sure my thinking around this was good, because sometimes you end up with a experiential definition versus a, you know, actual definition. So I think SRE is a term that started out of Google, but has gained acceptance. And a site reliability engineer is responsible for basically keeping production up and working. And so the two do have, in some ways, overlap, but SREs are responsible for the production environment. So making sure that things, databases are staying performant, websites are staying up, monitoring traffic, alerting if something goes down,

and either restarting it, fixing it, whatever themselves, or involving engineers or DevOps and sort of that. So DevOps can definitely make an SRE's job a lot harder if they're not doing a good job helping to check along it. But SREs are really the people responsible for keeping it up and going in the end. Yep. Yep. Yep. That's right. Yeah. And DevOps is extremely, extremely important. You know, outages, I mean, depending on where you work, outages may or may not make the news. They definitely have a huge impact on your company. If people feel like they can't trust your company to be up all the time, that's going to create problems i mean as i said when we were on vacation and we lost mobile data had like a huge impact and one of the things we thought is maybe we shouldn't

have the entire family on one plan um i mean we're we're still going to do that because it'd be a huge hassle otherwise but it made us think about that right so like that so i think it can have a huge impact on your business um your company's reputation. Also, your engineering time, right? So if DevOps is done correctly, then your engineers are getting very quick feedback. Anyone who's built something knows, you ever go back and look at your code from a year ago? You're like, you you know what idiot wrote this like this guy needs to be fired and then you look at the top of the file and it's your name on it or

something right no never done that never happened so um so you know that uh you know so if you if you have a problem that you don't find for three months um you know you've lost context you move on to something else um If you have a problem and you find out in five minutes, well, that's a totally different story and you can rectify that pretty quickly. At worst case, you can roll back. Even if you don't know what's causing it, you can at least roll back five minutes and just say, okay, let's take our time and figure this out. If it's a problem that's maybe been there for three months and people are just starting to hit it now, that can be extremely stressful for your engineering team and for yourself.

So DevOps is meant to address all of these issues. And it's absolutely critical if you're doing any type of software that goes to other people. Yeah, I mean, Jason mentioned outages potentially making the news. But I mean, it can be an absurd amount of money if you think of something like Amazon going down. And everybody's role is to make sure that doesn't happen to some extent. But even in, like you mentioned, in smaller companies, if, if, you know, you're trying to, if you can't figure out why something is acting different today than it did yesterday or the day before, and like, you're doing a lot of running around because of some of the techniques that we're going to talk about,

that, you know, DevOps, not just being a specific person or group of people, but really everyone's responsibility to do some of the stuff that's involved in this and really making sure that you can quickly get to the bottom of what's going on. Nothing worse than banging your head. If you ever encountered that, my code, I haven't changed anything, but it's doing something completely different, or I changed this one line and everything is different. It's very, very frustrating. And that can happen in more than just your compiling if you're not careful. Yep. Yep. Yep. That's right. So yeah, jumping into it, maybe we'll just start. So, you know, the general DevOps cycle is, you know, build, test, release. And so we'll structure this

episode in the same way. So looking at the very first step there, building, you know, building software that you intend to give to someone else or to another machine is actually really hard. You know, we've talked about this in the past, but you have a bunch of libraries on your machine. So stepping back a bit here, you have your operating system, which has a bunch of core C libraries. Then you have a whole bunch of packages that you've installed, OS packages. And then some of those packages are themselves package managers, like Python's pip or nM. So you have all of that context. And so if you build a piece of software, it could be using any or all of that context. And then you give that software to somebody else or to a machine in the cloud, or to a customer. And and that software doesn't work, right, Or it doesn't work the way you intend. And so this is called, I always have a hard time saying this name, but it's idempotency, idempotency.

And what that means is something is idempotent if it has no external dependencies, right? And so imagine like a function that doesn't use any global variables or system calls or anything yeah that function has no external dependencies every time you call two plus two you get four every single time no matter what the context is or time of day anything like that um and so you want your build system to be like that um the challenge is it's it becomes kind of uh like getting a hundred percent test coverage where it just becomes exceedingly difficult to to sort of get get that much uh idempotency so for example um you know one really popular tool for building software, you know, at least in C, C++, and Python is Bazel. So Bazel,

what you can actually do with Bazel is you can actually have Bazel build its own, you know, compiler and have Bazel build its own, you know, C libraries and all of that. And then you have, when it builds other software, it's using those compilers and C libraries that it built. And so that's kind of the ultimate. I mean, I think the ultimate idempotent solution is Bazel runs inside of a Docker container that has an empty Linux image. And inside that Docker container that has like an empty Linux image.

And inside that Docker container, Bazel goes and builds everything from the ground up. Now, again, that's extremely expensive. There's a bunch of challenges there. It's hard to actually execute that. And so for most people, they don't need to go that far. But just know just using basil at all is a good first step to um to get idempotency yeah i've heard this called hermetic builds as well i'm not sure if there's a slight difference between the two but um i think like you mentioned there's the first

line of things which is you know hey we pin all of the libraries to a known version everyone's on the same os version and you can get 99 to the benefits that you know if things are reproducible across everyone's but it still may be uh i'm trying to think of like some specific examples but depending on what you have in the build as well not just your build infrastructure if you go to the level of i want the you know hash of my output binaries to be identical no matter whose machine I build it on. It is literally a bit identical regardless. Then you can end up with all sorts of interesting things that are getting picked up or used or the ordering of things. If you really, really go to an extreme that everybody's machine builds a bit identical

binary, it can take a lot of running things down and making sure that things are exactly as you think they are. But the big unlock of having, you know, as Jason mentioned, even like building it from source is one, but at least knowing the version of something that everyone is on is definitely super useful and what everybody's depending on because if you do none of it at the opposite, it's not just that your builds aren't reproducible, but I've had this on Teams. Someone sends you something and you can't get it to compile

because there's 15 dependencies that aren't called out anywhere because they just got picked up from their computer. And so you need to one by one resolve, you know, brew install this, pip install that. And you're just going at it one by one by one, you know, brew, install this pip, install that. And you're just going at it one by one by one, you know, trying to find the set of packages to install. And it's super frustrating. Yep. Yep. Yeah, exactly. Um, so, you know, getting it completely hermetic or idempotent

is really difficult. Um, one thing that's a nice middle ground, which I think is the most popular approach, is to have a build server which matches your production hardware. So imagine if you are deploying a website to an AWS EC2 instance, you probably don't want to build that website on your desktop and copy it over probably want to do at least that final build on another ec2 instance with exactly the same specs and and heart and software setup um that way you know if there is a problem let's say you're running the gcc 10 and um or like libc you know 2.13 and they're running 2.6 on the cloud, then what will happen is the build will work fine on your machine. But then when you try to do that final build before pushing it to customers, that final build would happen on this EC2 instance, and it would throw some error, and at least you would catch it before it made it to the production servers um and so there's a bunch of tools for this uh you know

this is called a build farm or a test farm um there's a bunch of tools for this i've used build barn in the past i've also used jenkins um there's uh there's github now has github actions um and so these these do uh um a lot of different things you can you can run commands off triggers um but at least in the case of build bar and it also is just transparently hooked up to basil so the way it works is you configure your basil um basil actually has an API for submitting jobs and getting results back. And so you can set up Bazel to point to a build barn instance and say, hey, you can build the code locally on my machine, but you could also build it on this build barn instance.

You could also say, only build it remotely. Don't do anything on my machine. And then you could have a lot more control over the machines that are doing the building. And then when you go to actually build the release candidate, maybe you do that every day or once a month or what have you. You could use something like Jenkins, which would spin up a machine in the cloud, do your build on that clean machine, and then tear it down. And so getting a good handle on these tools is super important. I noticed we did skip over one thing that we probably just took for granted, which is having code in source control and every commit being checked that it builds so we're talking about building it

but all of this starts from a basis that you know you're not compiling code that's you know a thing or that that you know main main branch is currently broken or whatever you know you're on is not buildable and that yeah that everybody knows that you just had to do these six things and then, you know, and then it builds has been on that, then been on that before. So you aren't allowed to merge to main unless that's proven that that branch afterwards will be good. Yep. Yep. Yep. That's a good point. Yeah. I think, uh, um, yeah, I mean, this is, this is, uh, there's a lot of complexity there. But yeah, I think on the surface level, you should have a set of smoke tests

and say, look, if this PR breaks one of these tests, or if the Bazel test command isn't clean, then we can't merge a PR, or you need like a special override flag or whatever. So I think, you know, there's sort of a meta point here, which is,

you know, we've talked a lot about setting up, you know, build farms and testing farms and, and Jenkins and all of that. But then there's those meta question of like, how do you set that up? You know,

how do you guarantee that your jenkins cluster you know matches your production cluster and if somebody let's say upgrades the production cluster to ubuntu 22 but the build cluster they didn't update the build cluster to ubuntu 22 how do you keep those errors from happening um and the answer is this concept called infrastructure as code. I think it's also the acronym is IAC or IAAC. They take the A in infrastructure. And so what you want to do here is you don't actually want, in a corporate setting or a business setting,

you don't actually want to go into the Amazon web interface and click through and create a new VM and assign it an Ethernet adapter. Doing these things is great as a hobbyist or if you're doing a research project or a one-off thing. But for production, you don't want to be using the Amazon or Google Cloud, any of these UIs and spinning things up because you might just click an option that someone else didn't thinking that's the right way to go. And then it becomes really hard to debug that.

Or what if you need to spin up like a ninth machine, but Bob span up the first eight machines and Bob isn't there. And so everyone's waiting for bob right so so um so what the most popular tool for this is or language for this is terraform um terraform is a tool and a language i believe uh there is a you know dot tf files that have their own domain specific language there's There's some actually issue, some drama around this, where I guess there's like an open-source fork and HashiCorp, which runs the closed fork. There's some issues there.

But I think from what I've been reading, Terraform isn't going anywhere, and it's not going to get completely proprietorized or anything like that. So I'd highly recommend Terraform. The way it works is it's really clever. Um, so Terraform looks at your cloud, uh, whenever Terraform, okay, let me take a step back. So Terraform is this script that procedurally like lists, uh, a bunch of things that should exist. Like you should have this VM,

it should have this much memory, it should have this, you know, Ubuntu image on it, and then it should also be able to tunnel through to this other VM, et cetera, et cetera. And so the first time you run this, you have an empty cloud. And so Terraform goes through the list and creates all of those pieces of software

infrastructure right um but when it creates them it puts special tags and then it creates this this cache file which you put into your source control and that keeps a history so terraform hey, I Terraform, you know, created this VM from this git commit of Terraform code. And so then all you have to do is change that Terraform file and say, oh, actually, it has 32 gigs of RAM. And then the next time you run TF apply Terraform again, because it's tagged everything, it knows what it created versus what you know patrick created and so it looks at all the things it created it looks at the new terraform file and it realizes what it needs to change and so it will actually just add ram to a vm or rename it

or something like that instead of just deleting everything and starting over. You can, as you would expect, you can do weird things that make Terraform very confused. And then it's like, I don't really know how to fix this. But if you're not trying, you'll almost never have a situation like that. In practice, Terraform pretty easily knows how to make that leap um and so this will help guarantee that everything's on the same page so imagine you know you'd have one place in Terraform that says this is my production Linux image and someone just changes that one line to like Ubuntu 22 and under the hood Terraform might like reformat a thousand machines but it's going

to do it all in one step and uh and that's going to keep uh inconsistencies popping up i think for you know not only being able to do that but also having as code checked in and a history of of the changes is of course uh you know definitely useful so having something that you can have other people approve before you just apply because i think it's another instance everyone's heard about is someone goes in and just makes a change and nobody knew or wasn't documented and then everything's broken and no one knows why so yep uh another popular thing to do is what's called blue green deployment have you heard this term is that ab deployment i yeah pretty much so blue green just means you have two copies of your

infrastructure and so for example like the upgrading ubuntu example so you want to upgrade all the machines at once um and just have one clean atomic operation. The problem is, of course, it takes time to upgrade a thousand machines. And so you don't want your website to be down while you're doing that. And so Blue Green is just a fancy way of saying have two copies. And so you upgrade the first copy. And while you're doing that, all the traffic goes to the second copy. And then you switch the traffic copy. And while you're doing that, all the traffic goes to the second copy and then you

switch the traffic over. And so it's kind of like if you've ever done a ropes course, you know how like you're you have two harnesses and so you disconnect one harness from a rope to connect it to another rope. But if while you're connecting it to other rope, you fall, it's OK because you still have the second harness. And one time I went to a place where I don't know how they did this, but some kind of mechanical or magnetic thing where basically if if if. Yeah, I think it was it was a circuit like if you hadn't completed the circuit by locking one of the locks, then the other one couldn't be open so like you you physically couldn't unhitch both of the harnesses at the same time and you know do something dangerous um and so blue green is kind of like that where um you know while you're making a change to blue you can't send any traffic to it and then it flips i only, you know, this being singular stuff,

but I think part of what we're talking about is leading to making sure that all of these things are repeatable, and that they happen frequently. So I, you know, saying, Oh, we haven't deployed a new one of these or checked on anything in a month or three months or four months or six months, you build up a lot of changes, assuming your code's going under, you know, active deployment or packages that you're using are getting upgraded. And, you know, security vulnerabilities, we've not talked about, but monitoring and updating because of those. And so making sure that you're continuously integrating your code into the system, and then, you know, deploying those things out at some cadence. I mean, it's not in the mathematical sense continuous, but just doing it at a constant repetitive interval prevents there from being

large, gigantic changes. And it somewhat becomes a self-fulfilling problem where you wait a really long time because each time you do it is hard, but then the longer you wait, the worse it becomes. And so then of course it's hard because more stuff has changed and more stuff can break if you're doing that process of build test deploy you know just consistently and constantly and there's nearly not too much to be afraid of and so you may still have a acceptance testing phase in there or whatever so it's not although it has its own merits but it's not the the classical build go fast and break things you're not saying it has to be that way it's just being always in the process of doing these things so they're not a we do this once a year and everyone forgets what to

do in the meantime yep yep yep totally right and once you have a lot of this automated and in code, then you can do this continuously. Imagine if you had a software librarian who was cutting a release every five minutes. It's just not practical. I mean, to be clear, there is a role of software librarian. I'm sure in some companies they're still there but i yes i think we used to work at a place that had a software librarian who at a once a week interval actually was responsible for making sure all the code compiled i would not recommend that yeah no i wouldn't recommend that either but to your point they couldn't do it every five minutes so they got kicked out to say once a week it is this person's job to take all the code review all the changes make summary reports and and you know what we now

just do constantly with computer systems they would do by hand yeah yeah exactly um yeah i remember getting just like software librarian coming over to my desk hey so uh broke the build again it's like um now we have a machine that continuously comes to our desk. Now GitHub puts you on blast. Yeah, that's right. Have you ever done anything interesting, like any interesting real-life triggers? I'll tell a couple from my background.

At one point, we built, we found a way where, okay, you know those... They're like cardboard, but they can be life-size. Like life-size cardboard. Yeah, like statues, I guess is what you'd call them. So there was this one guy who kept breaking the build. And we found a thing, I think it was Zazzle or something, where we could get a life-size cardboard cutout of this engineer.

And we had a Raspberry Pi or some Arduino, something set up where his eyes, we put LEDs for eyes. And whenever the build broke, the LEDs flickered. What's the craziest build-breaking thing you've seen? I don't think... I've only ever been on one team, I think, they had some stuffed animal that they would you know take from to whoever was like most recently yeah screwed something up and they would take it to them and that they would you know put it on their desk or

whatever because they were the most recent person that but no in general i i've always heard stories or you know there's a lava lamp somewhere now still does that there's like lava lamps because they take a while to heat up and start going so if the build is broken the lava lamp somewhere now still does that. There's like lava lamps because they take a while to heat up and start going. So if the build is broken, the lava lamp turns on. And then, you know, you got to try to fix it as fast as you can so that, you know, the whole office doesn't see that the lava lamp is happily bubbling away and the build is broken. This is amazing.

We should have like a New York Times square style ball that drops from the ceiling and if if the build is broken for a whole day it hits the ground and then it uh opens up and there's a pink slip i was thinking there's like a kaleidoscope of uh multi-colored lights but yeah maybe firing is this opens up and there's a little receipt printer and it's a dot matrix printer prints your your yes that oh man um oh man that's so good we uh this is another reason why we need a hybrid workplace that we can haunt each other just working remote just isn't the same um oh man okay so how do we measure uh good and bad devops i think is a really interesting part of this um you know one thing actually maybe i'll start by saying you should measure it um in general my my philosophy here

maybe this is like a pessimistic view but if you don't measure something you should expect it to be the worst thing possible that's kind of my overall view of it and so or maybe like the worst within some like common sense reasoning and so devops is the same way if you don't measure it then it's not going to be very good. Or another thing that I've seen happen where you don't measure something, and what actually happens is a few people heroically kind of keep it up and running, but then they end up getting burnt out and just feeling like they're not really productive and hurts morale. So measuring things, extremely important.

How to measure DevOps. I'll just start with the simplest one, build times. Build times, test times, deployment time. Is it taking three weeks to build your deployment image? That's a huge problem. Our engineers just sitting there like wrapping their fingers on the desk waiting for the build.

That's not good. And so just having a graph of that, I think is a good place to start. We didn't talk too much about that, but I think you're absolutely right. Like builds, if not managed properly by everyone, they become sprawling and take a long time and you know it really kills your ability to iterate so it's very valuable to do what you can to make sure your

builds run fast yeah i heard a story from the games industry where um they weren't using dlls um there there was this philosophy that like we just want one ex EXE, and the linker was taking 90 minutes. And then at some point, somebody just broke the paradigm and said, look, we're going to put DLLs. And we'll put the DLLs in the same directory as the EXE. Nobody will even notice. And then the linker took like 40 seconds. But people were waiting an hour and a half it's like oh i missed uh well i guess if you miss a semicolon you get a compiler error

but it's like it's like oh this number was a three it should have been a four 90 minutes you know and that's just that just destroys you know the morale and the productivity um another thing is really important are uh things like release cadence um things like uh bug tracking you know how long uh does it take to fix a bug what's the bug rate you know per week and trying to get that number down um and then the last one which is the most important are surveys um people hate surveys you know i mean outside of work you know i mean you know if someone comes to your door and wants you to know you like are you voting for this you know a school board member or something uh or tell us you know the survey on like this or

you know do you have energy efficient lights like it feels like one of these things that you just want to slam the door in somebody's face um But at work, especially if it's done right, surveys are extremely important. One example of a case where surveys worked really well, when we built the machine learning backend at Facebook, whenever someone canceled a job, there was a survey. And the way it worked is they could click cancel

and then just there was a submit button, they could submit. And the survey wasn't in any way forcing people. But there's a little radio button said like, why are you canceling this job? Again, you didn't have to choose, but you could choose user. I think it was called, I don't want this job anymore.

Basically, there was a choice. It was like, I made a mistake. Then the rest were all infrastructure. The job isn't finishing, or the system has crashed, or the machines keep going down. Like a whole bunch of choices that were all relevant to things we were trying to do better. And then when you clicked one of those,

it popped up a little feedback form, free form. And that was unbelievably useful. So for one, it immediately set a target so people could try to like you know do better on those forms or get get the you know the button press radio button press rates down the second thing is the free form gave us like a whole bunch of really useful context um and what we end up settling on which i think every company should do this is is a a regular survey where we ask people, what is their percent time doing work that they enjoy? And then calling out, what do they not enjoy?

Now, some of these things are, oh, I don't enjoy that we don't have chocolate milk or something. Okay, that's fine. But some of them were like, I don't enjoy just sitting here waiting for my build for an hour. And that is extremely useful information. Have you ever personally, like on any of the teams you've managed, ever done like a survey? You made me think.

I don't think I've ever done that. I've always had like higher level organizational surveys. But have you ever done a survey of just like, you know, send out, hey, everyone, send in information about stuff specific to your team's roles? Yeah, so it's a good point. I'll do it at a low cadence, because you know, the company will do a survey every quarter. Sure. And so sometimes I'll ask for questions to be added to that survey. But But no, I think I'm trying to remember if it's every if it's twice a year or once a year um but yeah we'll do a survey and we'll ask uh you know basically how's it infrastructure you know what are things that are slowing you down what are things that like frustrate you and yeah i think it's it's extremely uh nice to get, get that feedback. I'm going to be cool.

Yeah. One interesting thing about surveys is the best results are actually from the people who are the most reluctant to give feedback. Not because they're afraid or anything, but, but you'll have the people who are angry and the people who are ecstatic, you get the feedback from them like that day. Right. But it's the people who are ecstatic, you get the feedback from them like that day. Right.

But it's the people who are like on the margin and the people who are like, I'm too busy to do this survey. Those are actually often the people who have the most interesting things to say. And and so it becomes really difficult because you really have to push for, you know, as close to 100% feedback as possible, because as you get closer to 100%, the data gets richer and richer. All right, so I guess we'll end it with any DevOps horror stories. I'll tell one real quick. This wasn't a company I was working at, but we hired somebody. This was, oh gosh, probably like 10 years ago.

We hired somebody who's telling me a story about their past company, which was a startup. And they had this. So all of their code was Tomcat which was this java uh web service a library and so the way it works is you create a jar file and the jar file i don't think it spins up a web server but i think it it answers web requests and then you hook it up to a web server i'm starting to like get a little hand wavy, a JAR file, for folks who don't know, is literally a zip file. It's a zip file that has certain files

in certain places of the zip. And it's just renamed to JAR. That's what it is. So this JAR file knew about how to handle web requests. And it had a bunch of Java bytecode in it. And the way they would do DevOps or deployment is when they made changes to the source code, they would remember what files they changed,

like what.java files. They would take those.class files from their machine, and they would inject them, like replace them in the jar in the zip so they would like ssh into the production machine that's serving all the customer traffic and they would open this in like a zip editor and they would drag their class files from their machine and the website he said would go down like once a week uh because of this no one would know how to fix it oh wow i don't think i other than like having an actual software librarian like we were talking

about being ridiculous now i don't know that i have any particular horror stories what about like a downtime ever have you ever been on the crunch where something went down no see we did mostly like internal facing stuff so we've never had like a problem with like slas but you know i i guess we've had issues you know just the kind of normal stuff like somebody built you know with a on a machine that had like a really old version of a library and we didn't even know and then you know like for some reason the code that was everybody was trying to run which does not work we couldn't figure out why no one could reproduce it and there's always this thing when you know when you can't reproduce some problem to just hand wave and say well it's not

broken now but then every time they would build it it would you know it would have the same issue and so finally we we tried to figure it out and it turned out oh yeah like you need to upgrade this library like this this library had a known bug in it. You missed the email that told you like, please update. I got one last story. This is a, this is what happens when DevOps goes too far in the other direction. I was at this place, which the, the PRCI. So in other words, the tests that run when you have a pull request that you want to get merged

in, we're taking like a day and somebody pushed a change where, um, it was a JavaScript or a react, a react bug that they introduced where basically the submit button was never, uh, available. Like it was always grayed out.

So you could like create a machine learning job to define the job and everything, but you couldn't click the submit button to actually launch your job. And so people were having a hard time. And I said, for now, let's just always let people click the submit button. And if people submit a bad job,

it'll just fail on the back end. And that'll just get us through today because a bunch of people are upset about it. So I submitted just a one line change. So literally in the react, it just like in the submit button, the code, it's like active equals a bunch of logic, I just deleted that, that part of that HTML tag, so the button could always be pressed um and then the prci took like half a day and then failed because of like some other change and and basically like you know people were super upset and i couldn't get this like one line html change in uh for like two days um it's just a total disaster.

So I think that's a case where, you know, a lot of tests were running and, you know, I'm sure they're catching things in general, but it was also kind of destroying productivity in a different way. Yeah, that's true. We didn't talk about that. Sometimes you may need to just do the crazy thing

and just push a change and just, you know, get rid of all the other stuff because it's in a bad state and the bad state's not letting it get out of the bad state yeah yeah that's right um so yeah i mean you know software librarians probably not a thing anymore but devops is going to be a thing for a long time devops is not going anywhere um if if this kind of stuff is really interesting to you uh you could definitely get a nice uh career for many years doing devops and i think it's a great feel well thank you everyone for for listening through another episode and uh appreciate all of our listeners all the feedback and uh people on. So definitely a shout out to everyone who makes

the world go round. Yeah, totally. Thanks, everybody. We'll catch y'all later. Have a good one. Thank you.