Python Bytes - #260 It's brutally simple: made just from pickle and zip

Episode Date: November 23, 2021

Topics covered in this episode: Using cog to update --help in a Markdown README file An oral history of Bank Python C Pyxel How to Ditch Codecov for Python Projects tiptop (like glances) pyc64 Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/260

Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode 260, recorded November 19th, 2021. I'm Michael Kennedy. And I'm Brian Ocken. I'm Chris Petty. Chris, welcome to the podcast. You're a podcast veteran, but the first time here on Python Bytes. Happy to have you here.

Thank you very much. It's a pleasure to be here. It's good to be back. We're going to jump into a bunch of the news, but before we do, you know, tell people a bit about yourself. Absolutely. I've been in the tech business for about 35-ish years now, something like that. And I've been a self-identifying Pythonista for about the last five or six years. I work for, these days, I work for, I just recently changed jobs within Amazon. I used to work for the web services, Elastic File System team.

And I now work, I switched gears from, you know, living in the clouds to living in the Amazon fulfillment centers. We're now a network. So does that mean you're doing what? Like programming in COBOL or are we using now? Yes, nothing but COBOL and Fortran. They're just all the worst, all the most possible, you know, not at all.

We do, my team, Optitech IT, we do network automation in infrastructure. So it's a lot of fun, actually. It's really kind of neat going from building microservices and things like that, being infrastructure oriented at that level to like working with being responsible for automating things like, you know, network protocols, DNS, DHCP, fun stuff like that at scale in the fulfillment centers.

It's a really cool gig. Yeah, that sounds really interesting. And you get an interface with physical things that go physical places, which is interesting. Yeah, it really is. Thank you. I only ship bits. I don't know.

Brian, you also work on physical things. Yeah. I like working with actual, we make things. It's nice. Yeah, absolutely. It's a nice change of pace. Yeah, I bet it is. Out in the audience, Ned Batchelder.

Hey, Ned. Thanks for being here. This is Chris, Boston represent. Hey, Ned Batchelder. Hey, Ned, thanks for being here. This is Chris, Boston represent. Ned is amazing. He does so much work with Boston Python and I can't wait till we can get back to actually like being in a room together with other people again. I know, absolutely. So Brian, I used to feel like a cog in the machine, but now I work at a pretty small company where I make a lot of the choices, so I don't have to feel like that. But I think we're going to talk about something that I think is awesome because Ned is here. Yeah, it's kind of nice that Ned's here.

We picked this just for you, Ned. So Ned does a lot of stuff for Python. And one of the things that he's done is this cool tool called Cog, which I've been watching for a while, but I haven't had a really good reason to use. But what Cog does, as you point it, you can have these Cog, like three brackets and the Cog word, and then some code, and then some code,

and then another magic incantation at the end. It's pretty simple, but it's hard to explain. But you have some code, and when you point cog at it, it will run the code and then put the output of the code somewhere else where you put it to. So it seems neat. I wanted to use it, but I didn't know what a use case I would use it for. Well, Simon Willison wrote this article called Using COG to Update Help, the help output for a Markdown readme file.

And I think this is brilliant. So this is super awesome use case. So Simon, he even has it going up on his, uh, C S V S two SQL light, uh, project. Um, and the idea is you've got at the end, uh, the end of the project, he actually lists on the read me. He lists the help output. Now I actually have a project that the help output is out of date because the way I deal with it is I run the help and then copy and paste. But this is Simon's right. This is a great use for cog to just point cog at it. So so he wrote a little snippet of code that goes in his readme.

And since cog ignores comments, you can put it in an HTML comment block. And it so cog will run that file and took the output stick it out and then simon didn't stop there he said uh i want to make sure that this gets run when it needs to get run so he wrote a test in pytest of course uh to um to check to see if cog needs to be written so this is just this is really pretty cool i like it this is the is the kind of thing that's so cool. Pardon me, Mike. This is the kind of thing that's so cool because anytime you can eliminate DIY, right? Anytime you can make one thing, the one true source of whatever piece of the puzzle you're looking to build, you win. And this is a great example of that.

Yeah. Out in the audience, a real-time feedback. Ned says, I also just updated the coverage and cog docs to do the same thing and to get help output into the docs. Nice. Yeah. Very cool. I love the inception of it. Yeah. This is definitely a cool project. I want to take you all on a journey. Come with me into the world of BankPython. My drink. This is an interesting article here. It's one of these things that says,

I can't exactly tell you precisely where I worked or what we did. And I have to make up names for the projects that are not the actual names. It's by Cal Patterson. And it says, the strange world of Python as's by Cal Patterson. And it says, The Strange World of Python as Used by Big Investment Banks. So this is quite the read. If you go through it, if you

click the little reader bit here, it says, you know, 30 minutes of reading time to read it. So it is a non-trivial bit of history. But let me see if I can give you some of the highlights that I took from this. And you can go read it if you really feel like you want to dive into it. So one of the things they talk about is the database and all these things have interesting names. I'm sure that they have actual interesting names as well that are not these names. So for example, like the database like system is called Barbara. And I guess the place to start here with this oral history that Cal talks about is that bank Python implementations are basically forks of the entire Python ecosystem,

not just a fork of CPython, but like the libraries that it uses and everything, which are used at many, but not all of the biggest investment banks. So it says, let's talk about this system called Minerva. And it's a global database of Python objects. And Barbara is this way to talk to it. Okay. So a lot of interesting keywords to keep it hidden, right? But one of the things that's wild, this Barbara thing is it's like a hierarchical database

that stores basically zipped pickle objects so it's and when i say hierarchical i mean it's in what they call rings so you would talk to a database and there might be like a nearby replica that you talk to that might have a record or it might not and so then there's like a bigger replica of sort of fallback to go look in this place. And if it's not there, go back and look at another one. They call these rings. And then you would ask for something by its key and then you get it back. So first of all, what do you guys think of just pickling all of your data objects? It's really, that bit made me chuckle because it is exactly this the kind of like

at the time pragmatic engineering trade-off that somebody made right years ago before we all saw pickles as the the the the great villain of the python uh ecosystem before we knew that there could be downsides to pickling all the things and they're like well it's perfect pickle idea what more could you want and now we're all just kind of looking it's so simple it's small it's fast yeah yeah how do you index something actual database just curious yeah this is a fine question they do if you read it they do also use databases sometimes. Okay. So, but the primary way seems to be this, this system called Barbara. So one of the things that comes to mind for me here is, oh my gosh, what if you upgrade something? What if you change the file, the structure

in code of something that you put in the database five years ago. Whoops. Yeah, that used to be, you know, signed up for the mailing list, yes or no. And now when you change the order, that yes or no became yes or no is admin. Oh, oops. Or it just doesn't match anymore and it crashes and you can't read your data because it's incompatible. So this would happen with Python, the runtime,

but also the libraries you're using, it seems to me that it's just incredibly unstable. So that's... How much would I make a bet they also have like some kind of really highly evolved over many years validation system that ensures that the, that's going to, that the peoples that you think are being used at the way that you think they're being used are actually still used that way. Yeah, exactly. There's got to be layers. Sam out in the live in the audience live stream says, good luck upgrading your new Python. Yeah, exactly. Yeah. Like as in never. And that's probably why part of it is like, yeah, we just forked it and froze it because that's crazy. Let me show you something else. I will try to jump around the article by a search because I took notes. Applications also commonly store their internal state in Barbara, writing data classes straight into and out of Barbara with very simple locking and transactions, if not. And one of the

things they decided is there's no file system available to this platform. And the little bits of data that the script gets picked up and outputs, it all just goes back into this key value store pickled database thing called Barbara. That's pretty wild, right? Yeah. Okay. They also talk about having these overlay features, which is pretty wild. This has to do with the different rings that I talked about. So when you connect, you can say, I want to connect to the middle office ring and then the FICC ring and then the default ring. So then when you do a database query, which by the way, it looks like a dictionary index, just bracket key. It first is going to look for it in the middle office database.

And then it's going to look in the ficc database and then the default database so there's like these layers of settings and these layers of specificity and they they even do things that are to some effect of like monkey patching like if you if you need to change it you can have like your dev ring and then the middle office ring and then the outer ring and you could change the value and kind of like for mocking or something. So yeah, to your point, Brian, they say there are some good reasons, or Cal says,

there are some good reasons not to use Barbara. If your data, let me count the ways. If your data set is large, it may be a good idea to look elsewhere, perhaps in a traditional SQL database or KDB plus, which I have no idea what that is. There's a limit on

the size of Barbara objects, which is compressed at when it's zipped 16 megs or smaller, which is pretty interesting. And then paired with this is this direct this thing called dagger, which is a directed acyclic graph that allows you to set up relationships between instruments like okay, I've got a bond and then I've got a credit default swap on that bond. And then I've got like a fund of default swaps or, you know, like this whole layer of stuff. And all of these things are like linked objects through these pickled things that you can ask questions about and like get sort of like graph database type things. That's pretty interesting. They talk a lot about how you model these things, like instruments and swaps and stuff in Python. That's not super interesting,

but I wanted to highlight this section and see what you all thought. It says, if you understand Excel, you'll be starting to recognize how a lot of the stuff here works. In Excel, spreadsheets are cells and they have dependencies, which is like an acyclic graph, like this dagger hierarchy graph database thing has. And it has a lot. So they basically modeled so much of the code around how Excel works. So you model the data first and then the behavior rather,

usually we write code and then we have data that comes out of it so it's it's pretty interesting there's also this thing called uh wallpole which is a database runner so when you get to this section so there's time to drop a bit of a bombshell let's talk about deployment versioning etc the source code of this entire system itself even including a barbara is stored stored in Barbara, not on disk. And it remains composed. And it's kept in a special Barbara ring called source code. All right.

What do you all think of that? My hidden release engineer just left the building. Oh, man, that's frightening. Yeah. So not keeping the source code in the file system breaks some assumptions. Like how does the program run? Yeah.

So this wall pull thing runs, it's built to basically run source code that originates from a database. And it's all one huge database. So like if you think monorepo, it's like that because you import from database and it's just like whatever you need is there.

So versioning, it's like the anti microservice. Basically, the entire whole thing is literally there. And then the way that you deploy it. Let's see there. They talked about vouch to the way that you deploy to it is really interesting in that you, it reminds me of the, it works on my machine certification program. So the way you deploy to the system is you just commit it to the database and boom, it's deployed. It's running, right? Because there's only one place the code lives. It's in the database. So once you do a commit, that's a full on deployment. Now, how

does it get into there? There's a vouch system where you can get somebody who can code review what you've done and say, yeah, I vouch for this. You can commit it unless you have a high reputation for not breaking the system, in which case you can directly commit to it without question. So I guess the culture of code review is not like a thing there. I guess. So another interesting knock-on that comes from storing your code in a database is how do you edit it, right? Like what editor to use? VI won't work on it.

PyCharm won't work on it. VS Code won't work on it because those work on files. So this Minerva system has its own IDE because most IDEs think you have files, not databases that contain the data. Does that make sense? It is. That makes sense. The guy, Cal says, what I can't understand is why does it contain its own web framework? Investment banks have a one-way approach to open source software. Some of it can come in, but none of it can go out. Well, I've seen projects that like people building IDEs within a company. And I always shake my head and say, this isn't your business. Why are you making an ide within

your company there are a couple things there are a couple things that strike me from this article the first is as i read it because i i really enjoyed it did you read it i most of the way through it's crazy right it really is it really is the thing that struck me is this is a perfect example of large organization, complex code base and engineering system and time. Because this really just smells like something that just got like built up, like, you know, the layers of rings around a tree over time, engineer after engineer, year after year making decisions. And I'm sure like in the modern day everyone who walks to the door

they probably have to plan for the period of time where the new engineer just freaks out when they tell them all this stuff as they adjust to the new reality that they're being exposed to yeah and that in some sense this is a particular wacky case but it's that way in kind of every large corporate environment in a in way that's true yeah that's totally true he actually cal actually discusses this a little bit here and says um this fact that you can't do very much of open source you're stuck on some version of python that's never going to see never going to go back to sync with the real python because how do you even do a database migration to that thing says it could be the biggest disadvantage for you as your profession.

Every year you spend in the Minerva monoculture, the skills you need to interact with normal software atrophy. By the time I left, I had pretty much forgotten how to wrestle pip in virtual EMV into shape. Essential skills for normal Python. When everything is in the same repo and all the code is just an import away, software packaging just does not come up. So, yeah, there's a lot of stuff. That's one way to deal with retention is to have a system where people need to be skilled to use it. And it removes all of their other skills so they can't get another job.

I was just about to say, this seems like a retention add into pattern to me but uh you know your mileage it depends on which side of of that uh leave stay story you're on because if your job is like i want job security and i don't want to have to learn new things like welcome to minera demer demer nervo whatever this is right right? You're home. You can stay here forever. They won't replace you. On the other hand, if you want to use like FastAPI, you're like, I got to get out of here. We can't be on Python 2.3 this long.

I'm out. So I did want to mention that both Minerva and Barbara are things that you can pip install, but those aren't things we're talking about. No, these are like secret code names. I suspect the Minerva you installed doesn't have its own ID. I'm just guessing, right?

Yeah. There's a lot of interesting conversation here about this. Just a quick little note. I actually ended up, I'm like, it was late last night. I'm like, I don't want to read this like 30 minute thing, but I do want to talk about it tomorrow. So I actually use this thing called natural reader. You just throw this in here and it'll just like read it back.

Super nice. That's so cool. So I was, it was like 1130 last night when I started on this. I'm like, oh, I'm a little too tired to focus on this much stuff. But yeah, it's pretty nice. You may have just made my day because I actually, you know, being part. Yeah. Because being partially blind, I, I, I feel like, you know, how some people who have like lupus and other kinds, they talk

about spoons, like they only have so much energy. My eyes only have so much like ice cream spoons. Yeah. It's the course of any one day that I can, I can do it. There are times when I'm like, I cannot stare at a screen for even a moment longer. And I really love things like audio books or the rear services like pocket that will actually read you articles. And so this sounds like an awesome tool for those things that don't have that

feature already. Nice. It's really neat. It has different, different modes. I don't know how to make this thing come up without creating an account because I don't know what my login is here, but it has like,

if you're dyslexic, it has like a dyslexic font you can turn on and it'll highlight as you write. It's pretty neat. So yeah, not directly related, but I've found it to be useful in doing this research. I was too tired. A little bit of audience interaction here. Brandon says, run the source code from a database. That's a one heck of an injection attack. Yeah, I mean, little Bobby Tables would be just all over this thing, right? It's one big injection risk.

Although on the other hand, I feel there might be some big safety here as well because everything is pickled. You've got to know what version of Python they forked it from and any variations they may be changed to the object model to even get it to parse it back.

So it might be naturally safe as it's just like an opaque. It's almost like encryption. Really, really, really weak encryption. Yeah. And then let's see. Yeah. David says absolutely everyone in the live audience has a furrowed brow right now. Yes, David.

Or some nervous laughter. There is one serious aspect of this, though. We're talking about retention and the article talks about as he worked in this private ecosystem, he felt his external skills atrophying. That is a common problem that I have heard so many people who work in these large

kind of like pocket universes, whether it's an Amazon or a Google or a Microsoft or an IBM or whatever, people can kind of feel very disconnected from where the field is currently going at the hot leading edge in the outside world. And I think it's a real issue because you can't always justify pulling in the latest greatest. Sometimes there are engineering and patience to that that would be really problematic to your business's bottom line. And so I say, and I realize time is finite and this is hard, but this is where I say even dabbling in having side projects, even if it's just the kind of thing you do on weekend every once in a while when you have time. It's a great way to sort of like feel unconstrained by the requirements of your business and to go sort of play on your own terms. I mean, write helper scripts for yourself, even for your job in modern Python.

Bingo. Yeah, absolutely. Good advice indeed. Python. Bingo. Yeah, absolutely. Good advice indeed. All right. Over to you, Chris. Okay. Thank you. So I decided I was thinking about this and thinking about the fact that, you know, most of the people who come in and are guests here on this podcast are really sort of accomplished hardcore Python programmers.

And they're familiar with the latest and greatest tools. There is no way that I'm going to scoop you all. So I decided to go a completely different direction and go in the direction. Scoop us with fun. That's right, exactly. Of the whimsical side of computing because I think we all need to stay in touch with that.

I know I had an issue a number of months ago before I changed jobs where I was feeling very burned out. And I felt like I just, I need to sort of get back in touch with like, why am I passionate about computing and programming in this space? And one of the things that really was balled into my, you know, soothed by my burnt out soul was these retro consoles. I used one called Tic 80, which was not Python based. It's Lua based. But the idea here is you can build like basically retro style video games very, very, very easily with very few lines of code. Because these things, they do have some constraints, but the constraints are completely artificial, right?

Like in this case, this one is fixed 16 color palette, 256 by 256. You know, like the constraints, the world that you're building in is uncomplicated and small. And so as a result, you can sort of like step in and wrap your head around this thing so quickly and come up to speed. And suddenly you're like, you know, oh, I need a flying saucer. Fly across the screen. I built a sprite. Oh, I'm making music. I'm coding, you know, the main loop of my game.

And you can do it in a very, comparatively speaking, short period of time with just a few lines of code i'm just skipping down a little bit here um like let's take an example they have the sort of like you know um sneak eating dots kind of game and it's comparatively speaking like when you compare it with like one of the modern you know game uh framework code bases Like Unreal Engine or something. Yeah, Unreal Engine or D2D where you're talking about here's my 60 pages of coordinate transforms

and you just look at it and you're like, I can't, I can't. And my 100 gigs of textures that we're going to export. Exactly. So this project is called Pixel. I'm guessing that's how you,

P-Y-X-E-L. And it allows you to create extremely simple 2D animated games in Python, right? Yes. I apologize. I shouldn't have said that. Yes, exactly. And so back when I was using the other one, I kept thinking the entire time, like, wow, you know, I'm really having fun with this.

I actually really enjoyed learning Lua. And it's a really fun language. And it has a lot of characteristics that are wonderful. But I kept thinking like, wow, I kind of wish I didn't have to get my brain to switch gears, which is not very good at doing always. I wish I could be coding this in Python. And so this framework gives you the opportunity to do just that. And so you can use these. It almost feels like for those of us who are old enough to remember, it almost feels

like coding in basic on the microcomputers that some of us grew up with in the 80s, right? Like it's that sort of kind of way back when you could type it in. You're like, I saw it printed and I typed it in and I ran it. Exactly. Exactly. Exactly. You know, it's like, yeah, it really, really is. And so I really, this one, what's, what's kind of interesting about this one is like, this is a great example that the sort of like sound and music section of this snake game here is, um, this one is actually a little bit more pythonic, which I think is actually

really kind of cool for Python folks like us, because the other ones that I've seen kind of like drifted more back towards the like that Python has to offer. So you can use this really kind of nice express and tax to, to write your games. And I am just super excited. I, I, I played with it a bit. I wrote a few lines of it and I, I, you know, kick the tires just to see how it worked, but I really have not had the opportunity yet to, to dive deep and kind of rewrite my side project in it. But I totally intend to, and I'm super psyched about it. And I'm really, really glad that it exists. Yeah, I'd love to do this. I mean, I got my start typing in games from the back on a TRS-80

from the back of magazines and have an example. So one of the things, some engine like this, one of the things I love to do with somebody is take an existing game and say, let's look at the code and see what happens if we modify this field. Is there any way we can get the character to run faster? Does that make it more fun to play the game or less fun? Things like that. Absolutely. And that kind of thing, that kind of like, well, let's take a look at this and tweak it a little bit is so simple with this kind of thing, because it's like everything is so like uncomplicated and straightforward. Like chances are like the here is the number of frames that we skip as we transition our sprite character across the screen. Oh, let's just tweak that.

Now you've done your sort of like, let's make him run a little faster kind of thing. It's all so satisfyingly in a world where, I don't know about you guys, but my day job, especially the old one that I did, was like, you know, make a code change or make a configuration change and then wait, you know, eight hours for it to deploy through the pipeline. Like having something that's immediate instant feedback

like this was just a delight. Sorry. I'm just shocked at the eight hour turnaround time i'm so sorry that was it that was it that was just a really bad example there that you know a couple of cases it's not that's not my day-to-day life i actually love my day-to-day life i'm very happy but that was nice you know what i mean we all have these like long build times yep yep will mcgougan, it says this makes me nostalgic for my ZX spectrum in the 80s. Exactly.

There's an Atari 800 XL behind my head that I, I enjoy as well. So I feel that who's not surprised that Will's been coding since the

80s. So anyway, now, now, and then also a few more follow-up items for your comment on side projects. Sam in the audience says a hundred percent agree. Chris side projects are a great way to keep yourself sharp.

And Jose says plus one for the side projects to try out the latest indeed. So before we move on though, Brian, I want to tell everyone about our sponsor. This episode is sponsored by shortcut again. Thank you you Shortcut for sponsoring and supporting Python Bytes. So there's a lot of project management tools out there, but they often suffer from either being too simple for engineering teams or too complex. And they end up looking like Bugzilla or something insane. And you're like, I really don't want to go here. So you want something nice and simple,

but that actually scales with you. Something that's good both for managers and for engineers. So that's what Shortcut is. It's built to cater to both of these groups. So it's built for software teams based on making workflows easy. Big scale objectives, goals, roadmaps can be easily tied into the team or individual work sprints. Progress

on those are easily flow back to higher level reports. And of course, as all things should be, it's keyboard friendly. You don't have to take your hand off and do a bunch of mousing around. You can use the UI if you like, but there's a bunch of hotkeys, which is fantastic as I think they all should be. So it makes you work faster and just keep going. Good version control integration works with Git, GitHub, GitLab, and so on. A lot of the places that you probably already know. So you can update your task project just by doing a commit or a PR, which is fantastic. The last thing you want to do is update Git and GitHub and then go update your system, right? As iteration planning, you can see your team activity when you want to, or just focus on what you're doing. And you can even let it schedule your project for you. So

nice burn down charge, cycle time, all that kind of stuff. So finally, you can have a project management tool that's both for engineers and managers. Give it a try at pythonbytes.fm slash shortcut. It's a cool project and it helps support the show. So thank you for supporting us. Brian, what do you got next? What are we ditching now? it's a cool project and it helps support the show so thank you for supporting us brian what you got next what are we ditching now we're ditching codecov so harsh proud um so okay so codecov has been great uh for a lot of a lot of people have used it it's a it's a a third party third service that you can attach to your repo or something. And it makes, it makes sure that you,

and this has been helpful for merge requests and stuff. So if somebody does a PR, it makes sure that the, the coverage is, if you've had a hundred percent coverage and make sure that they don't degrade it, the, the code they add makes the coverage stay at a high level. But so it's, it's not without problems, however. And, uh, Hinnick, uh, mentions that on some of his projects, it's been, um, sometimes a CI pipeline is blocked just because it timed out with code code curve. And that's not a good reason to, to fail, uh, uh, fail a build. So, uh, so anyway, anyway he so hinnick has this this project where he said wait a second i'm just trying to make sure everything goes to 100 coverage.py hey ned

second second thing here for you but uh coverage.py from uh ned batchelder maintaining it of course uh has a thing that it says fail if under 100, under a percent. So you can do 100%, but you can do 80 or whatever your current percentage is. But anyway, if you want that, something like CodeCov, you can get that with Coverage.py, but there's some tricks around it. So if you've got, one of the things Coverage does

is let's say you've got a project that has some, some code that is, that is run, like you're trying to take advantage of some of the newer things, but you also newer Python things, but you want to run on older Python too. So there's some if statements for Python version, for example, but it might be a different hardware as well, like windows versus Mac, but there might be code that's run in one test suite and not in the other. So in order to do that, you kind of have to run in parallel mode and then combine the coverage reports at the end.

And it sounds complicated, but mostly we just copy somebody else's work. So and Hinnick has the examples here. So Hinnick goes through using GitHub and coverage to do everything that you used to be able, you were doing with CodeCov. So this is a nice article with examples. So we've got for each project, we're downloading the coverage report and then combining it at the end with a report with a fail under 100. Really cool. If this is something you want to do, it's basically, it's a short article with examples.

He even shows, he's got in his struct log, he shows his main workflow for GitHub pipelines that does all of this. So just kind of read his code and make it work. And I was actually thinking of doing this on a couple of projects I've got. I wanted to make sure that they were up at 100% and do something like this. But I actually, for some reason,

I was dragging my feet and I didn't know why, but now I don't need to. I'll go ahead and use Hinnick's suggestion. Now you don't have to invent it, just set up that job on GitHub. This is actually something I've been kind of struggling with. I shouldn't say struggling with. I should say this is a journey that I have been on

in that testing and code coverage and that whole class of proof of correctness is my weakest suit. And one of the things that I've really been sort of try to wrap my head around is I see a lot of like unit test code that seems like it was just it's just there to keep the coverage percentage up and as far as i can tell it doesn't actually test anything like we're you know creating these mocks and and like and everything

and but they don't actually like enforce any contracts or prove anything out. And I'm trying to figure out how we can use these tools in ways that actually sort of raise the bar on code quality without encouraging this kind of like, I don't know, T-crossing and I-dotting behavior. Well, we're kind of getting on a tangent, but I would say that code reviews on tests are important because that's just, I mean, one of the easiest ways to get a hundred percent coverage with no tip failing tests is don't put any asserts in your tests. Just saying. Good. And that makes an excellent point unto itself. Yes. Thank you. Nice.

All right. Well, let's talk about the tippy top, the tip top. So I'm sure you all are familiar with top. It's like this somewhat dreadful way to tell what's happening on a Unix system, on Linux to see like what's happening. Is it busy? Is it not busy? I say dreadful because it's so simple and there's better tools. One of them being this tip top thing. So for a long time, I use glances.

I didn't know. I still do use glances. I'm not sure if you're all are familiar with the glances, but the UI that it creates is fantastic. So you just type glances and you get this sort of progress bars that show you the CPU, the memory, all the processes. It has like hotkeys. You can sort by CPU or memory,

filter by process name or ID, all kinds of stuff going on here. And this is what I use now, but Zach Villers sent in TipTop as a competing thing. And by the way, Glances is Python and TipTop is Python. So very good candidates for pipx because no project needs them. You just want to run them as commands,

but they're Python, right? So if you look at the same picture over on the tip top page, you can see that you get similar information, but you get a little bit better historical understanding of how your CPU is doing, how your memory is doing. It even breaks it down per thread or per core. It shows you your memory broken down by what's being used, what's being cached, what's free. There's a graph of that over time. Something that made me sad is it shows the network over time as well, the in and out flow.

But what happened to me is one of my servers, I ran this and it only would show the output from like some kind of, there were multiple network adapters and would only show the one that's not being used, not the one that's being used. So I'm like, well, that part is useless, unfortunately here. And then it shows all the programs like the CPU usage, the user it's running as, the arguments that were passed to it, how much memory it has and stuff. So I don't think I'm going to give up using glances because I really love that. But this feels like another tool to get a view of that sort into it.

What do you all think? I like the visual design on the face of it. It like especially comparing it to glances screenshot showed. It feels like it's a little prettier and it may to me at least it looks a little bit less like dense and maybe a little bit easier to sort of like, maybe it's just the bounding boxes, but I also feel like it's the way the screen is laid out a way to visually an easier way to visually parse the information that's presenting. That's always something I look for in tool. Absolutely. Right. Why do you need to, I don't use anything like this. So. All right. So for example, if you log into, let's say you logged into a web server that was running

a search job and it said, you're running a little bit low on memory as a, over the last few minutes. And you've got five different web apps set up running their web processes with like a bunch of their sub processes running. And you wanted to answer the question. Okay. Which one is it? Which one do I have to restart?

Okay. Yeah. Things like that. It's easy to see where you might not encounter this or have a need for this when you're working in the, the world of things, right? Like the world of not servers,

but embedded devices that might run Python or C or whatever. And, you know, this is okay this is like managing infrastructure so you don't use this on your desktop no i don't use this on my mac but i use this on all of my not necessarily tiptop but glances on all of my linux servers including the python bytes one so it's it's nice to go in there and see what's going on in the servers with all i have is ssh i'm more of a high level person i'd rather pay somebody else to do this for me well let's like um but it looks pretty let's see if i can um pull up a picture of top on behalf of my employer brian thank you thank you yes we're happy to take

to do that for you no problem like this this is the thing it's replacing by the way is it's just a list of processes and it moves around all the time and the stuff shoots off the screen it's like why is it doing this you know it's i mean to be fair it was also written probably in like 1968 or something yeah yeah absolutely yeah it's i'm not knocking on it I'm just saying I would rather use, you know, something like this that gives me way more real time, like graphs, not just snapshot. Now it is, now it is,

now it is, but like, here's, here's what's happening as the things are running and doing their work. Right. So that, that's super helpful to me to see. To be,

to be fair. I also just, just installed tip top just for the heck of it on my Mac and a really quick install. And it does look pretty. I just don't know what to do with it. But yeah. And by the way, let's just tie it back to the audience again.

So TipTop uses Textual for layout, which is based on, which is Will McGugan's thing, also based on Rich. So it's quite modern. I think if we had Ned and Will in the audience, we would always touch on something they were doing every year. I'll tell you what, Brian, here's a way you could use this totally in your day-to-day life. Your Mac is running slow, right? And you say, why is my Mac running slow? You could totally use this and at a glance say, what's hoovering on my RAM? What is throttling my disk? What is using up all my CPU, et cetera, et cetera.

Okay. Yeah. Like, for example, you can run activity monitor, which will show you the process thing. And you go to the memory thing, which will show you just a single graph. But like, why is it using all that memory? Is that actually being used or is that cache? Because you loaded a bunch of files and now it's kind of could be free, but it's not. What's happening on the network?

You can't see very well on Mac OS. I don't know why activity monitors are adverse to showing you network traffic, but you could see like, am I downloading something or not right now? I can be taught new things. Okay, thanks. Yeah, yeah, yeah. Anyway, I'm not excited for that use case. I'm excited for using it on the servers to understand them better.

Yeah. Yeah. All right. Let's bring it home with some more retro, Chris. Back in the world of Wimsy, I've always been into, well, you know, always ever since they weren't retro and they were just 8-bit machines. I have, I grew up, started with 8-bit computer.

I was an Atari kid, never owned a C 64, but someone has actually written a Commodore 64 emulator called PI C 64 in pure Python. And the reason that I picked this, it really is, isn't it? I think it is so cool because not only do you get this Commodore 64 emulator, it's not 100% coverage. Like, he has openly emulated the display to the point where you can display PET-SKI characters. But the bottom line is, it is, other than that, a largely full-machine emulator. It runs on the Pi65 library, which is a 6502 CPU chip emulator also written in pure Python. Josh Pugh, that's hard work.

It really is. And, and it's so neat. I was able to pull this thing down and run it. And I was actually writing like Commodore 64 basic that hoped to bits. It's a screen memory and, you know, drew pictures off the screen. And, and one of the really cool things is this thing even concludes, you can sort of like pop out of your emulated Commodore 64 with the go-by command to get a Python REPL.

Skype is actually doing that, like, qualifying the emulated Commodore 64's memory by tweaking the Python data structures inside the REPL. It is just, it is. That's pretty deep. Thank you. It is really cool. And it is also, from my perspective, one of the reasons I picked it, it's a testament

to this programming language that we all use and love for our work. And, you know, whether it's webby stuff or testing devices or in my case, automating network infrastructure, it can do so much. You can reach out into all of these other really kind of full niche areas, whatever you're interested in, and do neat things with Python on your computer. So I just thought this was incredibly cool. I have enjoyed playing with it and I definitely intend to enjoy playing with it some more. There might be some interesting ways to go back and study operating systems back when they were simple

if you were like a computer science student as well. Yeah, for sure. I think there's a lot to learn there. Even if you're not necessarily someone who is keenly interested in writing your own operating system, I think there's a lot to learn just in terms of the programming paradigm, right? Like when you wanted to solve a problem and you wanted to display something on the screen

and a perfectly reasonable answer was, I'm going to poke the right bits into screen memory because that's a more efficient way to display this thing as opposed to using whatever my high level language provides. It's such a different way of thinking from the reality that most of us have it day to day. It's just a fun, refreshing place to visit, at least from my perspective. Yeah. Very cool. It's a good one. Thank you. Brian, it's time for extra, extra, extra. I hear all about it. How many extras do you have? I have zero today. You have no extras.

You're not adding to the number of extras. Chris, anything else you want to throw out there to let people know about? Sure, absolutely. My team at Amazon is hiring. Hopefully we can put a link in the show notes. I work for OpsTech IT. We work with the fulfillment centers

and I cannot say enough how much i love working with this team people are incredible and they're sort of like uh at least the folks i'm working directly with are people who started out in the fulfillment centers and were interested in technology right and they were able to prove that they could actually walk the walk. And now they're being promoted into full on engineering positions. And I just find that so incredibly refreshing because that's fantastic. Yeah. These people just have a very different perspective. You know, they're not the jaded, like I've been in the tech industry for 20 years, types like me. It's, it's a really inspiring to work with them. Yeah. Not, not too many debates of like, why don't we just switch to a functional language?

Come on. More practical, we'll just get stuff done. I love it. Before we move, yeah. Yeah, so we'll put that link in the show notes. Before we move on to my extras, which are not many, Sam says, it's so helpful to understand how a computer works when you're writing code.

These kinds of things are a good way to learn that, talking about the Commodore 64 emulator. Awesome. All right. So I just have one and that's I'm going to be speaking at FlaskCon 2021. And so FlaskCon is the first week of December, which has a remote attendance. I'm going to be talking about HTMX and Flask once again, because I'm super psyched about all that stuff. And so if you want to see some really cool apps get built using HTMX and Flask, drop in over there. There's a whole bunch of other talks going on as well. So I'll put the link to that in the show notes.

All right. Shall we finish off with a joke? Yeah. Levity is the soul of wit. Absolutely. It is. So we just recently went through the whole daylight savings unraveling and went back to normal time. And I don't know what it's like for you all,

but when I look at the weather and stuff here, I suspect Chris, it's similar in Boston. I mean, I know it's similar February. Yes. It's like sunset is at 4.38 p.m. and we're headed towards the winter solstice. So it's only getting earlier in the day. It's like, ah, that doesn't make me, it doesn't spark joy in my heart that it's already dark at 4.30. I like it. Yeah.

I don't mind that it gets dark. I just wish it wouldn't, I wish we could just move it so there's a little time at the end of the day where there's still sun. I remember. I'm with you, Michael. I'm solo powered. So I enjoy this sunlight. I don't like it getting dark earlier either. Yeah, absolutely. I don't hate the winter. I just don't wish it shifted around a little bit. So this is a technical take on that. Our joke is this comes from the New Yorker. I mean,

we're pretty highfalutin, right? We're not doing like devjokes.io or whatever it was anymore. So this is by Matt Reuter. And it's a little cartoon of two people, clearly in kind of a darker space, a woman sitting in her chair with a laptop. And you could just like the beaming laptop screen is like lighting her up, right?

And the guy here on his tablet, same thing glowing on his face. And underneath it, it says, I hate how the screens get bright so early this time of year. Nice. This is so much about our modern world. It's both, it's both hilarious

and kind of sad. All it would go. Absolutely. Yeah. So welcome to daylight savings or standard time, I guess, everyone. The screens do get bright this time of year. That's funny.

Nice. And Chris, thanks for being on the show. Brian, thanks as always. Yeah, thanks a lot. Thank you so much. And I really appreciate everything that you guys do for the community. It's really, you guys are great examples of why Python is such an amazing place to live. Thank you.

Thanks, Chris. Really appreciate it.