Python Bytes - #339 Actual Technical People

Episode Date: June 7, 2023

Topics covered in this episode: pystack Securing PyPI accounts via Two-Factor Authentication Propan - a declarative Python MQ framework Makefile tricks for Python projects Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/339

Hello and welcome to Python Bytes where we deliver Python news and headlines directly to your earbuds. This is episode 339, recorded June 6th, 2023. Is it the 6th? Yeah. I am Brian Ocken. I'm Michael Kennedy. Today's episode is sponsored by InfluxDB from InfluxData. Thank you, and we'll talk about them more later in the show. If you want to reach any of us or the show, we have a contact form, of course.

And then also M. Kennedy at Fostodon, and Brian Auchin in Python Bytes, all Fostodon, Mastodon locations. If you're listening to us on a recording, or on YouTube, or on a podcast player, please join us on YouTube at pythonbytes.fm slash live, at least occasionally, because it's fun to have people hanging around while we're recording. It's usually Tuesdays at 11. And you can watch older versions there too. So let's kick

it off with something stacky. Something stacky. You feeling like some pancakes? A stack of pancakes? Yeah. How about a PyStack? So the reason I was late to this recording, Brian, was I was just talking with Pablo and Matt, maintainers and creators of PyStack. Have you heard of PyStack? I have not. PyStack is a tool that uses forbidden magic to let people inspect the stack frames of a running Python process or even a core dump that was captured from a Python process that crashed, helping you quickly and easily learn what it's doing. How cool is that? Pretty good. Yeah.

So here's the deal. Yeah. So here's the deal. I've got a Python app. This is especially important if you have mixed code. So if you're talking with C, C++, Rust, those kinds of things, because it'll cross those boundaries as well. But let's just say pure Python even.

I've got a Python web app and I go to the server and I try to connect to it. It won't really respond. It connects, but it just hangs. Go to the server. It's not 100% CPU. In fact, it's 0% CPUpu in fact it's zero percent cpu usage so it's it's not like spinning and busy like what the heck is it doing is it a deadlock

is it waiting on like the database what is going on so what you can do even in production you can go up to that process and you can say give me a snapshot of exactly what this process is doing. And what you see is you see a call stack. Let me find an example here of what it looks like. It looks like this, silly zoom. So what you'll see is like, hey, on this particular thread, we're seeing on this file, on this line, this function was called and check it out.

It even has the arguments passed to the function. Oh, wow, that's nice. Yeah, and then you can see what function that's even has the arguments passed to the function. Oh, wow. That's nice. Yeah. And then you can see what function that's calling with the arguments passed to it and what function that's calling with the arguments passed to it. You can do this on a running function without altering its behavior. Basically, it doesn't inject any code or anything. The only behavior it alters is that like freezes it for a second, potentially, which could, I guess, make something timeout. But other than that, you could do this in production even to see what's happening. And what's extra cool is if even if the process crashes, you can grab the core dump and it will go back and analyze that as if it was

a running process. Oh, that's pretty cool. Yeah. So there are a ton of features. If you can get to the section where it says, what can PyStack do? So it works with both running processes and core dump files. It'll tell you if a thread is currently holding onto the GIL, if it's waiting to acquire it or is trying to drop it. So you can, you know, one of the examples that Matt and Pablo spoke about was they were calling into custom C code that was a Python extension. That call that was coming Pablo spoke about was they were calling into custom C code that was a Python extension. That call that was coming in, that was acquiring the GIL.

But then in the destructor for some object that it was waiting for it to go away, it was like waiting on a background thread to do some cleanup. That background thread also was trying to do a callback to let Python know what's happening and was trying to acquire the GIL, but it couldn't. So because the one that was waiting on it

was already holding the GIL and wasn't going to give it up because that's how the GIL works, right? And so you can use it for like these deadlock situations. You can see if it's running a GC, you can see both the call stack intertwined for both Python and C or C++ or Rust altogether. And it'll even do things like go out and find the debugging symbols for, say, your Python runtime. Even if you don't have it, it can potentially go and get those and bring that extra information in. What else should we see here?

Safety using and running processes. You can run it on a process in memory, running process without pausing at all, which will minimize the impact it might have, but it'll also potentially have like not 100% precise information. It could be out of sync. So yeah, it even works on like corrupted process core dumps

because the process died because it got corrupted memory or something so if you've thought about gdb or some of these other types of things because you're like oh my gosh i've got to i've got to figure out why this crashed here's a core dump let me start looking at it well high stack may be the thing you want that's pretty cool and one final bonus for you, Brian. Suppose you have a PyTest test. Is it in here? This one. Suppose you have a PyTest test

and that test, while it's running, deadlocks or is very slow or something like that. You can have PyStack as a PyTest plugin. And then when you run your code, how do you do it? I think it's you just, where is it? Anyway, when you run it, you can say basically,

analyze my tests and here's a certain threshold to consider a failure and take a snapshot of that and so on. Yeah, I like the threshold notion of just like, if it gets this bad tell me why yeah nice cool so if people want the full details they i suppose they could go check out the youtube live stream channel for talk python now or in three weeks they could listen to the podcast but super super cool tool if you've got a process that is crashing that is hanging maybe

it's doing this in production and it only gets deadlocked after, you know, 12 hours of being hammered on. You can't easily just debug it locally and get this to happen. Or if it's completely crashed and you have a core dump, these guys are doing lots of magic to make it possible. Nice. Yep. Cool.

Well, next I kind of want to give everybody some news. So last year, actually, it was in July last year. So we talked about in episode 293, we talked about some giveaway, some PSF, the PSF saying that there's a there's like the top 1% of the critical packages are going to have to use two-factor authentication. And it was big drama at the time, right? Yeah, well, because there was some confusion over the keys and stuff like that, or hardware keys, and yeah, some pushback against that, or just some confusion around it, I think. But we've seen some even more attacks against Python projects in the last year. I mean, it's only been less than a year since that. And so the change is this year, PyPI is going to require everybody to use two-factor authentication.

Not the top 1%, the top 100%. And you got until the end of the year i think um and it's a c um we're linking to a an article from the python package index saying uh securing pi pi accounts via tooth factor authentication and uh as of today um they're going to uh require they're announcing that every account that maintain every account that maintains a project or organization on PyPI will be required to enable two-factor authentication on their account by the end of 2023. So that's the news, really. There's some discussion as to why in this article, but there's some information on how to prepare. But it's not,

I mean, it's not that bad. I did it last year. If you've got, especially if you're already using a smartphone, I think that using something like off the, or something like that on, on a smartphone would work just fine.

So what else there's, it's kind of, I guess there's not much really more to say about it is that this is happening. And you got to kind of do it. You got to do it by the end of the year. Yeah. Wait, but why not? Why wait, just go ahead and do it. And it's really everybody. So it's so let's say you've got an open source project, and there's like, you know, 20 people contributing, that would be cool. Maybe

there's like five, but if only one of you is ever pushing to pipey i then it's just one of i think it's just one of you unless you're doing an organization thing uh i think it's just a person pushing so if other people are on get um and not using two factor forget um but they're just pushing uh to your repo i think that's still fine that doesn't matter it's the people actually actively interacting with pipey that need to be authenticated yeah that's what it sounds like to me as well it's kind of if if you're actually have an active account on pipey it's not necessarily get up although i think github itself also has a 2fa requirement now and there is some discussion here about like people that don't interact with their with a project but still have a PyPI account,

and I'm not exactly sure why. Apparently there's some people that need it. Why would you have a PyPI account if you're not pushing stuff? Yeah, yeah, that's a good point. It is a good point. Well, so there was so much drama about it, and there was that person that deleted their, all their packages because they were frustrated as like a thing of protest

and it caused some issues. And well, I'm fine with this. This is great. I think it's supply chain issues are really, really serious. So it's, it's okay with me. Yeah, me too. So shall we thank our sponsor? We shall. But first, I just want to point out, yeah, I think Authy is a fantastic option for the 2FA stuff that you were pointing

out, right? As you mentioned, like one of the things that is a huge hassle for a lot of the systems is, guess what? You can install this 2FA tool onto your phone and it's completely safe and all that local that 2FA it'll never go anywhere until you want to get a new phone and then you're completely out of luck and you've got to somehow reset it or worse you lose your phone but it's not there's no way to recover the 2FA code so what I really like about Authy is it will you can install it in multiple locations like you can install it on your desktop and your mobile device, and they're just in sync. If you add one somewhere, it appears elsewhere.

So if people feel like TFA is a huge pain, I think Authy is one of the choices that's pretty good for that. I didn't know that. You can also do like 1Password and so on. But to me, having the passwords there and the 2FA thing in the same place seems to violate some aspect of security. I mean, I know one password is pretty safe, but 2FA should be about having the password

and the thing separated in my mind so I don't use my password managers 2FA thing. Yeah, I just thought I had like just a couple accounts with Authy and I just looked and I've got like, I got a scroll. I've got a whole bunch of things on Authy. Yeah, I think I have 40 or so myself all right now let's tell people about our sponsor all right this episode of python bytes is brought

to you by influx data the makers of influx db influx db is a database purpose built for handling time series data at massive scale for real-time analytics developers can ingest store and analyze all types of time series data, metrics, events, traces on a single platform. So dear listener, let me ask the question, how would boundless cardinality and lightning fast SQL queries impact the way you develop real-time applications? InflexDB processes large time series datasets and provides low latency SQL queries, making it a go-to choice for developers building real-time applications and seeking crucial insights.

For developer efficiency, InfluxDB helps you create IOT analytics and cloud applications using timestamp data rapidly and at scale. It's designed to ingest billions of data points in real time with unlimited cardinality. InfluxDB streamlines building once and deploying across various products and environments from the edge,

on-premise, and to the cloud. Try it for free at pythonbytes.fm slash influxdb. The links are also in your show notes on the podcast. Thanks, InfluxDB, for supporting the show. Yep. Thank you. Thank you. Everyone check them out to help support the show. All right. Let's talk about queues, Brian. Okay. Yeah. So I want to talk about Propan. Now Propan is a project that's not, you know, tens of thousands of GitHub stars. I think it looks, it looks pretty compelling. It's put together by Lance Nick, Lance Nick, Lance Nick, I'm going to go with over here on GitHub. And it is a powerful and it looks it looks pretty compelling it's put together by lance nick lance pick some lance nick i'm gonna go with over here on github and it is a powerful and easy to use python framework for

building asynchronous web services that interact with any message broker so what are some of the options of the message brokers here we've got rabbit mq redis n, NATS, Kafka, SQS, some of the other ones like Redis Streams. If you're using these and you want a cool declarative way to interact with them, then Propan might be your thing. So right now what they have is async APIs for you, and they're working on synchronous ones, but they don't have them yet. So let me just give you an example, Brian. Over here, it says, first, let's take the quick start from AIO PICA, which is a way

to talk, way to listen for events. This is the important part of listen for a certain set of events coming into a message queue. Okay. Okay. So what you do is you say, I'm going to connect to the message queue server, and I'm going to listen to a particular queue. Then you await creating the connection. You await creating a channel, you await connecting to the queue. And once you do it, then you, you know,

use the iterator, you loop over the iterator as messages come in and then you get them. And then you of course, you know, run that code that does that, right? That's the iterative imperative way where you do all the steps yourself. So this other way is what you do is you go to, you basically create this thing called a broker using Propan and you point it at one of these queues like Redis or something.

And then you just kind of like you would in FastAPI or Flask, you say, you put a decorator on a function and you say at broker dot handle and you give it the name of the queue. So if a message comes into that named queue, call this function. Oh, I like that better. Isn't this nice? It's kind of like I'm listening for this URL. Like if, you know, slash courses slash ID of a course, I want to get you details about that course, Right. You would put that in Flask or Pyramid or FastAPI. This is the same thing, but for message queue. So you say this function receives stuff that goes to that queue. Oh, I like it. Yeah. That's what those interfaces should be like.

Yeah, absolutely. It totally should. So this is pretty interesting already, but it gets a little bit cooler. You can go and create one of these apps and just run a server directly, right? So you can say, I wanna run this as a system daemon on Linux, let's say, and it's just gonna, you know, use the propan server to run. That's fine, but there's tons of infrastructure around running these types of things as web applications.

And if you already have a web app that receives like JSON requests, you know, it's got some kind of API endpoint, but you also want to have it handle stuff that might be put into the message queue, then it has integration with, if I scroll down, into, you can do it manually into any web framework or it's got things like a fast API plugin,

which is pretty cool. Oh, cool. Yeah, so let's see API plugin, which is pretty cool. Oh, cool. Yeah. So let's see. Actually, if I go to the examples, I'll pull up a Flask one. That's probably the best, which you got to use Quart because it's only async. That's the Flask async variant. So what you can do is in your, let's see, I'll just say in your Quart app, you create this broker to sort of listen as well, in addition to create your Flask or Quart app. And then you might have a function that says app.route listen for forward slash.

And that's a JSON endpoint. Or you might have broker.handle some queue message. And that's the queue coming in. So it's kind of like, well, here's the messages coming in over the web and here are the ones coming over message queuing. But it runs in MicroWhis or G-unicorn or whatever. Yeah, that's nice. Yeah.

Last thing, this is inspired by Pydantic and FastAPI. And so let me see about a good example here. You can do things like declaring that the body of the message is a dictionary. Or you can have Pydantic base models that are coming in. So you can say, here's a Pydantic. When a message comes to the message queue, it's going to be represented by, let's say, JSON.

And that JSON, I want to parse into a Pydantic model. You can just say, much like FastAPI, in your handler, you say body colon, you know, the name of your custom Pydanda class, boom, now it's automatically parsing that based on the type. Oh, based on that's, that's neat. And the last thing, they also have this concept of modeling PyTest fixtures. So you can create functions that will do things like, you know, process requests or give you extra information or whatever, you know or what you would do

with PyTest fixture type things. And you can have those as well in here, which is pretty cool. So there's a lot of cool, it's like a fusion of interesting Python frameworks for message queuing. I like it. So ask your doctor if Propan is right for you.

Ask your doctor, that's right. Yeah, it's interesting because this message queuing type of architecture is super powerful at unlocking tons of interesting asynchrony. Like, well, if I've got a request come in and I got a, you know, place an order and we got to check the warehouse, whether we have them. And that's the jinky old API call. That's slow. Like, well, how do I scale? That would be one option with threads and async in a way. The other one would be just like, threads and async in a way the other one would be just like well throw that into a queue to say check that out and then you know okay let it run completely disassociated right yeah um scroll to the bottom the key features so one of

the things i want right down there testability uh propan allows you to test your app with without external dependencies you do not have to set up a message broker to test. You can have a virtual one. That's pretty cool. Yeah, yeah, this is cool. So it's not super popular, like I said. However, it does look pretty neat. Sure does.

All right, over to you. So that was a little bit of a new thing. I want to talk about a little bit of an old thing, which is Makefiles. We haven't talked about it for a while, but um make files are still fairly popular for python projects i think i i've got them on several uh internal projects at least um and they come in handy you got to be careful that a lot of people sometimes people on your team won't be familiar with them but if it's a common thing for your team to use make files, or for you, why not use

them on a Python project. So this, what I'm going to cover is a, an article, forgetting the author name right now, let's see Ricardo and reg called make file tricks for Python projects. And I'm going to hop down to the actual template, what it is, it's a it's a little, it's a small template as a starter template for a Python project. But it has some pretty cool features. And the actual template's at the bottom of the article, but we kind of go through some of the different things that you might want to put in there. And so to start off some, a little bit,

I always forget to do this. These are things I always want to do, but I forget in my makefiles, things like making sure that it fails if anything throws a incorrect error code. And also warning, if you did something wrong, like undefined variables, or you're using, you can turn off this built-in rules. And I don't really know what the built-in rules thing does. It's just, find my makefiles more pleasant if I disable them. So this is good.

The virtual environment thing, so there's a little snippet that he includes that you can use the py variable to select which Python to run. So if you already have a virtual environment, it uses that, which is cool. If you- Oh, that's pretty clever, yeah.

Yeah, and if you don't, it uses the global one. And then also with pip, so it uses that py variable to pick pip if it's there, or not, and it uses the global one. So that's pretty cool. Actually, it'd probably be better to just blow up if you didn't have a virtual environment. So anyway, some stuff like PWD and

current working directory and work route, these are, these are good things to add in, because sometimes you'll call a make script from a different directory. So your actual current directory is different, and it mucks things up. So there's some good correction there. I do like this, this is, there's some little magic stuff about default goal and help message. And I had to read this a little bit to understand what's going on. But what happens is it the default goal being help means that if you just type make with no arguments, what should it do? And a cool thing to have make do is to print out all the things that you can do with the

make file, like all the all the things that you can do with the make file, like all the all the targets and what they do. And so that's what this does. By having these, having this little greps thing. Is it grep? I don't know if it's anyway, it's searching through your file and using awk and saying, Hey, if you've got a comment against the site of a target that means that's the help message so it'll print that stuff out um oh that's cool yeah uh some i don't really muck with my python path too much but if you have to muck with your python path for make file to to find libraries or something like that um or find the code that you're running. There's examples on how to do that, which is nice. Yes, that's really kind of what I wanted to talk about. And I was

surprised it's doing all this stuff. And it's really and some examples on how you can use the path thing. Oh, having adding a little create virtual environment within a make file. This is nice to just so that people working on the project. Yeah, make.vmv. You could have vmv also. And it just makes your virtual environment. Why do you need a target for that? And it's because, and you've discovered this, but sometimes new Python developers kind of forget,

is that it's kind of annoying to just create a virtual environment. It's good to, after you've created it, update the update setup tools and wheel and build. And then also, if you have a requirements file, why not just install it right away instead of having that as another command? So kind of a fun template for starting Makefiles with Python Project. Mm-hmm.

Yeah, that's, what is that, a modern take on an old idea. Yeah. And if you are new to make files, one of the things to be careful about that some people don't quite sometimes remember is spaces matter within make files, kind of like they do in Python, but spaces and tabs matter. So in in make files, you're using tabs, it has to be a tab. It cannot be a space. Unless something's changed that I don't know about. But that has messed me up before. So use tabs within Makefiles. All right. Yep.

Sounds good. Excellent one. I guess that's everything. Yeah. Yeah. Any extras? No, not really. I got a couple here. Just a couple of conference ones.

So PyCon Portugal has their call for participation. So got a little bit of time left on that. What is that? Till the 30th of June. And when will it be? It will be September 7 to 9, which is cool. So if you're in and around or want to go to Portugal, there you go.

Cool. On the other hand, if you happen to be interested in Django and are in Europe, that just got announced as well. So people can check that out. I want to go. I'm not going, but I want to go. Yeah, indeed. So that's also announced.

Link to both of those in the show notes. Are you ready for a joke? Yeah. want to go yeah indeed so that's also announced link to both of those in the show notes all right you ready for a joke yeah well this becomes because apple.com did you see that they announced this crazy vision thing brian yeah but it doesn't come with the the snorkel it's just the snorkel mask yeah it's just the snorkel mask it's it doesn't even come with that little uh handheld sub submarine thing that you can drag yourself around either. So yeah, so they announced, if you haven't noticed yet, Apple announced Vision Pro, which is a $3,500 ski goggle looking thing that is both augmented reality and virtual

reality, kind of turn the dial. I'm highly suspicious of this. I think it's gonna not do great, but it does look pretty awesome for certain use cases. Like for example, you could sit on the sidelines of a football game and get like a 3D view. So you could look to the right and see down the sideline and then look ahead and watch the game. Like that's pretty epic. Is it worth $3,500?

I don't know, we'll see. But okay, so that's setting the stage for the joke. So here's the game. Like that, that's pretty epic. Is it worth $3,500? I don't know. We'll see. But okay. So that's setting the stage for the joke. So here's the joke. The average pseudo technical person has got like an Oculus Rift in there. They got their handheld controllers that they're doing, right? And then we have the pseudo, the rich pseudo technical people wearing the Apple one sitting there watching TV. And then Brian, you want to describe the actual technical people, advanced high-tech setup they got here? That's just a dude at a desk with like, you know, with using a computer, but it's Oh, there's important stuff to it though. It's dual monitor. We will note the dual monitor

and the mechanical keyboard these this is not your average desk worker yeah anyway there's my follow-on to wwdc i know i would and that that's even okay i've got the the big curved monitor so i don't have two right now but you have two monitors i have one big monitor for my my working desk and i have a big curve monitor for my gaming pc but just one for both as well i used to have dual monitors and i would i was always trying to like juggle them i'm like you know just one big monitor that's better yeah actually when we did the pandemic thing i went to one big monitor at home and then at work i still had the two split ones but then i was just tired of doing this all day long. So I'm like, oh, we got to just, so I went to a big one, but that's kind of privilege speak.

So I don't know. It is a little bit. I just, for people who are interested, I, if I do need a second monitor, like sometimes when I'm recording a course, I want to be able to see what the recording is doing. So I want to see my video overlaid with maybe what's on the screen with whatever settings like scale, like exactly what's being recorded as the person's going to see it in case something goes weird with that. So I'll take my iPad, plug it into my mini and then use duet. Duet is a really cool software that I think works on Mac and windows and basically turns that into

a second monitor just periodically when you want it, you so that's that's what I do if I really feel like I need extra extra space so so go back to the Apple Vision thing or the iDork what's it called Apple Vision yeah I think it's iDork Pro um there's part of one of the the things on there is somebody doing a like a meeting where you can supposedly see other people in the meeting like as if you were still there they were with you or something and i thought well that one go up a little bit yeah this is like the group facetime is what that is yeah except for oh no yeah yeah i see it okay wouldn't wouldn't they see you with the goggles on so if everybody's doing it wouldn't everybody just you just be able to see people with goggles um that's interesting i think it might scan you and put an avatar of you up there oh yeah it's

an ai you it's not really i think it's i think it's an ai you actually yeah i think so i haven't tried this out and where's your? Where do you put your camera? Like for, so anyway. Yeah. There's a lot of interesting and stuff. I'm actually interesting things and stuff. I'm actually excited about announced at WWDC Vision Pro. Yeah.

There's, there's like, for example, large language model dictation for iOS and Mac. So I don't know how many people know who have tried this, but I, for multiple reasons have tried to do dictation on the Mac, partly because I have like mild grade at this point, RSI issues. And so if I can limit typing, that's good. And maybe I've got a lot of stuff I need to blaze through like a bunch of email or something. I'd love to dictate to it. But the dictation system on Mac is like 10 years old or something. It's really bad. You can't even say new paragraph, for example. Like, nope, they don't just write out new paragraph. Whereas on iPhone, you can say new paragraph or do this, or you can navigate around way better. They're not

the same systems. So both of those are being replaced with like chat gpt level of ais and so dictation to your computer or your device is going to get way better and so that means less typing less rsi just different input modalities if you need a break like those kind of things i'm really psyched about vision pro we'll see yeah there's there's potential there but there's also way more potential for jokes yes it's going to be good. Speaking of jokes, we'll wrap it up with one from Kim in the audience. If an avatar is an option, T-Rexes will be meeting with the elves and talking frogs in no time. That would be great.

I'm here for that. Let's do it. Yeah. I'm here for it. All right. Bye, Brian. Bye. Bye, everyone.