Python Bytes - #395 pythont compatible packages

Episode Date: August 6, 2024

Topics covered in this episode: py-free-threading.github.io Python’s Supportive and Welcoming Environment is Tightly Coupled to Its Progress Status pages for sites! PEP 751 – A file format to list Python dependencies for installation reproducibility Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/395

Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode 395, recorded August 6th, and I am Brian Hocken. And I'm Michael Kennedy. And it is way too hot still. I thought we were going to be done with the heat by now, but it's crazy. It is August. This week's episode is sponsored by us, so please check out the offerings we have. We've got courses at both Talk both talk Python and at Python test and we've got Patreon supporters. We don't shout out to them much, but we do appreciate Patreon

supporters. So thank you. If you'd like to connect with us, we're all on Fostedon at M. Kennedy, at Brian Ocken and at Python Bytes. But of course, those links are in the show notes. And if you were listening to us later than Tuesday at Pacific time, 10 o'clock, you can catch us either on podcast, of course, but also you can watch live if you ever want to do that. Usually Tuesdays at 10, 10 a.m. Pacific time. And we appreciate that. And also really appreciate everybody signing up for the show notes email. That's been going really well and it grows every week. And we are going to do a giveaway.

We haven't met our milestone that we're looking for yet, but we would love to give out some free stuff to people. So sign up for that newsletter. Brian, let me motivate some folks. 13. That's how many we need. 13 more and we're doing our giveaway all right okay we just need 13 more people um uh so get on that what do you have for

us first michael well you know what i really like about that giveaway is that it's totally free as in free threaded python let's go so last time we talked about um some of the member we talked about i think was simon willison hopefully the attribution is correct about installing how there's a binary installer for free threaded python yeah right well in there i think he was linking to but certainly i talked to stan siebert from anaconda and that's coming out in a talk Python shortly. And he pointed out pi-free-threading.github.io, which is a website that tracks, gives you some advice on how to create your package or update your package so that it works in pre-threaded Python. But also, most importantly, it has a status tracker for popular packages so you can look and

see are there wheels for free threaded python because apparently did you know that there's special wheels like you could have wheels for linux wheels for mac os intel version wheels for mac os arm version but there's also specific wheels for free-threaded Python. So that's interesting. Maybe you have different compiler flags with, you know, threading primitives built in versus skipped, you know, in that version or something along those lines. Anyway, so it has a compatibility tracker for that. Talks about installing it, which really means installing it on Linux. It's really, I think, all it covers.

But yeah, it gives you some more examples, how to verify that you're running free-threaded Python and so and so on yeah what i'm really excited about here is the last two bullets at the top talking about setting up ci and finding testing and debugging currency issues yeah yeah for sure that didn't really occur to me till you just brought it up that if you have a popular package or even a semi-popular package or one you rely on, you probably want to start testing it to make sure that it's not going to break in free threaded Python. Indeed. Alright, so let's look at the compatibility real quick here. compatibility, it says this page, and you see it's got a big grid of statuses. Basically, it's not

that long. So if you have a a package and you expect it to work with free-threaded Python and stuff, submit it here so we can see its status. Anyway, the page says, this page tracks the status of packages for which we're aware of active work on free-threaded Python support. It contains pages with extension modules as well as build tools and packages that need code specifically to support rethreaded. Now, it also says, note that pure Python code works without changes by design, so it does not track pure Python packages. Oh, okay.

Two things. One, so if you have a pure Python package, don't try to submit it. Hey, here's our status. It's like, well, you already got that, so you can just probably do any for your platform and your wheel, and you're good to go. But there's a difference between executes and works.

In my mind, it says pure Python code works without changes. Pure Python code will run in free-threaded mode without any changes by design. It's kind of like saying it compiles in a static language, so it's fine. It may be built to have a bunch of race conditions, but it doesn't need a special wheel. You know what I mean?

So don't take too much confidence that it's just pure Python, right? Your pure Python code can have five steps, four of which have invalid state. And if those things run in parallel and one of them observes that state while the other one is halfway through its steps, you're going to have a bad time, even if it's pure Python, even though it executes.

So just a bit of a statement, I'm threading there in general, right? But like, if you look through here, there's some interesting things, Brian. So for example, CI build will is tested in general, right? But if you look through here, there's some interesting things, Brian. So for example, CI build will is tested in CI, has a PyPI release so you can pip install it. It talks about which version

first supports it, but it does not have nightly wheels or a nightly wheel link. On the other hand, CMake only has PyPI releases. Contour.py only is tested in CI, but is not available on PyPI. And it talks about which platforms have nightly wheels, and then their links and so on. So you can go through this list.

For example, numpy is tested, but there's no release of numpy at the moment. But there are nightly wheels, I theoretically, I guess you could use if you want to pip install from there. We got pandas, same status, a lot of of them are that's the most common they're tested in ci but they're not yet pushed to pi pi which is fair you know it's there is no release of python that has free threading officially there's only betas right so the fact that they're not just pushing the stuff out to pipey eyes i think uh also interesting notice here brian there's pip even though it's pip here, Python, I don't know, I thought so. But check this out. So if you go over to the installing section,

it says, okay, basically, you can get some binary installers off of pypi.org. Or here's a bunch of talk about how to set it up on Linux, how to set it up on Docker for Linux and build it from source. If you run it, it says, here's how you run with the guild disabled. So if you run with Python dash capital V capital V, you'll see it'll start and say Python, such and such experimental free threading build, right? Sounds lovely and, you know. Yes, it does.

You're living so far in the future, aren't you? And then here you can, there's a little one-liner you can execute to see that it is working yeah talks about a porting extension modules to free threading and here you go brian get up actions let's go it talks about how to set the get up actions and a bunch of code on how to actually test and debug all the things. I love that. How to set up CI. And by that, we mean how to set up GitHub Actions. It's fair enough at this point. It is consolidating a bit, isn't it?

It is, yes. All right. So if you are interested in rethreaded Python, a.k.a. the one that has had the galactomy done to it by Sam Gross and crew, this is a good resource to have around. And if you have a package that you think might need some special treatment to work in that space

and you have intentions to potentially support that, let these folks know about it. You can see their GitHub in the top right. I'm sure you can submit an issue or something there. Yep. Nice. Well, if you need support, transition here, supporting, if you want to support free threading, great. But also we want to be a supportive Python community.

So I did it a nice transition there to Python supportive and welcoming environment is tightly coupled to its progress. This is a post on the Python Software Foundation blog, and it is very nice. I think that everybody should go read this. And one of the reasons that indicate that maybe this is kind of important is this is signed by a whole bunch of people. Deb Nicholson, Don Wages, Tanya, Kwan Han, Kushal, Georgie, Janice, Christian, Christopher, Denny, and Chuk, and Simon Wilson. Lots of signers of this. So what is this? I want to highlight a couple statements here. Python is as popular as it is today because we have gone above and

beyond to make this a welcoming community. Being a friendly and supportive community is part of how we are perceived by a wider world and is integral to the wide popularity of Python. I totally believe this. We even apparently won a wonderfully welcoming award at last year's GitHub Universe. And it's often referred to as like, come for the language, stay for the community sort of a thing. However, with the recent bylaws changes, there was a lot of drama on the mailing list. So much drama that we had, even though there were only a thousand people approximately subscribed to a PSF vote mailing list, 45 of those people dropped out. I think probably because of the toxicity that was going on on that mailing list.

So this is a good reminder that we are a welcoming community and we want it that way. I like another, I'm going to hop down. There's a lot of information in history. If you're curious, I like this statement here. We have a moral imperative as one of the very best places to bring new people into tech and into open source to keep being good at welcoming new people. If we do not rise and continue to rise every day to this task, then we are not fulfilling our own mission to support and facilitate the growth of a diverse

and international community of Python programmers. Goes on to talk about how technical skills are a game changer for people that are trying to better their position, try to get a better job. Python's a great way to get into tech and into programming. And we need to make sure that that stays a great pathway. So kind of a serious topic, but it's serious to me that the future of programming stays with this nice on-ramp of Python. And we maintain this to be a good good place to be. So further continuing some of the consequences of some of the bad actions that happened is that the the form and

the mailing list and some of the other avenues of discussion will be will be or they've turned on moderation, and they're going to continue with moderating it to make sure that there is not toxic behavior going on. And I applaud that I think that's the right thing to continue with moderating it to make sure that there is not toxic behavior going on. And I applaud that. I think that's the right thing to do with something as important as communication from the PSF and Python. So box over. Cool. Thanks for sharing. I do think making it welcoming to new people.

I think one of Python's powers, powers superpowers is really that people come and they don't feel intimidated right they they don't consider themselves programmers and they kind of accidentally slip into like wait i think i just published a package does that make me a programmer you know like six months later uh and having the community being welcome welcoming not just the syntax and the tooling. Yes. It counts, yeah? Yeah, I see a lot of times people will, like a lot of PyCon goers, for example, will bring along one of their kids,

even though they've been going for a while, bring one of their kids, and having the kids be able to go off and learn also and realize that, hey, this is kind of a neat place. We're probably getting software engineers from the community just because of this that maybe would have not considered it. All right, planting the seed there.

So, all right. You've got some good news for us, don't you? I do. Well, last week I have planted a seed and this week it has grown. So I was going to cover a new topic, but I want to have more time to digest it than I've had. So I want to do a follow-up instead, Brian. So remember last week I talked about Uptime Kuma, which is a fancy self-hosted monitoring tool.

And a lot of folks chimed in over on Mastodon saying, you've been using this for years. It's super awesome. And so on. I said, hey, maybe I'll play with this. That would be cool, right? Yeah. Well, guess what? If you go to Python Bytes and you scroll to the bottom, server status right there, and

you click on it, it shows you the Python Bytes operational status. What is the status of the website? And all these little green dots, each one of these are a sample of either up down partially up partially down I guess those are the same thing but so you could see those sort of like over time I wish I could control how much time this is but this is basically just the samples some number of recent samples and I think I've got it set to every minute or two minute it checks and if it goes goes wrong, it'll send me an email or something like that. But you can also see this 100% of time. It's basically since I set it up.

But I think that's 30 days or something. I'm not sure exactly the window of that. But that is more, that tracks more of it. But anyway, we've got the Python bytes web. You can see its certificate expires in 82 days. So that's good. Plenty of time. It's 90 days by default. So quite new. The RSS feed certificate expires in 82 days. So that's good. Plenty of time. It's 90

days by default. So quite new. The RSS feed, what is it? Status. The CDN status that actually delivers the MP3s, which is a whole separate bit of infrastructure at bunny.net and the MongoDB database that powers the website. So you could see, well, maybe the website's down because the database is down or, you know, there's just a problem with the website. Isn't that cool? It is pretty cool. Yeah. And even more involved because of all the course stuff is if you go to the one for the bottom of TalkPython, you can see you've got all those things I described, but also like the mobile APIs, the video CDN, the podcast CDN, the courses website, all those things. And all are making me smile with

almost a hundred percent of time. I think I broke when I set it up. I, uh, didn't configure the, the setup right for RSS. So it's got a little lingering broken bit, but I don't think it's actually had any downtime. Okay. I was good. I actually, I'm, I'm pleased to see that. Cause I was just wondering if you're rounding up to the nearest hundred. No, no. This is like to the, it's to the hundredth. So it's like 99.86% up. But I think it's just, I broke it when I was trying to set this thing up. So yeah, incredibly easy. This is cool.

Yeah. Where's, there's. So if you go here, basically just Docker run. I've talked about this whole Docker cluster. I've had, I've migrated everything to all the infrastructure too. And adding this as a thing that runs over there, incredibly easy. It took probably 20 minutes to get it set up, running as its own bit of infrastructure. And now it just chills in its own Docker container, watching all the other things around the world.

So very cool. Neat. Yeah. So if you want to play with that, you can check that out at the bottom of both of the podcast websites in their footers. Same for Talk By The Training as well. It's in there as well. But basically, you know, just it's just an example of someone putting up uptime Kuma, which so far I'm I'm pretty happy with. And if what you see on the status page is a public thing. But if you log in, you get a lot more detail of like responses response time over time and all this so you can kind of use it as a performance monitoring

thing as well oh cool yeah i should check that out indeed neat all right um well something else to check out is something upcoming and possibly in in python there is a new PEP. Well, there's often new PEPs, but here's PEP 751. This is authored by Brett Cannon and started at the, created at the end of July the 24th. What is this? This is PEP 751, a file format to list Python dependencies for installation reproducibility, which is a bit of a mouthful, but, and I, actually, to be honest, some of the details in here are a little detail-oriented,

but I like to look, the abstract and the motivation make total sense. So the idea is, well, I'll just read it. For the abstract, this PEP proposes a new file format for dependency specification to enable reproducible install, enable reproducible installation in a Python environment. The format is designed to be human readable and machine generated.

Installers consuming the file should be able to evaluate each package in question in isolation with no need for dependency resolution at install time, which is interesting. So what's the idea? I thought we had this already, don't we? We have dependency, but we don't have like this whole like lock file stuff is not a standard. It's just implemented several places. So there's no, under motivation, currently no standard exists to specify what top-level dependencies should be installed

into a Python environment. And no standard exists to create an immutable record, such as a lock file, of which dependencies are installed. There are at least five well-known solutions to the problem. We've got pip freeze we have pip tools uv has a solution poetry and pdm all have a way to deal with lock files in general but we don't have a standard for this so i am uh i guess welcoming the idea that that um that this will be a standard or something similar to it this This is early in the pep lifecycle. But yeah, anyway, since it's a packaging topic, these don't necessarily have to come out at

the same rate as Python. Because pip is not pip and other packaging tools are not on the same year timeframe as as as Python. So not sure if if or when this comes about. Yeah, definitely. This is cool, I actually had this open in another tab, so I can go close that now, thanks, Brian. Because I wanted to read about this, this is exciting. I always love what Brett Cannon's working on. Yeah.

I think, I like the idea of having a lock file that we can use because every tool you pick, it's got a solution, but it's incompatible with all other tools. So some project uses poetry, for example, to manage its dependencies. And you just, you don't want to use that. Yeah. When you, you want to use something else, like I'm a big fan of UV's implementation of pip tools.

Well, that doesn't work with it, or maybe it does, but you know, like you've've got to you've got to get um it depends on how you're using it right uh but having something interoperable between these would be nice to allow you to choose your workflow regardless one of the one of the um aspects in here uh under rationale talked about which i think is pretty cool is that the file format the the the tool that creates the file doesn't necessarily have to be the tool that reads and parses it. The important thing around that is that there might be like, yeah, anybody can create the lock file with whatever tool you want, but a service like, you know, something like Anaconda

or some other service or a packager that pulls everything in, but it might have their own standard way to do all of that. And it doesn't have to be, they don't have to support flit and PDM and all sorts of other tools. I guess flit isn't one of the on the list, but you know what I mean? Yeah, lots of different things. So yeah, it'll be interesting to watch this and see where this goes. And as always, I appreciate how much Brett thinks about packaging. And packaging was, I know that there's that XKCD comic about packaging being horrible in Python,

but it's a hard problem. And I'm glad that we have bright people like Brett and other folks he knows to work on the hard problem. So anyway. Indeed. One thing I hope is that whatever file gets generated out of this that it's not so noisy that diffs of them over time become hard to read for example using the pip tools version if you flip on hashing like the hashes take up so much of this of the text that um i feel like it's just super hard

to read all of a sudden to just like diff a thing and go okay what's the what's new what's what's changed what's the same but it's just ah you know um yeah maybe i should turn word wrap off or something on my editor but it's like you're trying to read okay this thing is installed because these three things depend on it but there's just blocks of hashes all over the screen. You know what I mean? I did notice that that was actually called out in the rationale around it. It said the format is designed so that viewing a diff of the file is easy by centralizing relevant details. Oh, awesome. Maybe it could have all the things and why they're installed there, and then it could have the hashes

in a separate place that you don't have to look at, you know, that don't mix in. Yeah. Cool. Yeah, this is great. I'm glad that that's part of the concern is to make sure. Because, yeah, I might not read maybe theoretically human-readable like XML, but the only time I'm actually reading XML is in a diff. Yeah.

Anyway, cool. Well, that's our items. Do you have any extras for us? I do feel a little extra today. And I hinted at this, some of the stuff I'm gonna cover here. So first of all, MyPy has a new release and there's some interesting things to call out, Brian. MyPy 1.11.

So if you don't know, MyPy is a faux compiler for Python, I will say it's a type checker. But what it does is basically go through your code as if it was a compiler and make sure that everything would hang together. Like here, you say this thing is an integer, and you're passing it to a function that can take an optional integer, right, not an optional string, and it would be an error. And it would tell you that it's not okay, if it weren't right, just like, like I said, like a compiler kind of would. So you can get that.

That's very cool. It has support for the new 3.12 simpler generics, right? Like class C of T, just bracket a T, and then you'd have T goes to et cetera, et cetera. Very nice there. The one I wanted to call out there was functools.partial. You talked about functools, no, iter tools last week. Now how about functools? Have you heard of functools.partial? Yeah, I've used them. Partials are fun. I've never used them. So in Python, we can have default values like F of A and B, F of A comma B, right? You could say B equals one,

so you don't have to specify the value, but it could be another B, right? You could say B equals one, so you don't have to specify the value, but it could be another value, right? It could be, if somebody wanted to, they could override that, right? You can't say, here's a thing with a default value, but never set it. So you can say partial of that function and specify the second parameter or whichever parameter you want, right? And it'll pass that in. But now MyPi checks those. We also get better type inference, better type hinting in expressions, and a bunch of other stuff. So this is a pretty big release.

People can check that out if they want. The thing that I hinted at at the beginning is FastHTML. I know there's been a lot of talk about this, right? Yeah. So fast HTML is a, they describe themselves as a modern web application, building a framework for modern web applications in pure Python.

Think kind of view components a little bit, but a Flask style programming model for the server side with a lot of those components actually written in Python, which is pretty interesting. You can create them, make reusable things. And its interaction and its dynamic aspects are based on HTMX, which is one of my very favorite JavaScript frameworks as well. So if you were thinking of doing stuff like Streamlet or Gradio

or some of these dashboard types of interactions and you're like i need to go beyond what those capable of this and not just put dashboards but more interactive elements and sort of more of a web app but not a ton more then this is a framework that it's upper bound is much higher where you can keep going and going so you can definitely create some some cool apps real quickly with not very much code. So I'm going to play with this a little bit more, and I think I'll talk more about it next week. I'm looking forward to that.

I was going to bring this up as well. Yeah. Looks interesting. We could share it next week if we want. All right. And then also we're coming up on last chance to go be part of the Python and web and database course in a castle in Italy, the Python Zero to Hero 2024 event that I'm doing in a luxurious Tuscan villa.

And hopefully you can be there. But if you've been on the fence and thinking, maybe I want to go, but not sure, I'll find I'm on your to-do list to think more about it. Well, you got to think soon because we're going to close enrollment pretty shortly. So please, if you want to go, jump on that. If not, that's totally good too. And that's it for my extras.

All right. I've got one extra. Pretty much what I've been doing in my free time a lot is working on the new course uh hello pytest and i really want it to be accessible to everybody so um i do have a landing page now it's if you go to courses.pythontest.com um uh you'll you'll see a link to it um there is i i don't know i haven't decided what the final cost pricing structure is going to be, but I'd like to start it at 10 bucks because I'd really like it to be

something that people can take a look at to D. So the idea around it is not to teach you all of PyTest. It is to get people started with it. So to cover some of the basics like parameterization and fixtures and stuff. And the idea being, I would love to have something to hand out to teammates so that everybody can get up to speed. So that the one Python expert on the team doesn't get questions all day long. That's where I'm trying to fill this in. So that's still up.

In order to do a pricing page, I had to pick a date where I was going to deliver it by. I picked the 19th, which is uh what uh 12 days from now i think it's going to be done before then but that's my deadline so working on that awesome congrats you know people if you look around online courses like especially udemy it's an arms race to have the most content we have the super mega course that's 36 hours of content, you know, and it's like, and it's only 50 bucks. That one over there, that's 50 bucks.

It's only five hours. That is nothing compared to this. That's an additional 31 hours of your time you've got to put into it, right? And not everybody has 36 hours of interest in learning PyTest. They just need enough to get going and get started. And so I think there's real value in having a concise thing.

Yeah, well, I was surprised. I was talking to a team that was considering my larger course, and they decided not to because it was because of them. They were like, oh, I really want somebody to be able to do this in half a day. Yeah, exactly. Or less. And I'm like, oh, okay,'s there's real value in half a day or less courses so i think i agree i don't want to spend 36 hours learning anything i mean i mean over the course of the history of working with a tool sure but over like right away no i'm not going to do that so yeah cool exactly

all right let's one joke about open source a little bit. Yes. All right. So this this joke is called open exclamation mark, AI exclamation mark. So it has this sort of meme of two people talking to each other. Hi, my name is open AI. So you're open source, right? Right.

No. Yeah, really? Hmm? No. Not really. It should be closed AI. Yeah. At the bottom, it does say, okay, open AI is not open source. The open means that their AI is open to receive our data. Okay. Very well.

Very well. Yeah. Okay. Yeah. Bart agrees with us. Ain't nobody got time for that. For long courses, not open AI. Yeah. All agrees with us. Ain't nobody got time for that. For long courses, not open AI. Yeah.

All right. Cool. Well, good again to talk with you as always this week. Thank you everybody for joining in on the live stream and everybody else for listening. And I hope you have a wonderful rest of your summer or next week. So talk to you next week. Indeed. Bye all.