Python Bytes - #407 Back to the future, destination 3.14

Episode Date: October 28, 2024

Topics covered in this episode: Python 3.14.0 alpha 1 is now available uv supports dependency groups dive: A tool for exploring each layer in a docker image pytest-metadata Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/407

Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode 407, recorded October 28, 2024. And I am Brian Ocken. And I'm Michael Kennedy. And this week we're sponsored by ourselves, so please check out our offerings. There's all of the fabulous courses at TalkPythonTraining. And then over at PythonTest.com, we've got the complete PyTest course and also the short course, Hello PyTest. And also thank you to our Patreon supporters who

we don't do a shout out enough to, but thank you very much. We appreciate it. If you'd like to connect with the show, we'd love to hear from you. You can connect with us on Fostodon. We're at mkedney, at Brian Ocken, and at pythonbytes, all at Fostodon. And also those links are in the show notes, as well as in at pythonbytes.fm. You can find out when the next upcoming episode is going to be. And I was going to mention this later. I'll just go ahead and mention it right now. Next week on November 4th, we are going to be recording early.

So it'll be at 730 in the morning instead of our normal time. And that's Pacific time instead of our normal 10 a.m. Pacific because of work for me. Anyway, so that's what's going on next week. But at PythonBytes.fm, you can sign up for our newsletter. You can get the show notes. You can sign up and look to see when the next live recording is. But right now, I'd like to hear from Michael to see what you have to talk about. Shall we teleport, transport to the future?

Brian, I want you to imagine a time when it is not so gloomy and rainy. About 11 months from now, when we have a new version of Python that comes onto the scene, 3.14. And that future can be for you now in a limited sense because 3.14 alpha one is now available. The Pi version of Pi, Thorne. That's right. So that's the first alpha one release. And there will be seven in theory, you know, by plan, at least unless something goes wrong. This is the first of seven alpha releases before we go beta, before we go release candidate, and then we finally ship, right? So it has some of the new ideas,

but not all of the new ideas. It comes with performance improvements and actually quite a bit more, which is pretty interesting. So if you read the blog post announcement, well, it looks like, okay, so we've got this new PEP called Deferred Evaluation of Annotations.

Now, I think this was meant to land in 3.11 or 3.12, something like that. And all of the frameworks and libraries that thought typing should matter. Like FastAPI. Yes, and Typer and Pydantic and so on. And I count myself as a big fan of all of those libraries. All those people are like, hey, hold on. This is not just a thing for MyPy. These types are not just for MyPy.

And if you take away their evaluation, all of our awesome frameworks go away. And I think that would be a massive step back for Python. A lot of people agreed, made some changes. So that wasn't done as far, at least in a limited way. But having these type annotations does have a performance cost of, say,

an import potentially. It does have weird issues where you've got indirect or sort of half-defined types that you want to talk about. I have a function or a method on a class, and I want to say it returns one of those classes. Or, you know, it's like an equals. It takes another one of these classes and it tells you if it's equal to it. So you had to do weird tricks like typing dot self, or you could have quotes the name of a type that's not defined yet, but eventually someday it will be defined if you get around to looking for it. So this will allow you to not worry about those types of things. And there's a new API. If you consume them, if you were Pydantic or FastAPI or whatever, there's a new way to say, let me read the annotation metadata about this. That's slightly different, but it's still there.

Improved error messages. We all love improving further improved error messages. So there's that. And then, so that sounds like when you read it, like, okay, that's the thing. But actually, if you go and look, where's the link? The release notes are in here somewhere. So you click here

and you, I don't know why there's a release notes. Really? Where did it go? Maybe I haven't in notes, but anyway, Brian, there's a ton of changes, a ton of changes from there. I think, oh, I know where it is. They don't call it release notes. Here's how you do it. You click on the pep and then you scroll down until you get past the PEP and it just has other changes. So yeah. So there's, if you go to either of those links, it'll tell you about their message, tell you about the PEP, but then there's, if you look at the scroll bar, a bunch of stuff below that. For example, incorrect usage. Yeah. I think that's actually the news, not the stuff that was called out. Incorrect usage of await and

asynchronous comprehensions is now detected. I don't know exactly what you could have done wrong before, but if you're doing it wrong, I guess anyway, if you're doing it wrong. Well, apparently like assert await one. Yeah, so I think what the issue is you could await a thing that is not awaitable

potentially in the list comprehension. So now you'll get a warning. Okay. Is it warning or error? Syntax error. Yeah, syntax error. But you can disable the syntax error because apparently it used to work. If you want to just keep doing that, that's fine.

Let's see, some stuff about dunder debug. The numerical things now have a float.fromNumber and complex.fromNumber, which converts to a float type or a complex type, which is somewhat interesting. So the thing is, this is not a parsing operation. It takes one number and converts it. And if you give it a string, it'll be an error. So this is like,

I just want to make sure I'm always working in numbers and convert them in probably a more efficient way. That's cool. Oh, better than like a cast thing. Yeah, yeah. Where the cast is just a type annotation type of deal, whereas this is a true transformation, right? So we get the new annotation lib,

which provides those features that I talked about for PEP 749, the deferred type annotations. ARG PARs get some updates. AST, concurrent features, C-types, decimal, daytime, disks, fractions, functuals, HTTP. It's funny. You know, HTTP, this is something nice.

You can say python-mhtp.server, I think is it? Whatever. So if you have just a directory and you're like, I need to have this as a web view. For example, if you have an HTML file there

and it says, I want to open a forward slash some CSS file or forward slash some JavaScript file or something along those lines. If you just open it, it goes, I don't know what these files are. Right. But if this web server were to serve it, it would easily work. So you could just type Python dash M space ACP dot server enter and it'll pull up a little web like no no off web server that you can then actually interact with that stuff with. So it now has a dark mode.

OK. Oh, you should have led with that stuff with. So now it has a dark mode. Okay. Oh, you should have led with that, man. That's cool. Yeah, this is contributed by Yorick Hansen. And I thought, okay, Yorick must be a huge fan of dark mode to want. As am I. Yes. Right on.

Okay. JSON. Get some updates. Operator ops. Pathlib. Pathlib actually, the changes to Pathlib are interesting. I'm going through this list to point out there's a lot of changes just in this Alpha 1 release.

But Pathlib now has add methods, like new functionality to Pathlib to recursively copy and move files and directories. Oh yes. I'm here for this. Copy. Copy is a file or directory tree to a destination. Copy into. Copy is

into a destination directory. Move and then move into. That's awesome. BarneyG to a destination. Copy into. Copies into a destination directory. Move and then move into. That's awesome. That's good. So Barney Gale did that. Thank you. PDB, Pickle, Pydoc, SimTable, Sys, and Unit Test. It ain't dead.

No, it ain't. Anyway. In space. So there's also optimizations. For example, AsyncIO is now implemented using doubly linked list implementations. Hat tip for the comp sci data structures level up there. For native tasks, which speeds up execution by 10% on standard high performance benchmarks and reduces memory usage.

So who wouldn't want that? That sounds great. Exactly. All right. There's more. You can read about it. And I just touched on it. Like each thing I said had a paragraph or list of bullet points you could go into if

you wanted. So that's it. Now back to the said had a paragraph or list of bullet points you could go into if you wanted. So that's it now back to the future, back to the present, whatever, back to their current time. Yeah. Or the fast time. So just last week on October 21st,

episode 406, we talked about, we talked about a lot of stuff, but one of the things was PEP 735 with dependency groups in PyProject.toml. And I didn't know when we could play with them and the answer is now because UV decided to do that right away. So UV just put a change in in version 0.4.27

in the changelog they say the dependency groups are supported, which is super cool. There's even support for PEP 735. I couldn't believe this was so fast. It was like just a few days after we talked about it. They must listen to the podcast.

That's pretty awesome. Yeah, probably. And this was like, oh yeah, three days ago, it was merged in. So this is pretty cool. So there's a whole bunch of added stuff. And so I went off and played with it. So there's,

there's a dash dash group is that added to UV add and UV remove. So the, when you add dependencies or remove dependencies, you can say group and it puts them in groups, the dash dash dev and things like that. There's a couple of those, those. So dash dash dev used to be an extra dependency.

Now it's a dependency group called dev. And and so dash dash dev is the same as saying dash dash group dev. It's more clear in the in the list in the documentation if the documentation is updated. So there's a couple of things where it isn't quite updated. So there's a lot of it was very clear on how to add dependencies to dependency groups with UV. Now you can say UV add or UV remove that that works great. But how do you install them then? So that's what I tried to play with.

So also the documentation for default groups is up so you can uh click around in the uv talks about uh default groups and dependency groups um and here we have like the dev for pytest of course you're gonna have pytest in your dev dependencies um and there's the uh there's the little tip for the the dash dev flags okay so i'm like okay so how do I install stuff? So I expected, what I expected to have work is to say UV pip install dash dash group dev and be able to just install those and not the project. Well, uh, that doesn't work yet. So, um, there is a issue, um,

filed by Hinnick and the, uh, of this, this should work and it doesn't. Pip install group. But the issue is that they want to they want to be like the same compatibility with the interface for pip. So they're just waiting for a pipe to define what that interface is. And that's fair. So so we've got all these dependencies and you can't install stuff, right? No, not quite so i asked also asked on linkedin and if anybody knew and uh uh tushar uh responded saying hey the docs

which they don't but it says uv sync group will work so i checked out the docs and i can't find it the sync doesn't talk about groups but it it does work. I tried it. You could, so essentially this works. So what do you do? You create, you have a, you say you added a dependency group, then you can say UV sync. And if you add the group, like dash dash group,

and then the group name, and you can add a bunch of those, then your virtual environment will sync to match what your project says. So basically that's how you can install. You can use sync to install your stuff. Long-winded, but that's how you can do it. Nice. Oh, that's awesome. Anyway. Yeah. So this lets you say, unlike a requirements.dev,

it lets you say, here's a separate set of dependencies that I need for a certain action or set of tasks on my project. It doesn't necessarily pull in the base dependencies, right? That's the big difference. Well, that's the idea. But with UV, it's different. So with UV currently, with utilizing sync, it will pull in your base dependencies and the extras. You can't do just the group yet.

Got it. Because it's not defined. It's not agreed upon yet. How to say it. Right. So the syntax is there, though. So you can play with syntax. And it's not defined it's not agreed upon yet how to say it right so um the syntax is there though so you can play with the syntax and i'm uh it's very comfortable i'm actually i probably there would be very seldom like for me i'm okay with this being having to

install the project and then the dependencies but there's there's there are times where you just want to install a group like to build the documentation or something like that. Yeah, I'm fine with it as well. You know, one that might be on most people wouldn't guess for, but in the ML space, some of these dependencies are massive. For example, if you want to use the large English model for Spacey, you specify that as, specify as a dependency. You don't Python, maybe you run it as a Python call.

Anyway, that thing is like 500 megs or something. All right, so you might want to just not download and install that as well. Right, especially if you have like a modular CI system where one of the build stages is like building documentation or linting, static linting or something like that. You might not want to pull in the world just to do one of your stages.

Yeah, that's a great case. Okay, let's dive into the next thing, huh? Okay. Dive comes to us unknowingly by Mike Fiedler. Let's see. Yeah, Mike Fiedler. So he mentioned this in a group discussion that I was also looking at.

I don't know that he sent it in specifically for us, but here it is. I'm taking it because it's a cool idea. So here's the thing. If you have some sort of container system like a docker image you have a bunch of steps and when you apply the way you make the containers as you go to a docker file and you put in lines like do this action then this action then this action and they can be they're usually some kind of linux command or maybe you'll copy these files or run this Python command once you've copied the right stuff over. And so they can be big, like apt install

this one thing, but this one thing has 20 dependencies. Or you might be running install the development requirements or a compiler that you didn't need. So you might end up with slow builds and large images. What do you do about that? You run this dive thing on it. So this thing's pretty cool. So what it does is there's a little animated graphic you can have here if you go to the GitHub page and, you know, hat tip for anything UI, please animated GIF would be awesome, wouldn't it?

So anyway, you can have this here. And the idea is it basically allows you to see every layer so each line of code that you've run like mega directory copy these files uv pip install dash r requirements or something along those lines and each line you click on it gives you all the files like a tree that you can explore with the file size of the of the image the image, and then also kind of a delta. So as you click around, it'll say these files were added, these files were changed, these files were deleted on this step. So you can see this is the reason this file is here

because on line four, we ran this command, which added these files. Why is that? Oh, that's cool. Isn't it, right? Yeah. So if you were shipping containers around, shipping images around you know like

pushing it to docker hub and then pulling it somewhere else like you should definitely look at this it's not something i'll probably care about because i just build my containers on the server then use them which is also you know a way to do it but uh if you ship them around and you care about their size or they're building slow or whatever this is awesome check this out also for security concerns to make sure things don't get added that you don't know about right or even just i i want to make sure that there's no compilers on here because if somebody were to break into the container and they could use the compiler then they could say upload some c++ code compile it and run it yeah right you know the so-called living off the land sort of issues after somebody breaks into a thing so you just say dive space your image or you can alias it to this is a this is pretty nice

actually. So there's something I've started doing with some of the tools if I'm already on a system that has Docker. So you can do this with glances. You can do this with dive. You can do this with other things as one way to do this is I could brew install dive onto my Mac. But then who knows what that's doing to my computer. I'm running arbitrary code off the internet. I probably trust it, but you know. Right? So what you can do is you can alias, so you can run this from Docker, and then you can alias

dive to just be the sequence of Docker commands that you want to run. So even dive itself runs in Docker, and then it talks to the other Docker container that you pointed at, and that's it. Which is a really nice pattern that's getting to be more popular. It's Docker inception. It's Docker's all the way down.

But yeah, you just create an alias. And then it's just Docker run dash IT, remove the image, map the volume it needs over, and then off it goes. Yeah, it's really cool. All right, what else? Real quick. I'll wrap this up. So it'll show your image broken down by layer, like I said. What's changed?

It has a metric, like cyclomatic complexity equivalent. But for Docker, they made this up, I believe. An estimated image efficiency. The lower left pane of the base layer shows an experimental metric that'll guess how much wasted space your image contains. That's kind of interesting. You could do a quick build test. So instead of going and building the thing

and then running dive, you can say dive build and it'll actually build the thing and then test it instantly. So it's like one line. And then there's CI integration for basically doing some of the stuff in CI and ways in which you can make it fail the CI

if it fails some metrics and so on. Oh, that's cool. Yeah, Pat Decker out there says, I've been using dive for a while. Neat to hear the testimony yeah so yeah people can check this out obviously it's it's open source written and go 46 000 github stars so it's um pretty well known and used that's pretty nice also i like the uh this ci integration because if i can't get it from to a feature from the command line it doesn doesn't exist. That was my concern. Anyway, cool.

Nice. I've got just, I guess, a little blast. We talked about the future. Let's talk about the past a little bit. I had a question. I haven't talked about PyTest metadata for a while, but I had a question just this last week. Somebody got a hold of me and said, hey, I've got these PyTest CI system put together. I'd like to send some extra data from the test environment to the report server. Is there a way to get metadata, like extra metadata in PyTest to add it to the results and then put it somewhere else in the result system. And I said, well,

have you checked out the plugin called PyTest Metadata? It does exactly that. So I guess I'm just reminding everybody that PyTest Metadata is a really cool plugin. It does exactly what you would think it would do. It bundles metadata with your results, or at least that's exactly what I think it would do. It isn't testing your metadata it's adding metadata to the to the results so some of the things it does right off the bat which is kind of fun is it automatically stuffs in uh four did i count them right yep four different keys it's key value pairs for the metadata it does the python version um the platform so python version is like you know whatever python, the platform is a string that

we get out from Python to say what kind of like, you know, for Mac OS, it's Darwin or something else. And then the packages, it lists all the PyTest, all the PyTest packages that are there. And then actually it says the description PyTest packages. I think it's, I'm going to have to check this out. I'm not sure if it's all packages that are available or just the ones associated with it. Yeah, the name says PyTest, but the example lists more than PyTest. Yeah, so I'm not sure.

But then there's plugins and it lists all the PyTest plugins that are there that you have bundled. But I don't really use it for that. I mean, that's interesting, but really I add stuff. So we've got like my, I'm often the target environment that I'm testing, what version is that? What extra packages are on that? So if I'm testing embedded systems, so I'm checking information about the system I'm testing

and adding that to the metadata frequently, but you can do whatever you want. But this adding is cool and it's really easy to do you can add it during um you can even add it at the beginning um like on command line if you know it at command line but within the test you can do it within within a fixture within a test as well you can add metadata and it's reported it's reported in the command line if you do the dash dash verbose that gets or dash B will add like reports all the metadata. But it also comes out in the JUnit XML and a lot of JSON results and other plugins will pull this metadata out also because it's a very widely used plugin. So very useful if you want to pass that along. It was, I believe it was generated

specifically for the PyTest HTML plugin because it's the same contributor, Dave Hunt and others, but the metadata is used even without the HTML. In our top PyTest plugins list, the metadata is number five and HTML is number six. So they're both very popular, but metadata is popular by itself. Yeah, very cool. I love it.

That's a great idea. Anyway, so those are our items. I have a couple extras or one extra. Do you have any extras? I have them. Okay. I got a few, but they're quick as well. It makes sense to start right here

because I did make some changes to the plugin list. So just the other day, I was looking through the top PyTest plugins list over on pythontest.com and noticed that there was stuff in there that I don't like talking about because they're deprecated and even the developers don't want you to use them.

So I've started to start filtering those. So I'm sorry, I'm going to go through the list and check out to see if they're deprecated ones, because you can already find this information out. If you want to find out with the, with the list of what, which ones I'm pulling out,

but why would you want to do that? You want, we want to know the top pieest plugins that I should look at. And if there's ones you shouldn't, don't. So yeah, those are coming out. And then I thought, you know, I've been wondering to do a new series on the Testing Code podcast.

And I thought, why not look at these plugins? So I'm not going to go through the entire list, plus it changes like every month a little bit, but I'm going to go through the entire list. Plus it changes like every month a little bit, but I'm going to go through some of these and pull out some that I think people should know about. So I'm going to do like a series of plugin episodes. So that'll be coming up. How about you? Any extras? A couple. Let's see what we got. So let's start here so Hugo van commande said guess what pillow you know the Python image library 11.0.0 is out with not just support for Python 313 but free threaded Python wheels so oh no I doing about a bunch of image processing you want to speed that up the free threaded

Python key or whatever it is you run is good drop support for three because that's so old school removed a bunch of stuff and then added many things so if you go look here I believe you look at the look at all the changes in this so pretty pretty awesome how much has changed here so if you do stuff with images and Python use pillow there's a new one that's a big release but if you're like what could I do with images and Python well check out pillow it's a new one that's a big release but if you're like what could i do with images in python well check out pillow it does a lot so that's cool we have uh you know you've heard of pip install anti-gravity you've heard pip install this or sorry import this and import anti-gravity but what about pip install deutschland and what does that do so this comes from grand nap says hello michael and brian here's a recommendation for the 14 of your listeners who are in germany i remember i talked about that it's analytics from imami a

while ago and it says just pip install deutschland for lots of useful german data sets regularly updated and so you go over here and it's got things like do you want to go and find addresses barrier lines building areas building borders i suppose and so on my german construction terms are not that good but anyway something along these lines right uh and you basically it just has data for all these different components or things you might want to know about in Germany, which is pretty cool. Yeah. That's pretty cool.

A lot more than just buildings. There's a, there's quite a bit there, isn't it? Yeah. Yeah. Addresses, um,

history points, underground stuff. Um, it's got Autobahn information. Yeah. Yeah. Yeah. We need a pip install for a lot of countries like pip install USA.

It chants a little USA, USA, then it installs it. All right. America, America, America's got borders. Okay, so another big thing that's fun is I just released a TalkPython blog. So I've got my personal blog and there's tons of RSS at TalkPython like for the podcast and stuff, as you would guess. But a dedicated blog so I can post interesting things and series that I plan on working on that are more TalkPython related, not just for my personal blog. So anyway, it's got a couple

posts already and yeah, I tried to go for the super clean, super readable version here, Brian. So trying to make it just, That's nice. Yeah, thanks. Go ahead. I was going to ask you what it's built with. Well, it's built with Hugo, which is a glorious static site builder. But what's interesting is it's hosted under talkpython.fm slash blog, not like blog.talkpython.fm. And I'm using nginx to point different sub URLs to different sections. So like most of the talk Python is a Python app obviously, but then this part is Hugo and they just kind of coexist under the same URL structure, which is fun. Okay, nice.

Yeah, I might write about that actually at some point, but yeah, it's built with Hugo. Kind of like our own stuff, right? Our own personal things. Yeah, I'm just having some Hugo, I'm having Hugo issues, but. Are you? Well, I'm still having Hugo love, so it's okay.

We'll even it out there. All right. I got a joke for you, but I can't find, this is a cartoon I saw, but I can't find it again. I thought I bookmarked it in our thing that bookmarks it and the title of it is there, but the URL to it is not. It just says dog joke. I'm like, huh, wonder what, where that is.

And I've tried to search for it and I can't find it, but I can tell you the joke. Okay. Okay. Are can't find it. But I can tell you the joke, okay? Okay. Are you ready? Yes. Imagine, if you will, a little girl just getting her first puppy. The dad has brought it home and they're out in the backyard.

And he's about to share some wise advice with his daughter to help her become accustomed to having a puppy and all the new responsibilities and so on, right? Yeah. So she's just sitting there like gleefully petting the new puppy and says, now they're thinking of a name for it, right? He says, now, sweetie, you need to think very carefully about the name for this puppy because you're going to have to remember it for the rest of your life and enter it on every webpage that has a stupid security question. Yeah.

So that's the advice of the dad. Remember this name of your first pet, because you're going to have to remember that for the rest of your life. Yeah. That was a joke. That's good. No picture because I can't find it. I've got a dog joke.

So let's add another joke. Let's do it. Okay. So kind of a dog joke. So a guy walks into a library and he's looking for a book and he's, and he walks up to the librarian and says, there's a particular book I'm looking for, but I can't remember the name of it. It's a book about Pavlov's dog and Schrödinger's cat. And the librarian says, that rings a bell, but I'm not sure whether we have it here or not.

Perfect. Pretty good. A lot of science in there a lot of some social science some hard science i love it yeah yeah all right some uncertainty yeah yeah might be there might be that and it rings a bell um cool well uh awesome episode again um i think so thank you michael yeah and thank you everybody for listening so i guess we'll uh talk next week bye yep bye