Python Bytes - #366 Put It In The Backlog

Episode Date: January 9, 2024

Topics covered in this episode: Python 3.13 gets a JIT UniDep - Unified Conda and Pip Dependency Management Don’t Start Pull Requests from Your Main Branch instld: The simplest package management Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/366

Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode 366, recorded January 9th, 2024. Welcome to the new year. I'm Michael Kennedy. And I'm Brian Hocken. And Brian, this episode is sponsored by us. Support our work by taking our courses, the TalkPython training ones, the complete PyTest course, and as well as through our Patreon supporters. All those links are at the top of your show notes and your podcast player episode page

and so on. Connect with us over on Mastodon. You'll find us all on Fostodon. That's right. Michael, Brian, and the show. It has its own pronouns and gets thrown in the group and it's its own website and all that. So come see a Python By bytes over there as well and if you want to be part of the live stream you're welcome that's

certainly not required to take part and enjoy the show but if you want to be part of the live stream pythonbytes.fm live usually you'll see some scheduled upcoming one you can click you can you know smash the bell crush the subscribe all the things they say over there yeah brian it's been a while welcome back to the show it has been a while. Welcome back to the show. It has been a while. I'm excited to be back. I am too. Thanks, everyone, for being patient with us while we took a little break over the winter

break. It's always a little bit tricky to juggle that kind of stuff, especially, Brian, when you're on a beach and it's 80 degrees Fahrenheit or 20 degrees Celsius, 22, something like that. It was, it was a windy day. It was, there was one day it was windy.

It was, it was rough. Uh, I'm really sorry about that. Thank you. Thank you. I appreciate your. Yeah.

Brian, can I, can we maybe not skip this episode because I'm in Hawaii hanging out on the beach, just sipping drinks. Yeah, no, no, no, it was lovely. It's good to get away, but it's also really, sipping drinks. Yeah, no problem.

Yeah, no, it was lovely. It's good to get away, but it's also really good to be back. So, yeah. And I missed you. Yeah, I missed you too. And everyone missed the audience. Shall we jit it off for 2024? Let's jit it.

Let's jit started. Let's jit started. Okay. So, Anthony Shaw, so one of the things that's possibly hopefully coming into python 13 is python 3 13 sorry we're not all the way up to python 13 yet we're still in 3 3.13 is a jit compiler it's um and uh and i'm i'm with because of like you know java and other languages i'm familiar with jit compil, but Python's is going to be a little different. So Anthony Shaw was kind enough to write up a article kind of walking through it. And it's a really nice.

There's a couple things that I recommend reading it if if you don't really know what a JIT compiler is, or if like me, you didn't know what a copy and paste JIT compiler or JIT, whatever implementation is. So that's kind of where he goes with this. And he, um, he starts off talking about really what a, what a JIT is. And really it's, um, it's something that basically generates your, uh, your semi your compiled code for a lot of times it's compiled code machine code. Um, but for Python, it's going to be, uh, the, not the compiled code, for a lot of times it's compiled code, machine code. But for Python, it's going to be not the compiled code, but the bytecode. So getting the bytecode ready, I think. I may have gotten this wrong. But he walks through kind of what a compiler is or a JIT compiler is as if you were writing

it in Python. So it goes through a little example of making your own Python version of the C Python interpreter, which is actually written in C. But if you were to write in Python, it might look like this, like switching on opcode of load construct, load const, and things like that. And you can see what those primitives

are by using the dis module for disassembler. And, and that's fun. The and then he walks through kind of, okay, so that's what a what a JIT compiler might be. What is a copy and patch JIT compiler, not copy and paste copy and patch. And, and that's a little a little bit more harder to describe over audio but he uh but but anthony does a really good job describing it and um and then talking about really this thing it instead of like copying a piece and then running it it copies like more than one bit like the entire function and then running the whole thing and it's a little bit faster and and then we're going to do something like that it's going to be something like that for python inside of the python c

interpreter and how that's it's just a good walkthrough to kind of understand what is going on what are these people doing and why and the why is it's going to be a couple of things. It's going to be faster. And in his, there's some benchmarks that show it's from like two to 9% faster, which is great. I mean, actually anytime you can make things a little faster, it's good. But,

but is it worth it for this? And it's, it's worth it for this partly because mostly it doesn't make it slower, which is the good part because, because of this JIT compiler, there's other tricks that they can do down the line to make things a lot faster. So this is great news and a good explanation. Yeah, it's super exciting. And you're on top of the news today, Brian, like this came

out this morning. So well, well done. This is brand new from Anthony Shaw. Anthony, excellent work writing this up and all the things. So one part that's interesting is 2% to 9% is great. Like, that's still really good to make things faster. But he also points out that this is just the foundation. Like, once you have a JIT, there's all kinds of interesting things that JITs can do on a per hardware architecture per platform story, right? They can say, well, you wrote this code, but I know we have these specific machine instructions,

specialized machines instructions on this CPU. So let's make it do that. Or I see that you could actually inline this function. Instead of just make it run faster, we could inline it over here and here and here, and then, you know, function calls in Python are expensive. So maybe they just become not function calls, right? There's a lot of possibility for where things go. It would be interesting, I wonder, if you could even do things like, you know, compile stuff to C and use C optimizations as part of the JIT compiler, right? Like maybe there's a bunch of like layers that could happen.

I don't know. But yeah, basically this is, you know, a foot in the door for compiler optimizations that we've not had before in Python. Yeah. Yeah, it's pretty exciting. So, yep. Excellent. Excellent.

So the future is bright. Also compounding is interesting, right? All right. We're supposed to get Python much, much faster by making it like a little a little bit faster over five years continuously right and this is just part of that right so you keep adding these things up and the python i don't know 3 14 15 versus 3 10 when this started could be super super different yeah all right let's i hope people like deploying packaging bundling talking about

managing your projects because that's all I got this week, Brian. I got one after another. So let's start out with this Mastodon post by Bass Niholt. And Bass says, we're launching Unidep, which is a unified conda and PIP dependency management system. Okay, so we've got the conda world, we've got the PIP world. Sometimes they kind of work together, but they're pulling from different sources. So you can like in a condo virtual environment you could pip install a thing i believe and it would still install but here's the deal like you can create a single requirements.yaml file and say i depend on these different projects those two come from conda conda forge or something these three come from pip ipi interesting huh i'm actually

excited about this this is very interesting yeah it's super interesting it works with pyproject.toml and setup.py it also is good for mono repos so why is this good for mono repos mono repos are i don't want to have to switch projects on github i just want one ginormous one for a whole company so let's just have one, right? Which is kind of an insane way of working, but it's also pretty interesting, right? Like I've got two libraries, they depend on each other. It's just all on the same project structure. So you just version them together, but that's literally every project for your company. And there's different teams potentially working with different versions and with different things. So maybe there's some data science folks and there's some web API folks.

And the web API folks are maybe PIP and the data science folks are Conda. So here you could express all of the requirements for the different projects through these different sources, right? Because some might be Conda, some might be PIP, but here's one way to express it. Also, it'll create consistent Ca lock files for multiple projects. It has platform support and you can just unidep install if you want. It's pretty cool. So looking over at the GitHub here, yeah, pretty much, honestly, pretty much says the same thing. But it shows you some examples of, you know, how you might express, you know, what is,

what is a conda dependency? What is a pip dependency? So like in your requirements.eml, you just have dependencies. Let's just say numpy, or you'd say conda colon Python graph is, or you'd say pip colon graph is if you wanted that one and so on, right?

And look at this. You can even include other you have like nested requirements.emo which maybe talks to the mono repo type of thing or multiple projects just in one repository right like this thing actually requires these four other things this this is even interesting you know we were talking about uh courses and i know you're going to mention something about your course later this is actually would be really nice for courses as well to do i suspect you probably could do with pip. I just, it never crossed my mind, right? I might have chapter one, let's do fast API stuff.

Chapter two, let's add Beanie and MongoDB. And like those have different requirements files. And you might want to just be able to express like, well, just run this one, just run this thing. You have everything you need for the course, but you also want to not like talk about everything before you get to it. Right. You could have like one at the top of your course repository and then every chapter have its own one of these and then just like pull them together with this, like this. Oh, interesting.

That's cool. Right. Yeah. I've never thought to do that. I haven't either, but it seems like I should, I've just been copying that stuff around. It seems like I shouldn't do that really. Awesome though.

Yeah, so Unidep, it's brand new. I don't know if this thing's gonna take off and be super, super popular or if it's gonna be kind of niche, but it seems like it's solving a unique problem. And it really is kind of a time of like a thousand flowers blooming in the packaging and dependency managing space.

Like previously we talked about Hatch. I have more to say about Hatch later as well. MARK MANDELBAUM CHEYENNEY- Do you remember, did you say that you can use a pyproject.toml instead of a YAML? MARK MIRCHANDANI- Yes, you can. MARK MIRCHANDANI- OK. MARK MIRCHANDANI- Yes. It's a da, da, da, da, da.

MARK MIRCHANDANI- Because there's already enough YAML in my life. MARK MIRCHANDANI- Yeah. Yeah, you can use a pyproject.toml. MARK MIRCHANDANI- Cool. MARK MIRCHANDANI- Alternatively, one can configure the dependencies in the pyproject. all under the tool dot uni depth section. All right. Nice. There you go. Yeah. All right. Well, over to you. work over the holiday break since we also took a break from podcasting a bit i've been doing some open source work and i ran into this is timely because i ran into uh some problems with um with

pull requests and wanting changes changes in pull requests and having some of the problems with if somebody does a pull request and their fork is on main instead of on a branch. And so, and it causes problems. And so instead of having to write this up as to why you should not do this, Hinnick beat me to the punch and wrote this great article saying, don't start pull requests from your main branch. And it's tempting, right? When I first started using Git at work, even at work,

the so personally, I when I was started using Git, I just use main and, and just checked, I just had one branch. And that's no big deal with my own stuff. With a with a team, we had to decide where we're going to use branches or forks, and possibly not both, but you can use both. And in open source, it's common to use both. And what we mean is you fork the repository and you create a branch. Why do you do both? Because it seems like if you're going to make a change, you just need one or the other, right? And you can't do a branch because you don't have permission. So you do a fork, but you do both a fork and a branch for reasons. And the reasons are spelled out here. First, if you, if you didn't do that, he's, he's giving the reasons in the sense of if you did

main instead, why that's bad is first, you can only do one pull request at a time. If you were going to do your pull request off of your main branch and it's not been accepted yet, well, you can't go and do another pull request for a different change. You can't do another fix because you've already used up your main. Whereas if you do a branch off of that, you can have as many fixes as you want. So that's reason number one. But people might think, I don't care about multiple stuff. I'm just fixing the one thing I care about. Great. If it takes a while, so somebody on the other end is going to have to pull, like, review it, maybe ask you for changes to maybe merge it even. That's great. end up merging your pull request that's awesome but the what happens then is it'll that branch

will have changes of both the uh both the main line and your branch all in this branch your main branch now that has conflicts and you're not able to pull it back to your local repository so pulls won't work and you just you i don't know what you do at that point you blow away your fork and start over delete you delete your about story you start over yeah uh lame um okay the third reason is um is the pragmatic one of getting something done and this is the one that bit me uh a couple weeks ago is um there's two reasons you're um uh i may i might be like messing this up but um you can if you have it on main it's and you have branch protection on which a lot of repos do most of most a lot of big open source projects the main is your lockdown um then uh and you don't want you want allow it's allow edits from maintainers is meaningless at that point.

Because what I want to do is if there's, if somebody does a pull request and I want to accept it, but there's a minor change to it that I want to like tweak it. Like maybe the, I don't know, the naming convention's wrong and it's not a big deal. I can just change it before I merge the pull request. You can't. If that's locked down, the original, the person pulling into their repository can't do that edit. And it's just not nice and it's not fun. So just remember to do fork and branch, don't do just fork. So that's the public service announcement for today.

Yeah, I mean, fork is required because a lot of times you don't have right access to the main repo contributor access. But doing it from your main, doing it from like some other, the main branch instead of some other one is, yeah, you only learn that that's a bad idea when you do it. And then you're like, oh, no, now what? I've done that once or twice. I'm like, I'm going to make a change and do a pull request and submit it. And what if they don't accept it? Then your main is out of sync forever. You can never sync your fork up again.

And like, there's this all sorts of, and even if there's no ugliness there, let's say there's, and this has happened to me many times. You, and somebody wants to, to merge your stuff in,

but they're merging other things too. And maybe they grabbed a few other things first now they they really would like to have the merge from yours be really clean and just see the changes so the great thing to do is to merge your the new main to your new main and then resolve it with your forked brand your branch first before going all the way back and if you don't have that it just doesn't work. Yeah, that's a good point too. Indeed. Indeed. All right. Let's get back to deployment. Okay. This one's different. Okay. Installed. This is kind of like a little bit of a web 2.0 name installed, but zero vowels installed in S I N S T L D. So this is an interesting project. When I first saw it, I was like, this seems like a bad idea. I don't know. This seems like just a bad idea. Let's not

do this. And then people kept recommending it and saying stuff like, you know what, actually, that's kind of cool, actually. So here's the deal. Previously, what we talked about is like the right way, right? You're going to create your requirements, your dependencies. You're going to have pin versions. You're going to have lock files. Someone is going to create a virtual environment or a Docker container or something like that. Then you're going to install all of those things, right? Right.

What if you don't? What if you have, Brian, what if you have a single dot py file you want to give to somebody and you want to just let them run it, but you know, it needs like one or two dependencies. Like maybe you really, really, really want to use rich because Will's project is awesome. And you just don't want it to look plain white, unformatted. You want it to look good, right? So you want to be able to handle one of your coworkers or somebody else, one of these things and say, just run it. Well, just run it becomes all of a sudden a conversation about virtual environments and all these things, right? And like, ah, geez, just, just want to format it just a little. Could we somehow, well, installed is

kind of aiming to solve that problem, right? So it's not like a high-end production sort of thing. It's a, let me be able to pass a file around without going through the whole process of you know like pi installer because like well oh geez i made it exe for you that's right you're on windows let me recompile it like okay great no so here's the deal it lets you run python code without installing the dependencies without mentioning the dependencies, without having any awareness said dependencies so exist. With one huge caveat, and that's you must have installed,

installed, OK? Once this thing isn't there, it can then bootstrap everything else. So maybe if this was a thing you were doing with your co-worker story, I said, OK,, well everyone pip install dash dash user installed or pipx installed or I don't know, something like that. Right. But once that's there, then you can run and use arbitrary things from PyPI without having them installed or requirements file or going through a step. You just Python space your thing off it goes. Also, if you were in a really weird situation,

but you had to do it, you could use two versions of the same library package within a single execution of a program. Okay, interesting. Which you might do that because you have incompatible libraries. Like I've talked before about, there are times where I literally cannot

pip install dash R, my requirements, right? It's just like, nope, this old janky version of some library depends, it's pinned less than something else, some other library, and then like another library is greater than that same library. Well, modern pip just says,

guess we can't run your program, take a hike, right? So this would let you sort of say, this part of the code uses the old janky thing let's use it and then get rid of it and then go back and use another all right it also doesn't leave behind junk on your hard drive it basically deletes everything so how does this thing work so basically in your code you can say you have to import installed and then you use a context manager you can say with installed some package, and then you can, like within that context manager, you have access to that. Pretty wild, huh? Okay. So basically that context manager will make sure that it's installed on your system. It'll download it, all those

things. And then like when, presumably when it closes, it'll delete it. You can also do it as a REPL. So you can, you know, interact with just like the Python REPL, you can interact with it that way. You can run a script, you can say installed some script, and then it'll be executed as if you were running it like regular Python, but it'll, you can also pass in dependencies that it might need to run it, right? If you're, check this out, if your program has imports of any packages other than the built-in ones, they be installed automatically oh right so you just say installed some python file and it goes oh you're saying import this stuff you import you know httpx hopefully you got the names right yeah exactly so yeah anyway uh you can do a bunch of bunch of more advanced

stuff you can talk about there you can pin the versions you can specify um which ipi which package repository it comes from if you like a the example here is a test one but if you've got like a private whitelist server and so on so there's a lot going on here i don't know how does this how does this hit you seeing this you know the w i was i was thinking i don't need it up until just a second ago. I know. test against multiple versions of PyTest, for instance, or multiple versions of something. Like in my own stuff, I'm testing against maybe a few different versions of a particular package.

And I do that usually outside in talks to be able to install a different environment, set up a different environment with those different things installed. But to be able to do it all in one run, that's pretty interesting actually. It all in one run, that's pretty interesting, actually. It is interesting, yeah.

Yeah, so I might play with that. It's pretty interesting. Apparently when it runs, it spits out basically the pip output. You know what I mean? It comes, but in the context manager, you can say catch output equals true, and then it's kind of invisible to your users.

It just takes a little startup time as it does the dependency stuff, you know? yeah yeah pretty wild so when i first saw this i thought ah this seems like it's encouraging people to do the wrong thing but it's it's also some interesting flexibility so there you have it and sold and sometimes people need to be encouraged to do the wrong thing that's right how else do you learn to do the right thing? No, just kidding. All right. What do you got for our extras? I got a couple extras. I am super excited. The other thing, one of the things I was doing is open source code this break, but I also was

completing the complete PyTest course. So I named it the complete PyTest course before it was complete. Now it actually is complete all 16 chapters. However, it's not, it's never really going to be done because this isn't a printed book. It is a, is a project. It's it's a course that's up on the web and I'm going to update it if I need to. So I've got a good community going with it already. And we're going to make sure that it's up to date with new versions of PyTest. Like PyTest 8 is coming out. And I don't think that I'll have to change anything.

But if I do have to change stuff, I'll change it here first. Yeah, you can just do a find and replace for a 7 to 8 in your code and then you'll be fine. I don't actually put 7 in there anywhere. I know, I know uh but there might be some new toys that we want to play with and there already were so one of the one of the fun things was uh was get in getting this is um is going through and being able to say oh i do things a little differently now like one of the parts is building a a pytest plugin and i package a little differently now than I did several years ago. So I got to use the new, new way and we're keeping that up to date. So that's the first thing. The other thing is this course took a lot of my time actually to get this to,

to the point where it is now. And, and so some things went on the back burner and a couple of things were the other podcasts. So Python people and Python test. Python people has new people. Will Vincent. And I've got a whole bunch more than I need to release. But Will Vincent was on in October. And I finally got this released.

And we talk about Django and writing technical books. And Will's awesome. He also is the host of Django Chat podcast. And then I don't have any. The last Python test episode was out December 15th but I've got a whole 16 of them planned they're not recorded yet but 16 are planned

so these will come out in the next few months so stay tuned awesome awesome how about you? well I have a whole bunch of talk Python stuff coming I think I have 12 scheduled or something it's kind of out of control to be honest but I've got some extras that are more relevant here.

You cannot see the joke yet. That is the end. All right, so Ophek just sent us a follow up. And it's a little unfortunate that we talked about this big release of Hatch 1.8 and then went on a two week break. Because like the next day he said, great coverage on Hatch 1.8, thanks. However, one small correction. Remember, I was so

impressed and psyched that the binaries of your apps, if you made apps with Hatch, which is a cool, cool feature of it, would be signed. I'm like, how are they doing this? How do they get away with doing this? Well, they're not. So one small correction. Only the binaries for Hatch themselves are signed by the certificate of the PSF, not the binaries created by hatch for you. Those are your certificates problems. So we're back to signing our own apps again and dealing with Apple and

Microsoft and all that stuff. Okay. But it makes more sense, right? Okay. Next. This is just interesting. High PI had new user registration temporarily suspended

there um the volume of malicious users and projects being created outpaced the ability to respond to it in timely fashion so like uh we just need to slow down no action required this is from last week just interesting and i guess you know thank you everyone for being on top of this honestly because what a hassle um i also would i don't know what they're doing right now but at least i would recommend um i don't know if it would really help but like cloudflare um turnstile at least stop stop the bots you know which is a way way way way way better way better thing than the google recaptcha thing so maybe if some kind of gate like that would be awesome then put that up anyway that's news also coming up next february february 12th which is in 33 days i'm doing a youtube only thing on talk python that i think

will be really fun with martina pugliese about data doodling. So she's got this whole cool series she's doing where she's doing data science by just sketching it out and just drawing pictures and trying to understand data before jumping into like notebooks and stuff. And so we're just going to talk through some really fun doodles. And it's all visuals, which is why it's YouTube only and not on DocPython. Exactly. But so if you're interested, please come and click the get notified to register and quotes to be part of that live event.

So hopefully people will. That'd be really cool. I think she's got some beautiful, beautiful work and you'll appreciate when we get there. And wrote an article called, which maybe should have R in the front r ai features a waste of time so my premise is that uh just wondering like how many cumulative programmer hours have been utterly wasted on adding very mediocre ai features to every app imaginable that you rarely ever want to use you know so i just lay out like a couple issues like look at all these

these examples of apps that just like they're adding ai for the sake of adding ai but it's not you know not bashing on like chat gpt or something like it's just like oh you can click this to like correct the date by ai's like i just type the date why do i need this stupid thing right and uh another this example i use with spark email which has this really cool um brian see how beautiful that is you can write beautiful emails in spark love it right but if you try to proofread it with ai it deletes all the content to the images and it just like jams it back as plain text like response from chat gpt or something like dude that's not what i wanted when i asked for you to proofread it.

This app is full of little tiny bugs, and they're adding this. Who would ever use that if this is the outcome of what you get? Anyway, another example would be the Dropbox fiasco, where they just silently turned on, hey, we're sending all your documents to OpenAI. I hope you don't mind. Do you have anything private? Probably not.

It's fine. Right? You're like, wait, why did you add that? Nobody wants this. Why hope you don't mind. Do you have anything private? Probably not. It's fine, right? And you're like, wait, why did you add that? Nobody wants this. Why do you turn this on? And why are you sending my data out of Dropbox? This is not right. So anyway, hopefully people will enjoy this article. I have another fun one.

Let's say another fun one coming. I forgot to turn that off. Oh, well, you can ask ChatGPT about that. Go on now. Yeah, no, honestly, it doesn't actually send it unless you interact with one of the, like a text box. It says, do you want to ask this thing and the AI about your doc?

But if you ask it, then it's on, right? So it's not clear by like entering that text box, that input, you're now sharing your data with some other company, right? Like, can we just have like more reliable integration with Finder and Explorer and better sync? Like, I don't need this AI stuff. I had actually on the topic, though, I had like a really cool AI experience. I went to the dentist, and they took x rays. And, and she showed me the x ray picture. And then she showed me the picture right next to it where an AI highlighted areas that might be problems that the dentist should check out and take a look at.

That's awesome. And things like that, like helping a professional make sure they don't miss things. Great use of AI. Yeah. Proofreading? Less. No, that's an awesome, awesome use of it. Yeah. Okay. Ready for the joke? I'm almost. I want to show one more thing I forgot to mention for a lot of people, New Year's resolutions time for a lot of people. And if your New Year's resolution is get better at testing, the celebrate the Py test or the complete pi test course you're getting

done the coupon code 2024 is open through january for 10 off so just mention that so now i'm ready for a joke all right this one has to do with uh managers and sprints and super agile stuff okay it was from work chronicles or chronicles.. Put it in the backlog with dashes. So there's a programmer, clearly a business, actually it says project manager, but whatever, like somebody on the tech team. And here comes a person, very cheery. Hey, I have a new feature idea. And the person, a little bit bugged, getting interrupted from work, says, awesome.

Just put it in the backlog. Person says, where's it? Where's the backlog? Holds up the trash can. Oh, dear. put it in the backlog person says where is it where's the backlog holds up the trash can oh dear put it in the backlog just go back we're busy yeah anyway um we actually we've got a real backlog but the things that are marked low priority it's never gonna happen just run a filter that just hides all the priority things yeah we're done backlog's all caught up no i know there's that feature i put in there oh It's never going to happen. Just run a filter that just hides all the priority things. Yeah, we're done.

Backlog's all cut up. No, I know there's that feature I put in there. Oh yeah, it must be a little priority. Got it. Oh, it's prioritized. We'll get to it. Beautiful. All right.

Well, fantastic to be back with you and everyone else, Brian. Good to be back. Yeah. Bye all. Bye.