Python Bytes - #337 Backtracking For a Package

Episode Date: May 23, 2023

Topics covered in this episode: Ruff PyCharm plugin Writing Python like it's Rust Pip 23.1 Released - Massive improvement to backtracking Markdown Code Runner Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/337

Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode 337, recorded May 23rd, 2023. And I am Brian Ocken. And I'm Michael Kennedy. I want to thank everybody that's watching, but anybody that's listening, I encourage you to at least once in a while drop by YouTube and see the upcoming list of events at pythonbytes.fm slash live to be part of the audience it's usually fun it's usually tuesdays at 11 pacific time

occasionally we switch if you'd like to connect with us we're we're all on fostered on or at least the three of us yeah michael and i on the show It's mkennedy at Fostedon and at Brian Ocken and at Python Bytes. And with that, let's jump into our first topic. Let's jump into it. It's not going to be a rough one, is it? So the first topic is ROUGH, which is obviously Charlie Marsh's project. It's very successful. People know that. It's not exactly ROUGH. It's very successful. People know that.

It's not exactly Ruff. It's a way to use Ruff. And this one comes to us from John Hagen. And John runs this project called Python Blueprint. And so he's been playing around a lot with that project and PyCharm and Ruff and realized that now there's a plugin for all the JetBrains IDEs, most notably PyCharm, called just Ruff. And the idea is that, you know, PyCharm has all these little squigglies and warnings and maybe even more importantly, the auto corrections. So it'll do things like, if you wanted to replace double quotes or single quotes, you can just hit alt enter,

and it'll suggest, hey, why don't we just do that for you, right? Things like that. And so this integrates all the rough functionality into that same basic UI system, right? You get little warnings or errors on the screen based on rough output. So it has inspection and highlighting. It can set it up so it runs rough on your code when you run reformat code or just hit the hot key, command alt L, control alt L. It has the quick fixes that I was just talking about.

And it will actually run, you could run rough dash dash fix as an action. And you can even run that when a file is saved just automatically. Like, Hey, if there's stuff wrong with it, just fix it rough. Just do that for me. You can configure which version of rough is running. So basically the plugin lets you specify, do you use a global one, maybe managed by pip X, do you use a local one in a virtual environment? I sort of feel like isolating that to a per project basis is the right thing so that's what i'm doing playing with this and yeah you can run it as a new process you can specify a config option like a set yaml i'm not sure what

format that is but whatever the rough config file format is so that you can say you know what i don't really care about the line length for this thing. So just ignore that and don't ever run that and so on. So yeah, you can even run it out of a WSL Windows subsystem for Linux. And there's some nice screenshots as all UI things should have in there. You can even see some of the settings if you want to try it out. So it has 4.8 out of 5 ratings and looks pretty new. But yeah, anyway, I think it looks like a good option. So people can check that out.

That's pretty cool. Yeah, I'm really excited about it. I've installed it. I'm going to give it a try. You can go check out the, it's open source, obviously. So you can go check out basically the repo for the plugin. And John sent over to us, and I'm linking to this thing where it says you can add additional high charm specific instructions for both black and rough so he's got this section that shows you basically how to integrate both black and rough at the same time as automatic code formatters

and pie charm using the file watchers plugin So just follow along with the steps there. It even has Nox support, but yeah, cool. So if that sounds interesting to you, if you want to have kind of auto-Rough just built into PyCharm and use PyCharm, then yeah, check this out. It looks cool. Thanks, John. Yeah, that's nice. And Rough, of course, is written in Rust.

And I'd like to talk a little bit more about Rust. So there was an article I ran across called, it's from Koblo's blog. Don't know if that's a first name or last name, but thanks, Koblo's. Writing Python like it's Rust. And I haven't written Rust yet, but a little bit. Anyway, the thing I liked about this article was really basically he's going from Rust back to Python programming in both. And one of the things he misses is some of the safety that types give you. So he's discovered that he's changing or they're changing how they're using types within Python.

And I kind of liked some of the suggestions. I think these are some of the things I didn't even think of before. The obvious one, of course, is for function signatures. So we really want to, if it's not obvious, like there's an example with a find item with records and check. But what is that? What are those types of those things? So it's really helpful to your call, the people using your API to specify what the parameters look like or what you expect them to look like.

And there's so many options within Python now. And also that to return value. And here is an example. it's an optional item. So I actually hadn't thought about that of like using optional as a return type. That's pretty cool, which means you can either, and you could probably do like item or none to say it can return none or something. That's a, it is good to, especially if it's something, if it, if it's possible to return none, it's good to have that in the types. So that's, that's the low hanging fruit. That's, I think a lot of people have gotten there yet already. The other, the other thing that I found

recently is I've got some types that are like tuples or dictionaries or specifically a specific kind of dictionary, like a string to any or even, I use named tuples a lot, which are a little bit better than just your normal types, but his recommendation is to go ahead and use data classes because they're more descriptive, even if it's similar information. You can write up a little data class

and then like in the example, instead of saying that you're returning a dictionary, go ahead and return like a person item, specific types. And I'm kinda used to that from C++ of like writing little types because they're more descriptive and they're easier to read sometimes. So bringing this into Python's kind of a cool idea.

The other, one of the things I thought was really awesome is this idea, there's an example using, where he's using like these packet types. And since I work with communications a lot, this really hit home. A packet might be either a header or payload or trailer this is a rust example but you can do the same sort of thing within python to say i can have different data types um using data classes to specify what kind of information you're going

to get from these and then you can create a union type and have that have that be a different name uh but related to everything and And I kind of love this idea of using a union type instead of using the ORs. I mean, using OR for types is good too, but combining that into a union type is pretty slick. Yeah, yeah, that is pretty interesting. I like that. So is that if you're going to receive, say, a packet off a network and it had first the header bit and then the payload and then the trailer or something like that? Yeah, or like a function that can count the bits in there or something like that. And it can either be a header or a payload or a trailer. You could do something different with those. And having a union type is pretty cool. Now, within it, and here's an example of how do you deal with that, you'd still, with inside, you'd have to either do pattern matching or if clauses. I've been using a lot more pattern matching lately to be able to just, you know,

this is great for you take the packet type, but then you can match it against either a header or payload or trailer. This is a really clean way to have union types and then unpack them with the pattern matching. It's a pretty cool way to deal with that. Yeah, that is actually pretty clean. That's one of the better examples of pattern matching that I've seen lately. And this is just like a third of the article, but the rest of it does talk about different ways, too. Basically, we have a lot of these tools within Python

now to make it more to utilize data types that we were used to in other languages. And you can kind of write some really clean-looking code and easier to read. So it's nice. Yeah, very nice. I like it. So Kim out there has a question in the audience.

Ryan, is there possibly some overlap with writing Python after writing a bunch of C as you do? As in, same way Rust leads to some good ideas in Python. Yeah, well, I mean, this person's writing it from the experience of Rust, but I'm looking at it going, this is a lot of the C++ stuff that I use. So it is, yeah, I'm looking at it.

We're actually, my team is having a lot more fun with Python with the pattern matching statement because I know it can do a lot more stuff, but it really does, even if you're just using it to get away from a long list of if clauses and make it more like a switch statement, it is so much nicer. It's one of the little easy bits. So that's a good example. Yeah. Marco out there says, but do a lot of rust, missing match as well. All right. Shall we move on? Sure um all right well go ahead go ahead with your first uh

i know you had an announcement as well you wanted to give out no i was just wondering do we want to uh today's episode is not brought to you by an external sponsor but it's brought to you by us and it is brought to you by us and i'd like to uh hear um some of the news from michael well the big news for me, which I have sort of put in as an extra now and then, is the mobile apps. So people go out there, download the mobile apps, talkbython.fm slash apps.

And in there, if you create an account or log in with your account, at a minimum, you'll find like six different courses that are free. You just tap on them and you can take those courses right away within the mobile apps. If you get this super, super quickly, if you're one of the very, very prompt listeners, there's a chance that our get course actually is free as well as part of a celebration of the mobile app launch. But that lasts for about eight hours from the time of

recording, which will be a few less hours from the time I actually released this to the general public, not in the live stream. So you'll have to act fast on that one, but do check out the mobile apps. If you're interested in Python courses, they're a great way to take it. They have some free ones and they also are the best way to take ones that you paid for as well. So four months of work, finally, a really, really nice apps that we got out there. I'm listening to a couple uh watching and listening to a couple a couple courses right now um not right now but uh in in some of my free time i love the the mobile app it's working great awesome thanks and you also have a thing to shout

out as well right yeah i wanted to um let people know i wrote a book no you all know that um but the exciting bit i just got this email this morning so my book python testing with pytest second edition um i've had a lot of great feedback of saying uh i i didn't people saying uh they read the first one and it was good but the second one really nailed they really got some of the concepts in so i I'm glad that I wrote the second. So the news, I just got this this morning, an email from Pragmatic saying that there is a sale right now. So there's a whole bunch of other books, but Python Testing with PyTest is one of them.

And if you use the coupon code SPRING2023, and you can save 50% off of the ebook. And that's up, that expires June 1st. So don't delay spring 2023 so awesome let's bring you to some testing um well that's us uh how about a new topic yeah thank you everyone for supporting the show by supporting our work let's talk about hip so this is about a month old i've been looking to grab a slot to talk about it, but there's been other sort of higher important stuff.

But I do want to point this out. So Zyrtex over on Reddit, who if you follow over to the GitHub and then from GitHub, I think to Twitter, maybe named Damien, not entirely clear, so I want to give credit, but it's not easy to do, says that PIP 23 or points out that PIP 23.1 was released with massive improvements to backtracking. And backtracking is an algorithm that is part of the requirements resolver. So if you say, you know, PIP install requests, flask, and, you know, something else, PIP has to say, okay, well, we know those are, but both requests and flask may depend on some

library i don't think that's true but i don't think there's an intersection there but you know follow along if they did then it's got to figure out well okay flask requires this range of versions and request requires that range so let me find a version that will satisfy both of those requirements right and there might be a transitive dependency upon that, right? Like that shared, potentially shared library itself has another thing that it depends on. So the article or not article, Reddit post says, you know, for example, let's say you require package A and B,

the latest versions of A and B are downloaded, pip checks their requirements, and it finds that A depends on C version two, but B depends on C version 1. And so it's got to figure out, well, can I go back and get an older version of A that would allow, say, C equals 1 to work, right? So prior to PIP 20.3, the default process would allow PIP to install conflicting requirements with a warning saying that this may or may not go well. We couldn't find one that was a good fit, but that's no longer an option. I have very mixed feelings about that. I love that it tries to be more strict and correct, but at the same time, the alternative or the trade-off is we guarantee

correctness 100% of the time in terms of a version match for all the transitive dependencies across all the different things. And instead of saying, well, install something that doesn't work or may not work because there might be some functionality that you may or may not interact with that is incompatible version-wise, we're just going to say your application cannot run. It's impossible for your app to use these dependencies and run. Personally, I would prefer to say, huge warning, this is not a great idea, but at least you can take a shot at it rather than it's impossible for you to run this application. But that's the trade-off Pip made. And because of that, it has to be even better at

tracking down these version matches as best it can because the alternative is your application cannot be installed and run. So there's a lot of interesting history here, but it talks about some of the algorithms that have been used and points out that pip now separates out the resolution logic into a library called ResolveLib. And it was discovered that there was an error, logical error, that both for performance and for correctness as well. You know, better backtracking technique called backjumping and an actual error that were fixed and implemented in ResolveLib and now come out in 23.1. So if you're using PIP and if you're listening to us, there's a real good chance that you are. You want to upgrade your PIP to 23.1 or higher. I, whenever I install requirements, I just have like a multi-step macro or alias. The first thing it does is say pip install dash dash upgrade pip.

Now go try to do the other stuff. You know, just, it's like a concept for me, but I know not everyone does that. So when you see that little warning that says hey warning your pip is out of date uh if it's less than 23.1 you probably want to um take uh i don't know take the advice and update it this time yeah yeah just this morning i got a message sorry we cannot install your website because some you know sentry requires this version of EuroLib3 and Request requires a different version of EuroLib3. So you can't run your app. Goodbye.

It's like, come on. I'm sure that it's not actually using the conflicting. And one thing that this does point out here is traditionally, people haven't had to worry about this. And so you'll see things, one of the dependencies will say it has to be exactly like 1.2.3 when really what it means is it's gotta be later than 1.2, you know, and they've just pinned it overly tight and you end up with this crash where, you know, for, for sure that it wouldn't actually be a problem in your situation. Like, how do you know? Well, because before PIP 20.1, it was running, you know for for sure that it wouldn't actually be a problem in your situation like how do you

know well because before pip 20.1 it was running you know those kinds of things and so there's this little bit of you know i know it's great that it's uh trying to be more accurate and precise but sometimes i'd rather have at least an option than none and so i don't know it's there's a whole interesting discussion down here people can check it out in the comments. It devolves into a debate about Python two to three for a little while, which is weird. And then just skip that. That's not productive reading. But there are some interesting conversations going on there. that there is a PIP 658, which is accepted for, what version of Python is this? I don't know, the 2021. So a couple versions ago.

That allows the metadata. It used to be that PIP would actually have to download and install a package just to see what the dependencies were. And now that's separated out the metadata so you can get a very simple, small download without trying to do stuff to it to go, what does this actually need? This version needs what? Right. So there's are some improvements that are being brought into here, but still, there we have it. Yeah. A couple of comments, which I kind of agree with. It'd be from Grant. It'd be nice if there was a warn unresolvable or something to keep the old behavior i agree with that yeah um so um i what i guess this is a reminder to library authors as well that your dependencies you might know that you you have a lower limit on some dependency that you need you need version

like 1.2 of this library or above, think about it. I prefer to have libraries in their dependencies, the transitive dependencies, in a lower bound and not upper bound version. Because, I mean, you don't know what the upper bound is. Unless you do. Unless there really is. There was a breaking change, and you really

know that there's a break. DAVID ABELSON- Right. The thing for me more is about, which I agree with you, Brian, and the comments, but it's about like there might be an absolute 100% conflict. Like if this library tries to do, so for example, let's say like request tries to use Kerberos authentication instead of basic authentication. And in that scenario, this other library does something something crazy it crashes unless it's high enough if i'm never using that authentication mechanism i'm never gonna so like yes it is actually a break and change but it's not a break and change for me oh yeah my use case of that the combination of those two things right like that's that's kind of where i've been thinking about this oh yeah interesting yeah because there Yeah. Cause there's a lot of like a lot of Swiss army knife libraries out there and you're not using all of them.

It's very unlikely you're hitting a hundred percent surface area of a thing and it's dependencies. Right. Exactly. Yeah. Interesting. Yep. Yeah. All right. Well, anyway, this is, this is an improvement. I feel like we're kind of like debating the 20.3 debate. This is an improvement on the stuff that's already been decided to be done. So 23.1 is a good way to go. Yeah, it's good. Last item was a kind of a cool tool called Markdown Code Runner.

So this did remind me of a tool that I've known before. So Markdown Code Runner is a package that automatically executes code blocks within a Markdown file. It can include hidden code blocks. So you can have, the code blocks can be in comments. And so you can't see it in the Markdown file. It just runs it. So it runs the snippet

and then you can have the output show up somewhere. So this, in the example, let's run to an example. You've got a little code block that says Python, but instead of just Python, you say, you know, you got your backticks, and then you say, Python space markdown-code-runner. And then it runs the code in the code block. And then it pops it out.

You've got another couple comments for output start and output end. And it'll throw the output in there. The kind of the neat thing is it's not just for Python, it's for bash also. So you can run some bash scripts. Like if you want to show a I was thinking, if you wanted to show what the the directory looked like with the tree command, you can go ahead and run tree within a Markdown file and run it or other stuff. It has several examples that they think would be neat. Things like, I don't know, like diagrams or tables or various things that you might want to output with python uh visualizations you can use that uh this is really cool yeah it did kind of remind me of cog um but the the syntax is a little different so uh cog from ned batch elder has a is a similar sort of thing you throw

you throw some uh code in in place with some special tags he He uses like three bracket tags instead of an actual code block. So the thing I kind of like about this is a lot of times I actually do want to show the code block. So you can go ahead and show the code block and then run it. So that's neat. It's fun, I tried it a little bit. There's an example of using Rust,

so you can have actually running Rust also. I couldn't get the Rust example to work, but I got the Markdown or the Python and the Bash examples to work. So actually pretty cool. Within the documentation, it also talks about running this as a GitLab, or a GitHub CI snippet.

And that'd be a great place to do it, is to just rerun, like, reproduce the code output from your readme on the new code. So kind of a cool idea. Yeah. My first thought was, oh, I have a code example, and it says the output is this. Like, rather than trying to maintain those,

just let it put the output there. But it also could be useful for just, I have a little Python bit of code that generates some other output that is is useful right like here's the go through and generate a little all the topics that it's on the sub directory or something like that yeah yeah like kind of just a markdown like a macro within a markdown generated table of contents or something yeah exactly something like that cool cool that's all of it right that's all the main topics that is our main topics yes all right i have a few extras to throw out here let's jump over there real quick so um same person who pointed

out the rough plugin john hagan also pointed out you, I had talked previously about how I wish pip would update itself when I create a virtual environment by default. So if I say Python 3-m vnv, vnv, the very next thing after activating it is your pip is out of date. Like, ah, geez. Okay. So hence, that's partly why my first thing to do is always upgrade it is just to not see the warning more than anything, honestly. But pointed out that since uh python 3.9 there's a an additional option you can pass to um um the virtual environment creation story so we instead of um just dash mv and v and um director you can also pass dash dash upgrade depends which will um automatically do that upgrade of pip and dependencies as well. Pip set of tools to the latest version as part of creating the virtual environment.

Sweet. Yeah. So nothing major, but quite nice, right? Yeah. I mean, I usually have that as a second step within my little macro. So do I. So do I. And I was looking to replace that.

And I'm like, you know what? Not quite, because I also want to install things like pip tools and a couple other things that this doesn't include. So I'm like, I still would have to write that line. So I'll just leave it. But anyway, still, it's really nice to have dash dash upgrade depths.

You want it to have a shiny new virtual environment all the time. Yeah, and one of the things I really love that came in like a few versions ago is the um if you do the dash dash prompt and give it a dot it uses the directory name yeah that's really excellent it names the virtual environment that containing directory name so it has like the name of the virtual environment is the name of your project often which is great yeah yep all right more things more extras uh one is icon south africa icon z uh za will be held in durban and the um most important part here is that um the call for proposals

is out um and when is the time frame kim will have to let us know. I know he's out in the audience, but yeah. Oh, here we go. Talks need to be submitted by August 18th, 2023. And it's pretty good. I think it's both virtual and in person. So some good options for people to attend and down in October. And it's pretty convenient for Africa, Europe, much of Asia,

although less so for, you know, you and me, Brian. That's all right. That's all right. Yeah, cool. So people, if they want to talk at PyCon South Africa, be sure to submit that talk. And Kim says that's a soft deadline, but sooner is better. Okay.

Generally true. I got a friend that's in the cyber security uh area and he's up all night anyway um so yeah there you go yeah you could definitely do it if you just live with auto off hours yeah uh real quick follow-up from something before brian and then we'll get to a joke i had put out a call to everyone and say help me find some off-road trails for this adventure bike thing that i got into. I didn't really get any, any feedback. So I'm going to instead pay it forward to other people out there who might be listening to my ride. So some really cool apps on X off-road, you can go through and find, you just click a spot on the map and it'll show you like, here's

all the public legal trails for you to go tear around on. And they're even rated like five out of 10 or seven out of 10 with pictures and distance and challenges. So you can decide upfront whether or not you, you want to go down that path, I suppose. There's also a GIA, G-A-I-A, which is a similar thing. And then backcountry discovery roads, which allow you to find like, how do I traverse my state, at least in the u.s or similar things in um in europe with the tet how do you say traverse all of oregon almost 100 off-road through the forest and the mountains and the deserts and so there's this app that has all these like gps trails so anyway i found some cool trails way up in the mountains using this app and can recommend it to people neat cool yeah

cool all right a joke you got a joke for us yeah i i can't remember where i found this but um there's a website called uh um user in your face.com so instead of user interface user in your y-e-r face.com so you just have to fill it out. Like an anti-pattern, a whole combination of anti-patterns into one UI. Okay, so just start right off the bat. So for people listening, I'm sorry, you're going to have to watch this on the video or something. Or visit the link. Or go visit the link. It says, hi and welcome to User Interface. A challenge exploration of user interactions and design patterns.

To play the game, simply fill in the form as fast and accurate as possible. With a button that says no. Big green button. So it says, please click underscore underlined click here to go to the next page. And the next page is highlighted and the trick is the only thing that you can click on is the here button that it's not a button it's just that's it um okay and then it okay the site uses cookies is that a problem for you yes and yes doesn't do anything okay no not really that goes away can we help um there's a there's a help field uh

with no enter just send the one of those dreadful chat messages that pops up yeah it's it's great um uh the choose your password you click on it and the the the highlighted choose your pet or the the preview text text is already there so if you add your password to it, it doesn't delete the old one. You got to like fill that out. Oh, you can't tab through anything. So you have to click. Oh, the email's bad too.

Yeah, instead of a placeholder, it's just gray, actual gray text. Yeah, but you also can't delete it. Domain, oh, this one you have to delete also and uh oh there's a hurry up time is ticking a pop-up one minute before um can i hit the oh the x isn't an x it's a maximize button uh lock right it's a close lock unlock this doesn't do anything oh the close is the copyright right that isn't obvious um yeah okay i do i do not accept these terms okay cancel oh i'm not no i meant okay uh you sure you want to cancel cancel the cancel uh next oh it says sell already selected

i do not accept so you have to uncheck it to accept the terms and conditions. What are the terms? Oh, yeah. Okay. Good luck getting out of that dialogue. You got to scroll down to accept them now to get out of it. Okay. Well, we're stuck now because you can't even... This is terrible. Anyway, I did get through it. The fastest time i have so far is like five minutes to get people are gonna speed run the in your face interface yeah i'd love to see a speed run to see how fast if somebody can really get through it in like a couple minutes please do a video i want to see that uh so and jeff out in the audience says, I worked for the company that made that site.

Yeah. So yeah, the company looks like Veerheart. So nice. Yeah. All right. Well, that's enough frustration for one day. Indeed. Thanks again for joining us and doing another Python Bites. Thank you everybody for watching and listening and supporting us through the courses and books and everything. And Patreon supporters. We still have Patreon supporters. Thank you, Patreon supporters.

Absolutely. And thanks, Michael. Thanks, Brian. Bye. Bye.