Python Bytes - #296 pip: Constrain your excitement

Episode Date: August 9, 2022

Topics covered in this episode: Pip constraints files async-cache Organize Python code like a PRO keyring Extras Joke See the full show notes for this episode on the website at pythonbytes.fm/296

Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode 296, recorded August 9th, 2022. I'm Michael Kennedy. And I'm Brian Ocken. Hey, Brian. We have a new sponsor. I just want to say thank you to Mozilla and the IRL podcast for sponsoring the show. Check them out at pythonbytes.fm slash IRL.

More on that later. For now, I want to hear what you've discovered to share with us. Can you constrain your excitement? Yeah, I want to talk about pip constraints. So there's a, I think I knew about pip constraints, but I kind of forgot about them. But there's an article called Pip Constraints Files by somebody named Luminousman. So this is what, so they're kind of neat. So one of the things I was just using pip requirements dot in file recently, in a course that I'm taking. And I like

using requirements dot in to generate my requirements dot txt file. But there's and then there's, there's a Yeah, anyway, that uses pip tools. So you have totxt file. But there's, and then there's, there's a, yeah, anyway, that uses pip tools. So you have to get pip tools, but so there's that. And then there's also pinning. So especially with applications, we see it more in applications, less in, less in libraries of pinning the application, but you can, even in libraries, there's a there's regularly uh sort of constraints around stuff to say um hey for this library i need this this range of versions or it has to be greater than this or something because i'm depending and that's fine but this is a way to say not what libraries i want to want to use but if i use a library which version so or constraints on the version without saying i want the library so

and in the actually in the pip documentation it says um constraint files are requirement files that only control which version not whether or not it's installed so how would you use this so um this uh this article talks about it and it basically says you, you kind of use it normally pip install requirements, um, that text, but you might have, if you do like a freeze for instance, or something,

or you just pin everything, you might have all of the versions, but you might only want like, uh, constraints on like one of them. Say, let's say you want your, your including pandas,

but you want a certain version of pandas or, or you want a certain version of numpy, even though pandas requires numpy or something like you can have a constraints file that lists um this just looks like a pip freeze file but you can you can put like less than or less than equal if you want or and you don't have to have everything so you could just pin one of the things and that way like let's say you were doing pandas and you wanted to constrain NumPy to be a certain version of NumPy. You can do that with a constraints file

and not have to specify everything, just have it be separate. And the article talks about actually just sticking the constraints, specifying your constraint file within the requirements file. And that way they're separate. And I was the requirements file. And that way they're separate. And, and I was thinking about that. And that's an interesting thing to say, the dependencies

of my application don't change, but the constraints might because of testing or whatever. And this, this separating of these two files would help with like, you know, when you have the two files of version control, you've changed your constraints, but you haven't changed really what you're depending on just the versions of those. So it's kind of a neat to have that separate possibility. So, yeah. So for people who are listening, literally the first line of the requirements.txt file is dash dash constraint constraints.txt, which I'd never considered doing that. That's interesting. And then one of the things I thought is, but I'm not, and this works if you're handwriting your requirements file.

But what if you're, what if you're not? What if you're using requirements.in instead? And this article doesn't go into it, but I tried just instead of putting that constraint thing in a requirements.txt file, putting it in a requirements.in and using pip compile to generate it. And the pip compile seems to also watch the look at this. So this constraint works for pip compile as well. Okay. So that's cool. Yeah.

So kind of a neat thing to check out. The usage of it's pretty easy. Just, yeah. Anyway, pinning your requirements is good, but don't pin them too tight if especially for application for, for libraries and then for applications. Yeah. I kind of like this because there's a lot of times where I know there's a bug in something or I've heard about it or, or I haven't gotten around to fixing my code to deal with the new version yet. So I'm going to pin something and I don't necessarily need to pin everything. I just need to pin certain parts of it. So that's kind of neat.

So I like this. Just also a little bit of a sidebar with the pip compile from pip tools. You know, you give it the end file and it generates the TXT file and it basically obliterates the TXT file, whatever is there previously. Yeah. And that can can be a hassle especially if you want to have like a requirements dot dash dev and then a production requirements and if you install the the dev the dev one you want to also pick up like a dash r on the the main ones and the pip tools blows that away so what i ended up doing a lot for my workflows is having pip tools generate some base txt file and then having like requirements.txt just have a dash r you know requirements production and then the dev have dash r requirements that tx you know broad and like sort of like put those commands just in real simple and have it actually generate

a separate file so it kind of makes it a little bit messy but it it gives you lots of flexibility cool yeah i'd like to see that written up. That's neat. Yeah. Maybe I should actually blog something like in the last three years. Blog. Yeah. What is that? Is that with words? Written words, not spoken words. Pam Phil on the audience says, does it take over requirements? Like if pip would resolve numpy to 1.19 and you say at 120, it sounds to me like it does but what do you think brian well like i tried it uh with uh typer so i knew i know typer pulls in uh click for instance and they're both command line things yep um and so then i said okay well the typer has a broad range of click things they can do and if i constrain click to be a lower number, will it work? And I blew everything away and tried it again.

And sure enough, it did. It like add those extra constraints on top of even like, so I was only declaring Typer, but Typer was specifying click and I could specify which version of click I wanted. So yeah, this is cool. Yeah. It's, I guess, adding one more complexity to your packaging workflow. But useful if you need it. Flexibility.

Yes, good flexibility. All right, well, I'll cover something simple. Async caching. Simple. It is simple in a sense. We have some nice stuff built into Python, like Async, sorry, with like func tools and the LRU cache and whatnot. But from what I understand, those are synchronous only.

Basically, they're decorators. Those decorators wrap functions and the decorators themselves are synchronous. And so it only makes sense for them to wrap sync functions. Yeah. Okay. And so if you have an async function, but you want to do the LRU cache where you just put the decorator LRU cache, and then if it gets called with the same arguments, again, it doesn't even call the function.

It just goes, you know what? You already called it with that. Here's the answer. So like if you're in a tight loop and you're pulling in some values and you got to compute something with it through a function, but there's a good chance of repeat of those values, and you got to compute something with it through a function. But there's a good chance of repeat of those values, you can put an LRU cache. And long as that pretty deterministic, and you call it again, you expect the same output, you can make it fly by just adding one of those caches on it, right. So short version of this is, this is the same idea, but for async functions. So I can have some function I want to call and I just say at LRU cache, and this one,

you give it a maximum size of results that it's willing, you know, inputs and matched up results, it's willing to cache up. And then if you call it with the same arguments, it'll give the same response back. So that's pretty cool. That's the last used version. And then you also have time to live a time to live an async TTL. So you can say any results, I don't care how many I've used, but just within the last 60 seconds. And one thing that's really, really nice about that is it will expire results. So maybe you're calling an API and you want to do rate limiting, but you know, you only want to call it maybe once a minute, right? That'll both make your code faster, but also not overrun your rate limiting

that you might have with your API key and so on. So here's a real simple way to add rate limiting is just say time to live. I guess you got to have the same input arguments, but, you know, assuming that you have the same arguments, that's one way to do it. And you can also specify the max size, which is pretty cool. Yeah, or if you're grabbing something off of a service,

like what's the temperature out um i don't really care if it fluctuates every second but every minute i might check it um sort of thing yep yep precisely and i don't there's not a conversion all those kind of things so there's not a ttl on the normal lru cache is there i don't believe Yeah, so that's actually cool. I like the time to live part. I do too. I like that. That really resonates with me.

The other thing that's pretty cool here is you can pass ORM objects, you can pass request objects, you can pass custom classes. Even if the classes are not hashable, it will still go through and actually... So one of the problems you can run into is if you say, if you've got a customer object or a product object or something, you call it once

you've created this object, maybe you got it from the database and you say, call the function with the LRU hash. And it says, well, what object is this? Have I seen it before? And maybe yes or no. And then you call it again. It might have the same effective value, but it's not actually the same object, right? You might get it from the database again. So it has a different pointer, a different ID and so on. That, I'm not sure what the behavior there is, but this one will actually look and see,

oh, is it actually a class? Well, then let's just get the dictionary and use the dictionary, the underlying field dictionary of the class to use as the, the match to see if I'm calling it again. So there's some really cool functionality here, simple little class, but if you want to quickly add some performance boost,

async functions, you can add this. Oh, nice. Okay. This is neat. Yeah. Yeah.

Yeah. It's fine. Also. Yeah. Thanks. Also it's entirely possible i i don't think it does but it's entirely possible to add async and synchronous support to a single decorator if if you need to right um for example i have the what was it called the chameleon templates

um fast api one that i created let's see see. Look at that. Number one result. What a search thing. So there's this fast API chameleon framework or library that I created that allows you to just do a decorator and say dot template and put a HTML template in the chameleon language on a fast API response. And it returns a dictionary and that just turns it into an HTML response. Okay. This one in fast API, you can both have synchronous and asynchronous functions. So this thing has to look and see if the inbound thing is a co-routine and async co-routine, and it will dynamically generate the right wrapper and async one or a synchronous one based on. So it's not super hard. It's, it's also not also not super easy but i did it so it can be done cool anyway uh that's a little bit of a diversions but this this async cache versus a

non-async cache i feel like you know it could just be it could be one thing if it really really wanted to be but i feel like the person who created prize just like i need this for async methods let's go yeah it's almost kind of too bad that the normal lru cache doesn't just do yeah and you know maybe it's been updated too and i don't know but i don't believe it does i don't think so right not currently yeah yeah not that i know of people can write us if we're wrong and we'll let people know next time be great to great to do it i am never wrong oh okay sometimes and because like i said it's because it's a decorator, like you could, you could make it start working that way. That's a good feature to add.

Cool. All right. Before, yeah, before we move on, Brian, let me tell you about AI in real life. Wow. Yeah. So this week's sponsor, this episode of Python Bytes is brought to you by the IRL podcast, an original podcast from Mozilla. So thank you, IRL and Mozilla for supporting the

show. If you're like us, you care about the ideas behind technology and not just the tech itself. Obviously, we do a podcast on these things all the time. So we love talking about it, thinking about it. And everyone out there knows that tech has an enormous influence on society. Many of these effects are hugely beneficial. Just, you know, you think about walking around with your cell phone. You have basically the entire sum of human knowledge just constantly with you. Other influences can have, you know, negative effects. And I really appreciate that Mozilla is always looking out for and working to mitigate these types of negative influences tech has on all of us.

So if these ideas resonate with you, you should definitely check out their podcast, IRL, the IRL podcast. It's hosted by Bridget Todd. And in this season, IRL looks at AI in real life. Who can AI help? Who can it harm? The show features fascinating conversations with people who are working to build more trustworthy AI. So just some of the examples of episodes.

There's an episode about how our world is mapped, like Google Maps style mapped with AI. And what's really interesting is the data that's missing from those maps tells as much of the story as the data that's there. So also an episode about gig workers who depend on apps for their livelihood. And it looks at how they're pushing back against algorithms that control how much they get paid, seeking new ways to gain power and autonomy over data, and creating better working conditions. And finally, for political junkies, there's an episode on the role that AI plays when it comes to the spread of disinformation around elections, a huge concern for democracies. You hear a lot about the US democracies, but more broadly, absolutely, across the world. And I just listened to the first episode, the tech that we won't build,

which explores when developers and data scientists should consider saying no to projects that can be harmful or, you know, strongly against their beliefs, even though sure you could technically build them, you know, just because you can, you know, should, should you? Anyway, if this sounds like an interesting show, try an episode for yourself. Just search for IRL in your podcast player or visit pythonbytes.fm slash irl links in your podcast blur show notes and thank you to irl and mozilla for supporting our show thank you and strong yes thanks so i want to talk about organizing your code to actually uh organizing your python code kind of structure structuring projects and everything but uh there's

more than that so i ran across this article called organized python code like a pro and yes it's got a lot of great advice and it's opinionated it's by one person of course but but i think it's uh it's for the most part really good stuff and also um a couple things that i don't normally see in these kinds of articles and there's not too much weird stuff. So, uh, sometimes it's a little, sometimes it's a little too opinionated, but this is obviously where you can see where some of the opinions are held. So, uh, take a look at, um, it talks about structuring your project. Uh, for instance, one of the first things is use a source directory, SRC. And so I try to do this, and I used to do it because there was an article about having your tests be seen. So basically, if I'm doing a installable package,

I'd like to have my tests see the installed package, not the local files. And that happens sometimes if you're running, like, say, PyT pi test or unit test from the top level directory, and it might it might see the top level module and you don't want it to. So source is a way to hide that. But there's ways to get around that and testing. So I don't, I don't really, it's not really a solid, solid argument as it used to be. This argument really is just it looks nice. In your in your code editor that you, if you, like, here's an example of a non-source project where you have a couple modules within the project, but alphabetically, they fall below, they're around, you've got your test directory and your pyproject.toml, and your source code's on both top and bottom of that.

That's confusing so i actually kind of love this simple argument of just combine all the source in one place it's nice uh so i do like that too i know the the the first reaction to this though is you're going to put a package level directory anyway and having a package level directory in there instead of your if you have a package instead of this, you know, source or something that works too. But anyway, this is kind of interesting. The one thing that, that kind of gets me and it shows up here is this, this, this author is considering what I, so we have a problem in Python of what a, what a package is. A package is something I install from PyPI, but it's also within this python documentation

sometimes it's just a directory with an init file in it um uh i don't know how you so python or michael you teach people about that do you ever like do you stumble with this part or just both seems complicated and overly simplistic for me i think one of the challenges really I often run into is how do I organize my files if I want like a sub module, I want simple import statements, you know? So, so do you think I don't have, yeah, go ahead. Do you think of directories with stuff, directories within a net as a module or as a package or do you use? I do, but often I often I try to dodge that bullet

and just not really get. Yeah, well, honestly, not talk about it. If you're building a library, this matters very, very much. If you're building an application, a web app or a CLI app or something, often it doesn't matter because you're just running the top level,

some top level like main or app.py or something. And it'll just pick it up, whether it's a module or a package or just a directory. So I guess regardless of what we call directories, whether we call them modules or packages, this article calls them modules. So then it goes on to talk about some other cool stuff. Let's go down. Naming things. So it talks about there's really no files there's uh

there's modules so uh it also also there's no directories they're all modules but the that's okay so this is some of the opinion stuff that you can it's interesting you can skip over it but the the the thing that i thought was interesting is these module names they should be um uh they could be plural names and i never thought about that and it it kind of makes sense uh like if you have um it gives an example like drivers drivers would be a module yeah it'd probably be the s on there makes sense so it said you know keep it keep config and main as as single but um most things have an s on the end. I never really thought about that before, but it does make sense of like from crawler storages,

import get storage or something. I don't know. Yeah, yeah. It's just a nice little extra thing. Then it talks about naming functions and stuff. So functions, this is something people get wrong all the time. So it's good to talk about it. Name your functions with verbs. It makes good to talk about it name your functions

with verbs it makes your code a lot clearer if your functions or methods or verbs and even unless you have to jump through giant hoops to make it work but if you have to jump through giant hoops to make a verb work maybe it's not really one function maybe it should be two or three but we'll see anyway yeah or property instead of a function or property right uh and then class names one of the things i never thought about also was class names should be um uh singular so classes should be singular unless it's a unless it's really a container so don't name don't name a class orders because it's going to describe multiple orders it's it's an order it's an order class not an orders class so that's a i it's a good thing that the. It's an order. It's an order class, not an orders class.

So that's a good thing. One of the things I loved about this article also is there's two things that we kind of talk about. We use a lot, but I don't think very many people talk about it too much. And well, it's the dunder init thing. I'm going to pop down. The dunder name equals dunder main. That's used all the time um and so it's good to talk about that of if you want to execute a module itself use that but the if you one of the things i tried to do recently that i kind of didn't know how to do right off the bat is a directory with an init if it also has a dunder main then you can use the dash M thing on it. So if you include like dendermain, then you can use like a Python dash M module name

when you're running. Interesting. Okay. Because I had a library I was working with and it was like, I'm using dash M for everything else. I'd like to have the entry point for my application be usable if I do dash M also. How do I do that?

And this is how you do it. So it's kind of neat to have this in right away because I don't know if it's really a beginner thing, but it's still kind of cool. Yeah, I like it a lot. It's like entry points, but simpler. Yeah. Anyway, so decent, decent article. There's some opinions there, but that's okay.

We like opinions. Absolutely. We do. We do. All right. Well, but that's okay. We like opinions. Absolutely. We do, we do. All right. Well, do you know what else we like, Brian? Follow-up. Yay! I was going to do this.

I'm glad you're doing it. Too late. I grabbed it. I grabbed it. Because this one is a good one. So remember last week you spoke about CLI apps and doing OAuth, and you've got to remember the tokens you get. The example you gave was from Twitter,

but it could be from all over the place. So Trent, we got multiple pieces of feedback. One about encrypting the stuff that goes into your user profile. I can't remember who, but I apologize about forgetting the name, but someone sent in a message that says, well, the AWS CLI just puts your tokens straight there unencrypted.

So there's that. I said maybe you should encrypt them somehow and i agree with that uh still but trentson in this project called keyring and keyring is keyring or vault vaults those types of things there are ways sort of more um managed central stores of this type of information right on mac os you hear uh and it put it in your os x keychain right you probably heard that or the windows credential store or those those things that the actual operating system is protecting from other apps to go look at it but it's basically just encrypted yeah login password or or tokens yeah so this key ring thing that suggested is something like that, but it works. It's a Python library. It works across platform and it works with different backends based on both what platform you're on and other things you might decide. from Python, which I think is fantastic. So on macOS, that's Keychain.

On Linux, it's the Secret Service or the KDE 4 and 5 K wallets. And then on Windows, it's the Windows Credential Locker. Okay. Right. And so in there, you can just call set password and get password and off it goes. And that's pretty much it, right? But it's stored in a nice encrypted,

not just encrypted, but protected access way for the OS. Yeah, so I actually forgot about this. I actually use this for testing all the time, but I never thought, I didn't think about using it for a command line application. Interesting. Okay, how do you use it for testing? So we have, some of the issues are, we have different devices that we're testing against that are password protected devices. And so you had, in order to access them, you need the, like if you're SSH-ing into something or something like that as part of your process, you have to have those credentials somewhere.

And we don't want them in our source code. That's the gist of it is we don't want them just to be, yeah, we don't want them in the source code and checked in to GitLab and to have the whole company be able to read them. It's still protected. It's an internal thing. But maybe you're on GitLab or GitHub or something, and it's a public repo. You don't want any passwords right there. But you can have them stored on your local machine and then pull them out with keyring. It surprised me a little bit that GitPassword's a thing.

I kind of expected it to be like a like get the uh get the password hash or something but i have to remember this isn't this isn't verifying passwords it's having them to be able to send them to another system uh so right ideally that one is storing the hash not the real thing yeah exactly so yeah yeah i don't know if it were to see so i don't know if this would be, this is still cool and I'm glad we're covering it, but I was, my original question was around, is this a re what's a reasonable thing to store passwords for sessions for

command line application? And I don't know if key ring would work, but I haven't tried it yet. And maybe if you have a set password, maybe it will work. Maybe it stores something locally. So I'll have to try it out. Yeah.

I think that it will. Okay. The question that I was wondering is, what about the get password? You know, is that restricted to the process that put it in there? Yeah. Or is it anything running on the system?

Yeah, exactly. Can you just start arbitrarily asking for stuff? Probably some restrictions there, but I don't know exactly what they are. Pamphil out in the audience says, I'm not mistaken, poetry is using the fits installed. So that's where your PyPI credentials get installed.

And you can check out issue 210 from poetry. And down here somewhere says, they talk about ways in which you could store. And it says, why not just make key ring a dependency? Yeah. If this approach, why not simply make keyring a dependency okay yeah if this approach why not simply make keyring and dimension and so on and so yeah it talks about um basically using this to store your ipi credentials and that's a cli app perfect then we have it sounds like a pretty good

match yeah nice example yeah you can just follow along what they're doing there so thank you pamphil for pointing that out yeah yeah well, I don't currently have any use for this. I think it might be useful even outside of, I have this interactive application, for example, storing secrets. You know, if you want to have the database connection string to your app, right? This might be a good way to do it.

One other thing that's interesting is you can have third-party backends. So you could have just encrypted text files. You could have the Dbus API for Linux. Google Sheets? I don't know about this, but it's safe

for use with IPython Secrets, so maybe it encrypts them. But more realistically, we've talked about Bitwarden before, an open-source password manager, which is really nice. I use that for a few things. And so that has a CLI aspect. You could have Bitwarden as a backend. You can write your own as well. And 1Password has a CLI option as well for storing SSH keys even. So you could even put

your SSH keys in there and whatnot. I don't know if this would pull it back correctly, but there's a lot of ways to store things and say 1 one password and then access it with a CLI. Maybe you could plug this in. So just, it's another provider, which is cool. Yeah. Anyway, seems really nice to me. If I have a use for it, I'll definitely look into it more.

Yeah, cool. Nice. Yep. All right. Hey, that might be all of our topics for the day, huh? I think so. Yeah. So, extras?

You got any extras? I just wanted to say that I am, I'm working on a couple of things. I'm editing my PyTest course, of course, still working on that. But the other thing that I just started, which I'm super excited about, is I just started taking a fast API course. Oh yeah. It's really neat. The instructor's awesome. Yeah. Awesome. Yeah. Thank you. That, yeah, that's the FastAPI course, this live course that I'm doing this week and next week, right?

Yeah. Yeah. So I'm taking it from Michael. And if anybody, if anybody is, you've never taken a, taken one of the online courses with Michael or a live course, it's just really excellent. He's a good instructor. So it's good. Thank you very much.

Yeah. I love it. We're having a good time just playing with code as a group. Yeah. I think I have some extras just really quickly here. Brian Skin, who's been a co-host before, sent us a tweet and said, attention Python bytes. And that went over to this message from Jeff, Jeff Huntley. And it says, GitLab, are you all right? And this is linking to an article from the register. It says, GitLab plans to delete dormant projects in free accounts, hoping to save a quarter of the hosting costs by binning repos that haven't been touched for a year. Yikes.

That's a little nerve wracking because just because it hasn't been changed doesn't mean it's not useful. Used. Yeah. Yes. Oh, maybe I keep my recipes up there and I haven't added any recipes lately. Yeah, maybe nothing's changed or whatever. So a couple of things. PSA, if you have a GitLab project, you know, maybe just touch, just add a period to some

text file or something and check that in. Spam your own repo with like uh trivial prs yeah exactly i fixed this misspelling here by changing the word and the more uh oh mario muñoz says they may be reverting this okay so this was just from four or five days ago yeah and pamphil says yes they did okay well okay it sounds like they had the same face many people had the same reaction that we are going oh boy this seems like a bad idea i'm glad this was changed yeah uh pample says because of the huge backlash i can imagine yeah all right well i guess i'll have to continue to pay their million dollars extra per year to host things that people put up on there where they said

they would host it yeah well hopefully they okay even if you where they said they would host it. Yeah. Well, hopefully they, okay. Even if you did that, hopefully they would like email people at least hopefully your current. Yeah, exactly. Kind of like Google voice, please log in with the next, within the next 30 days to keep your phone number or whatever it is they always did to me. Okay. Well with that, I think, uh, brings us to some jokes. You got some jokes to tell and I brought a quick one as well. Okay. Uh, sure. I'm, I'm not going to read the ones up here, but I got them. I got a couple jokes from a place called, from a GitHub repo. That's a dad style programming jokes, which is perfect for me. So I got a couple.

How do programming pirates pass method parameters? I don't know. With virargs. Awesome. Okay. Second um uh how do you get code how do you get the code of your bank vault like so you can break into the bank i don't know you check out their branch this is bad nice i love it and then uh one of the things i liked on the top of this it says unfortunately these jokes only work if you get them. Oh, so good.

And it's a GitHub repo. So that's actually fitting. So anyway, it's very self-referential, very meta. OK, how about you? Here's a quick one before I put it up on screen. You know how there's this constant not built here syndrome?

Like, sure, this key ring is cool, but did we build it? No, I bet we could build a better key ring than that. And like, we'll get a team together to build key ring. Right. So here's a picture of normal people acting like developers. So there's these two construction workers with their hard hats on and there's a screwdriver with a $2 tag on it because it was just purchased. And one guy's outraged. What? Did you buy a screwdriver instead of building your own from scratch? Exactly.

Yeah. Yeah. Pretty good. What? You're using Hugo? Why didn't you build your own blog engine? Exactly. First, I'm going to build my own Markdown parser so I can have better tables.

Let's go. Exactly. Yeah. Yes, indeed. So. All right. Well, excellent podcast as always. Thanks for being here.

Thank you. Yeah, you bet. And thank you everyone for Well, excellent podcast as always. Thanks for being here. Thank you. Yeah, you bet. And thank you everyone for listening, watching, however you've been part of this. Yeah. Thanks a lot.